goawk

What's Changed

• Fix evaluation of empty {} and END{} by @benhoyt in https://github.com/benhoyt/goawk/pull/230
• fix: close file in Cover.WriteProfile by @testwill in https://github.com/benhoyt/goawk/pull/232
• Fix number formatting on non-64bit architectures by @guilherme-puida in https://github.com/benhoyt/goawk/pull/235
• Make printf %c work on Unicode codepoints by @oliverkwebb in https://github.com/benhoyt/goawk/pull/236
• Improve error message when missing -H or in BEGIN by @benhoyt in https://github.com/benhoyt/goawk/pull/239
• Add printf %i, %a, %A; Fix %o, %x, %X (they should be unsigned) by @benhoyt in https://github.com/benhoyt/goawk/pull/240
• Make srand() use unix time in seconds, and set seed for next srand() by @benhoyt in https://github.com/benhoyt/goawk/pull/241
• Bump up minimum supported Go version to 1.18, and test on 1.23 by @benhoyt in 7f4905588acb85c33b31dcd09e49b6224a91293b

New Contributors

• @testwill made their first contribution in https://github.com/benhoyt/goawk/pull/232
• @oliverkwebb made their first contribution in https://github.com/benhoyt/goawk/pull/236

Full Changelog: https://github.com/benhoyt/goawk/compare/v1.27.0...v1.28.0

What's changed

• Fix wrong precedence of 'expr | getline' expressions by @fioriandrea in https://github.com/benhoyt/goawk/pull/216
• Use C.UTF-8 as locale when invoking external awk program by @guilherme-puida in https://github.com/benhoyt/goawk/pull/226 (first-time contributor)
• New feature: Make redirecting to /dev/stderr work on Windows by @benhoyt in https://github.com/benhoyt/goawk/pull/225

Full Changelog: https://github.com/benhoyt/goawk/compare/v1.26.0...v1.27.0

What's Changed

• Fix rare race in pipe-to-command close() result test by @juster in https://github.com/benhoyt/goawk/pull/213
• Fix #190: FS regex [^,]* fails on empty field by @fioriandrea in https://github.com/benhoyt/goawk/pull/214
• Fix issue #79: Incorrect parsing of complex ++ expressions by @fioriandrea in https://github.com/benhoyt/goawk/pull/215
• Allow optional newline between "do {}" and "while" by @benhoyt in https://github.com/benhoyt/goawk/pull/220
• Distinguish between /regexp/ and "regexp" in stringified output by @ypdn in https://github.com/benhoyt/goawk/pull/222

New Contributors

• @ypdn made their first contribution in https://github.com/benhoyt/goawk/pull/222

Full Changelog: https://github.com/benhoyt/goawk/compare/v1.25.0...v1.26.0

This release includes several minor changes:

• Build binaries with PGO on Go 1.21. GoAWK on Go 1.21 is about 10% faster than on Go 1.20, and PGO makes it another 5-6% faster.
• Bumps up the minimum supported Go version from 1.15 to 1.16.
• Makes the return value of system() and close() for pipes more closely match Gawk's. Internal refactoring of the input and output stream implementation to make this happen. See issues #203 #204 #205 and thanks @juster!
• Support the \u Unicode string escape that's been added to onetrueawk and Gawk recently. #212

This release contains several minor fixes and the addition of --csv (it's an alias for -i csv). This is for compatibility with onetrueawk and Gawk, which are both adding that option soon.

It also contains a minor backwards-incompatible change: in CSV input mode (-i csv), the behaviour of the two-argument form of split() now parses using CSV splitting, rather than FS. This is also in line with the upcoming --csv feature of onetrueawk and Gawk. This is very unlikely to affect anyone, as CSV mode is relatively recent, and it seems unlikely that anyone is using the two-argument form of split() in CSV mode in any case.

Full list of changes:

• Allow { blocks } with just semicolons in them in https://github.com/benhoyt/goawk/pull/192 -- thanks @raygard for the bug report
• Ensure exit without status code doesn't set exit code in https://github.com/benhoyt/goawk/pull/193 -- thanks @raygard for the bug report
• Make two-argument split() respect CSV input mode in https://github.com/benhoyt/goawk/pull/198
• Add --csv option, in line with upcoming AWK and Gawk feature in https://github.com/benhoyt/goawk/pull/199
• Update CSV doc to mention --csv in https://github.com/benhoyt/goawk/pull/200
• Wait for command to finish when a pipe is closed in https://github.com/benhoyt/goawk/pull/201 -- thanks @xonixx for the bug report

This is a patch version that fixes panics when compiling AWK code that uses complex, mutually-recursive functions (fixed in PR #187). Thanks @xonixx for the test case.

This is patch release to fix a bug where mutually-recursive functions would cause an "undefined function" error in the resolver (PR #184). Thanks @xonixx for the bug report (#183).

This is a patch release that fixes a bug in 1.23.0 -- there was a bug that caused a panic in the resolver step with code like function f1(A) {} function f2(x, A) { x[0]; f1(a); f2(a) }. Fixed in #178.

While we're at it, also fix a panic with certain obscure regexes (#179) and limit ARGC to a reasonable maximum (#180).

All three of these issues were found by fuzzing: go test ./interp -fuzz=FuzzSource

This release adds a single new feature: support for length(array) in addition to length(string) (#176). Calling length() on an array is quite useful and is supported by all other awk versions (onetrueawk, Gawk, mawk, busybox awk, frawk). In addition, it's been accepted for inclusion into POSIX, though not yet added to the main spec (which seems to take forever).

This release also includes a complete rewrite of the type resolver (#175). Before the code was quite messy and hard to read, now with the two passes I think it's easier to understand and work with. It was certainly easier to add the length(array) feature with the rewritten resolver than before.

A fairly minor release, fixing some edge cases and adding support for nextfile:

• Make constructs like $++lvalue not an error (#168)
• Optimize constant integer array indexes to avoid toString() at runtime (#169)
• Allow parsing of cond && var=value and similar expressions (#170)
• Add GroupingExpr to fix (a)++b parsing issue; pretty-print precedence (#172)
• Add support for nextfile (#173)

See full list of commits.

Significant changes in this release:

• Added code coverage support! Thanks, @xonixx, for the idea and implementation.
• Made -d, -da, and -dt print to stdout and exit (https://github.com/benhoyt/goawk/pull/155). Thanks, @xonixx.
• Allowed semicolon as terminator for normal pattern without action. Thanks @juntuu for the bug report and fix.
• Bumped minimum Go version to 1.15.

New features and bug fixes (thanks @paulapatience for the reports):

• Add support for -E argument (like Gawk's -E/--exec).
• Make FNR behavior with and without hyphen argument consistent with other AWKs.
• Set FILENAME to "-" for stdin to be consistent with other AWKs.

Minor changes:

• Disable DWARF generation when linking to shrink binaries.
• Order command line flags in -h/--help alphabetically.

Notable changes in this release:

• Resolve types using topological sort to avoid O(N^2) worst case during parsing
• Ignore UTF-8 BOM at start of CSV files (this is often added by the likes of Excel)
• Interpret escapes in -v and var=value arguments: https://github.com/benhoyt/goawk/pull/132 and https://github.com/benhoyt/goawk/pull/134
• Make regexes match multi-line strings like other AWKs
• Add support for hex and hex floating point number conversions

In other news, check out awk-demo, an amazing "old skool demo" written in AWK by @patsie75. It now works under GoAWK, at least on Linux. Clone that repo and run it with awk=goawk ./demo.sh!

Thanks to @ko1nksm for several bug reports.

See full list of commits since v1.18.0.

Relatively minor release with the following changes:

• Test on Go 1.18 and update the minimum Go and CI-tested version from 1.13 to 1.14.
• Wire up stdin in system() and don't buffer stdout in goawk command if stdout is a terminal so that you can now play Tetris with GoAWK. :-)
• Wire up ctx through system() for ExecuteContext.
• Various test and documentation improvements, including a comparison to csvkit in the CSV docs, updating CSV benchmarks to avoid huge.csv, and adding explicit tests using ASCII and Unicode unit and record separators.
• Repo cleanup: moved the awkgo directory and code to a branch and removed it in the master branch; removed the unnecessary examples directory.

See the list of commits.

Minor test fixes, no change in functionality:

• Remove tests of platform-dependent float conversions that fail on arm64 Macs
• Have tests build goawk binary so user doesn't have to

Now with proper CSV input and output support! For example, a simple example showing CSV input parsing and the new @"named-field" syntax:

$ goawk -i csv -H '{ print @"Abbreviation" }' testdata/csv/states.csv
AL
AK
AZ
...

Read the full documentation.

This feature was sponsored by the library of the University of Antwerp -- many thanks!

• Add interp.New ... Execute API to speed up and reduce allocations when executing the same program multiple times. https://github.com/benhoyt/goawk/pull/100
• Add ExecuteContext API to support timeout and cancellation. https://github.com/benhoyt/goawk/pull/103
• Optimized string concatenation when concatenating more than two strings, for example x = "a" "," "b". https://github.com/benhoyt/goawk/pull/99
• Reduce allocations in a few other places, such as print, printf, sprintf(), and field parsing. https://github.com/benhoyt/goawk/pull/102
• Add proper Go 1.18 fuzzing support for fuzzing the AWK source and input. https://github.com/benhoyt/goawk/pull/103

This release adds no new features. It's a significant performance improvement due to switching the internals of the interpreter from a tree-walking interpreter to a bytecode compiler with a virtual machine interpreter.

Results show that it's 18% faster overall on microbenchmarks, 13% on more real-world benchmarks. It should be fully backwards compatible -- please file an issue if you find a regression!

Read the details here.

This reverts the feature from v1.11.0 which changed the builtin functions length, substr, index, and match to use character indexes instead of byte indexes (as per the POSIX spec). The reason is because it changed those functions from O(1) to O(N), which created "accidentally quadratic" behavior in scripts that expected these functions to be O(1).

For example, @xonixx's grok.awk script on a relatively large JSON input file took about 1s in bytes mode (goawk -b), but 8 minutes (!) in the new unicode char default mode. That's extremely problematic.

Like v1.11.0, this release is again a small breaking change, but once again shouldn't affect many scripts (it will again only affect scripts that use constant indexes for substr on non-ASCII strings). I hope not many people are using interp.Config.Bytes or the goawk -b option yet, as those are gone again. Seeing v1.11.0 was only introduced a few weeks ago, I think it's worth the breakage for a performance problem of this magnitude.

Fixes https://github.com/benhoyt/goawk/issues/93: "Major speed regression for gron.awk in goawk 1.11.0+".

Support RS being multiple characters and regular expressions RS (#86), allowing significantly more powerful text processing. This is a Gawk extension to POSIX, which says, "If RS contains more than one character, the results are unspecified."