goawk

This release adds support for "getline lvalue" forms. See #85.

This release changes the handling of the builtin functions length, substr, index, and match to use character indexes instead of byte indexes, as per the POSIX spec.

So this is a small backwards-incompatible change, but I think it's 1) warranted given GoAWK tries to conform to POSIX, and 2) won't break most scripts, even ones that use non-ASCII, unless they use constant indexes for substr on non-ASCII strings. To revert to the previous bytes-index behavior, set interp.Config.Bytes to true when using from Go, or use the new goawk -b option for the command-line version.

This does affect the performance of those builtins, as some operations that were O(1) are now O(N) in the length of the string. Still, v1.10.0 introduced other performance improvements, and it's pretty much a wash on the "real world" benchmarks overall.

See PR #83 and issue #35 for further discussion.

This release includes a performance improvement and several bug fixes:

• Only convert numeric string values to number lazily when needed: this gives an 11% average improvement on the real-world benchmarks, and around 40% improvement on my countwords script
• Support ENVIRON special array
• Show parse errors in a simpler, more standard format
• Fix ARGV numeric string handling
• Fix FILENAME numeric string handling
• Fix field numeric string handling: this brings a slight performance decrease, but thankfully the lazy numeric string improvement above more than makes up for it
• Fix parsing of string concatenation with prefix ++ and --
• Fix output synchronization by flushing appropriately

This release also adds the AWKGo AWK-to-Go compiler as described here, but it's a separate executable that doesn't change GoAWK itself at all. Read more about AWKGo.

Fix builds on Go versions before 1.17 due to missing +build constraint. Fixes https://github.com/benhoyt/goawk/issues/74.

CLI: fix handling of "-" argument: https://github.com/benhoyt/goawk/issues/71

Includes the following features and fixes:

• Add support for getline <"-" and print >"-": https://github.com/benhoyt/goawk/pull/63
• Add ShellCommand config option and change default to /bin/sh (on Unix): https://github.com/benhoyt/goawk/pull/64
• Make number literal edge cases behave like awk/gawk: https://github.com/benhoyt/goawk/pull/69
• Disallow statements immediately after another one without ; or newline separator: https://github.com/benhoyt/goawk/pull/68
• Fill in unknown parameter types that are being called with arrays: https://github.com/benhoyt/goawk/pull/70 (fixes https://github.com/benhoyt/goawk/issues/45)
• Various other minor tweaks (see all commits)

Fix go install bug due to presence of -ftest file (see https://github.com/benhoyt/goawk/commit/0adcb1068d3f17cf59d50322e2e94aa3a2210617).

• Add support for "--", stop parsing flags when arg not prefixed with "-" #56
• On Windows, expand wildcards in program (-f) and input filenames #55
• Build and test on Go 1.17 - significant performance improvements

• Add support for fflush(): #42
• Don't buffer stderr: https://github.com/benhoyt/goawk/commit/86f709635946ade82d1c4da274778c3e3e5438df
• Switch from Travis CI to GitHub Actions: https://github.com/benhoyt/goawk/pull/50
• Change README links from godoc.org to pkg.go.dev: https://github.com/benhoyt/goawk/commit/aed202c7aa9c264f5f34d77e07f813d5c2fc75ca
• getline: return -1 instead of erroring when reading non-existent file https://github.com/benhoyt/goawk/issues/41
• Buffer output piped to commands for significant performance boost https://github.com/benhoyt/goawk/commit/89aae73fcb0ba6ebbe4981318901a0d696067911

Fix the precedence handling when parsing unary operators, which fixes things like the "print unique lines" idiom !seen[$0]++.

Thanks @sergeevabc. Fixes #33.

Fix handling of single-character FS.

Per POSIX spec (and the various implementations), a 1-char FS should be handled specially as a straight string split and not a regex. However, I had overlooked this, and was only doing this in the case of "\". This fixes things like FS="|" as reported by @shah in #29 (thanks!).

Bumped up minor version, seems like a fairly significant change rather than just a patch fix.

Also fix split() to have same behavior for single-char FS.

Fixes #29.

Change command line parsing to allow -F and similar flags without a space between the option and the argument (allowed by POSIX), for example -F: (thanks Arnold Robbins)

Also tweaked output of -version to be just the bare version tag so it's machine-readable. -h or --help now includes the copyright message.

First release with binaries attached (see make_binaries.sh). Thanks @cup.

Simplify exec calls to use in/out/err fields directly rather than pipe

Add config options for "safe mode" to prevent exec and file reads/writes. This is useful for running scripts from untrusted user input.

Add support for delete array (with no index). Fix bug with handling of (s)printf format string with three %'s in a row.

Make a[x] array access set the element if not present (per POSIX spec). This seems unintuitive to me, but it's very clear in the spec, and awk/gawk/mawk all do this.

Fix a number of issues found by running (most of) the gawk test suite.

There are still a number of skipped GAWK tests (work to do), and some of GoAWK's behavior doesn't and won't ever match GAWK's 100%. I've gone for the POSIX behavior in most cases.

The full list of changes to GoAWK is:

• Support additional POSIX escape sequences in string literals:
  \a \b \f \v \xhh (hex) \ooo (octal)
• Fix close() to allow reopening a file after it's been closed
• Make sub/gsub backslash handling per POSIX
• Fix bug in RS="" handling:
  • when input is just newlines
  • when there are newlines at the start of the last record
• Add support for ' ' (space) printf flag
• Add parser/resolver errors for:
  • can't use function name as a parameter
  • global var %q can't also be a function
  • can't call local variable %q as function
  • duplicate parameter name %q
• Treat FS of a single backslash as regex \\ (as other AWKs do)

Fix scope issue when calling user function with global var

Only split into fields when fields or NF are accessed. This speeds up scripts that only access the line ($0) and not individual fields.

Fix resolving unknown var types when calling a native function. Thanks @Jeffail for reporting. Fixes #17.