Quick Start

Visit the Brim Data download page to find the package for your platform.

Zed is distributed as a TGZ/ZIP file of command line executables for each supported platform. To install, unpack with with the appropriate utility for your platform. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v1.0.0/zed-v1.0.0.linux-amd64.tar.gz
tar xzvf zed-v1.0.0.linux-amd64.tar.gz

If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.

Release Notes

• Comprehensive documentation
• Substantial improvements to the Zed language
• Revamped zed command
• New Zed lake format (see #3634 for a migration script)
• New version of the ZNG format (with read-only support for the previous version)
• New version of the ZSON format

Quick Start

Visit the Brim Data download page to find the package for your platform.

Zed is distributed as a ZIP file of command line executables for each supported platform. To install, unpack with your platform's unzip utility. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.33.0/zed-v0.33.0.linux-amd64.zip
unzip zed-v0.33.0.linux-amd64.zip

If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.

Release Notes

• zapi: Rename the ZED_LAKE_HOST environment variable to ZED_LAKE and rename the -host flag to -lake (#3280)
• zq: Improve ZNG read performance when the command line includes multiple input files (#3282)
• zed lake serve: Add the -rootcontentfile flag (#3283)
• Python client: Improve error messages (#3279)
• Python client: Fix Zed bytes decoding (#3278)
• Detect CSV input (#3277)
• zed lake serve: Fix an issue where POST /pool/{}/branch/{} format detection errors caused a 500 response (#3272)
• Fix an issue where the ZSON parser failed to normalize maps and sets (#3273)
• Python client: Add authentication (#3270)
• Python client: Handle query errors (#3269)
• Remove support for the TZNG format (#3263)
• zapi, zed lake serve: Add authentication with Auth0 (#3266)
• Fix an issue preventing casting from ip to ip (#3259)
• zed lake serve: Respect the Accept request header for GET /events (#3246)
• Add function documentation (#3215)
• zed lake serve: Change the default response content encoding to ZSON (#3242)
• zapi load, zed lake load: Add the -meta flag to embed custom metadata in commits (#3237)

Quick Start

Visit the Brim Data download page to find the package for your platform.

Zed is distributed as a ZIP file of command line executables for each supported platform. To install, unpack with your platform's unzip utility. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.32.0/zed-v0.32.0.linux-amd64.zip
unzip zed-v0.32.0.linux-amd64.zip

If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.

Release Notes

• Add create_pool() and load() methods to the Python client (#3232)
• Allow a leading split operator (#3230)
• Remove the exists() function in favor of missing() (#3225)
• Remove the iso() function in favor of time() (#3220)
• Remove deprecated GET /pool and GET /pool/{pool} from the Zed lake service API (#3219)
• Add bytes literals ("0x" followed by an even-length sequence of hexadecimal digits) to the Zed language (#3209)
• When sending a JSON response for POST /query, always send an array (#3207)
• Fix a panic when compiling SELECT ... GROUP BY ... (#3193)
• Fix a bug in which data loaded through the Zed lake service was stored uncompressed (#3198)
• Add all lake index commands to Zed lake service (#3181)
• Reorganize language documentation (#3187)
• Make fuse() output deterministic (#3190)
• Use lake indexes to speed up queries (#3158)
• Fix bug where constants blocked from operator wiring logic (#3185)
• Allow the dot operator to work on a union containing a record (#3178)
• Disable escaping of "&", "<", and ">" in JSON output (#3177)
• Change collect() to handle heterogeneous types with a type union (#3176)
• Extend the join operator to support the anti join type (#3173)
• Make lake index create output the details of the newly created rule (#3168)
• Enable ANSI escapes in command output on Windows (#3164)
• Change zed lake query -stats output to ZSON (#3159)
• Fix a ZSON quoting bug for type value field names (#3154)
• Allow pool names (in addition to pool IDs) in Zed lake service API paths (#3144)

Quick Start

Visit the Brim Download page to find the package for your OS platform.

Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.31.0/zed-v0.31.0.linux-amd64.zip
unzip zed-v0.31.0.linux-amd64.zip

Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.

Release Notes

• Allow indexes to handle fields containing values of different types (#3141)
• Improve CSV writer performance (#3137)
• Fix an issue preventing use of a seek index containing nulls (#3138)
• Add float32 primitive type (#3110)
• Add len() support for bytes, error, and map types (#3136)
• Allow empty ZSON maps (#3135)
• Fix an issue affecting range queries on a lake containing records with a missing or null pool key (#3134)
• Allow from ( pass => ...; ) (#3133)
• Change Go marshaling struct field tag to zed from zng (#3130)
• Fix a panic when reading CSV containing an empty quoted field (#3128)
• Improve CSV output format (#3129)
• Detect JSON input containing a top-level array (#3124)
• Decode top-level JSON arrays incrementally (#3123)
• Remove PPL license (#3116)
• Change ZSON map syntax to |{ key: value, ... }| (#3111)
• Support revert for indexes (#3101)
• Rename zson_parse() to parse_zson() (#3092)
• Add zed lake index update and zed api index update commands (#3079, #3093)
• Add parse_uri() function (#3080, #3084)
• Add from pool@branch:indexes meta query (#3078)
• Fix an issue where sort len(field) produced incorrect output (#3045)
• Remove POST /ast and POST /search from the Zed lake service API (#3065)
• Fix an issue with with record aliases in drop (#3064)

Quick Start

Visit the Brim Download page to find the package for your OS platform.

Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:

curl -OL https://github.com/brimdata/zed/releases/download/v0.30.0/zed-v0.30.0.linux-amd64.zip
unzip zed-v0.30.0.linux-amd64.zip

Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.

Release Notes

As you can see below, there's been many changes since the last Zed GA release! Highlights include:

• The introduction of Zed lakes for data storage, which include powerful Git-like branching. See the Zed lake README for details.
• Enhancements to the Zed language to unify search and expression syntax, introduce new operators and functions for data exploration and shaping, and more! Review the Zed language docs for details.

The exhaustive set of changes is listed below. Come talk to us on Slack if you have additional questions.

• Revise Zed language to unify search and expression syntax (#2072, #2152, #2252, #2304, #2294)
• Add join() and split() functions for use on strings (#2098)
• Add array slice expressions (#2100)
• Fix an issue with connection resets after several minutes when posting data to S3 (#2106)
• Fix an issue with parsing IPv6 literals (#2112)
• Make the fuse operator work on nested records (#2052)
• Fix an issue where cut(.) could cause a slice bounds out of range panic (#2107)
• Add is(), fields(), and exists() functions (#2131)
• Add auto-detection of ZSON format (#2123)
• Fix an issue where cut to the root would exit if the referenced field was missing from a record (#2121)
• Fix an issue where put to the root would panic on a non-record field (#2136)
• Add support for parsing map types in ZSON (#2142)
• Add a fuse() aggregate function (#2115)
• Remove backward compatibility with alpha ZNG format (#2158)
• Simplify ZSON by dropping type decorators when a complex value is fully implied (#2160)
• Add a switch operator to allow branched processing (#2087, #2364, #2318, #2336)
• Add constants and type literals to the Zed language (#2181)
• The -I option in zq is now used for file includes (and allows multiple files), while -z now used for compact ZSON output (#2180, #2208)
• Add support for shaping arrays and sets (#2173)
• Fix an issue where outer aliases were being lost when ZSON was read into ZNG (#2189)
• Add the sample operator that returns an example value for a named field, or for each unique record type (#2200, #2211, #2623)
• Make the current record (i.e., this or .) an implicit argument to shape() (#2199)
• Begin deprecating current TZNG format in favor of ZSON (#2208, #2312, #2333, #2338, #2337, #2339, #2340, #2355, #2367, #2377, #2387, #2388, #2389, #2395, #2477, #2485, #2480, #2513, #2520)
• Fix an issue where accidentally reading non-Zed binary data caused a zq panic (#2206)
• Fix an issue where time-sorted aggregations were returning non-deterministic results (#2220)
• Add canonical Zed and the summarize operator as an explicit keyword before invoking aggregate functions (#2217, #2378, #2430, #2698)
• Add support for casting the duration type (#2194)
• Extend join to support inner (now the default), left, and right variations (#2210)
• Fix an issue where Zed would not compile on FreeBSD (#2233)
• Add the zson_parse() function (#2242)
• Fix an issue where filenames containing : could not be read (#2240)
• Handle aliases and typedefs in shaper functions, which also fixes a panic (#2257)
• Improve Zeek reader performance (#2265, #2268)
• Fix an issue where const references were not honored during query execution (#2260)
• Fix an issue where shapers did not handle aliases to different castable types (#2280)
• Add an unflatten() function that turns fields with dot-separated names into fields of nested records (#2277)
• Fix an issue where querying an index in a Zed lake did not return all matched records (#2273)
• Accept type definition names and aliases in shaper functions (#2289)
• Add a reference shaper for Zeek data (#2300, #2368, #2448, #2489, #2601)
• Fix an issue where accessing a null array element in a by grouping caused a panic (#2310)
• Add support for parsing timestamps with offset format ±[hh][mm] (#2297)
• Remove cropping from shape() (#2309)
• Apply a Zed shaper when reading Suricata EVE data, instead of legacy JSON typing (#2298, #2370, #2400)
• Add support for reading comma-separated value (CSV) files (#2317, #2858, #2942, #2963)
• Fix an issue where reading a Zeek TSV log line would cause a panic if it contained too few fields (#2325)
• Add a shape operator, which is useful for cleaning up CSV inputs (#2327)
• Fix an issue where querying a Zed lake index for a named field could cause a panic (#2319)
• Make casting to time and duration types more flexible (#2334, #2442)
• Fix an issue where null values were not output consistently in a group-by aggregation (#2363)
• Fix an issue where the confirmation messages from adding an index were sometimes incomplete (#2361)
• Finalize ZSON duration format to be an extension of durations in Prometheus (#2358, #2371, #2381, #2396, #2405)
• Add functions missing(), has(), and nameof() (#2393, #2708)
• Add prototype support for SQL expressions (#2392)
• Allow type definitions to be redefined (#2386)
• Fix an issue where casting to a named type caused the loss of the type definition name (#2384)
• Add support for Parquet output and rework the Parquet reader (#2227)
• Don't interpret the first zq argument as a query if there are no additional arguments (#2382)
• Fix an issue that was preventing the reference in an expression to a field name containing a . (#2407)
• Add support for ISO time literals and support durations and time literals in expressions (#2406)
• Add support for complex literals (#2403)
• Code/repo reorganization for phasing out "ZQL" or "Z" in favor of "Zed language", or just "Zed" if context allows (#2416, #2431, #2455, #2831)
• Support in with the map data type (#2421)
• Normalize map values created from Zed expressions (#2423)
• Switch to function-style casting (e.g., int64(123) instead of 123:int64) (#2427, #2438)
• Allow shapers to to refer to the contents of input records to determine the type to apply (#2426)
• Fix an issue where referencing a non-existent table in a SQL query caused a panic (#2432)
• Accept - (stdin) as a zapi argument for loading data (#2435)
• Fix an issue where a single bad cast could cause input processing to halt (#2446)
• Create the zed command with sub-commands like query and api, but shortcut commands (e.g., zq, zapi) still remain (#2450, #2465, #2466, #2463, #2624, #2620)
• Rename ZAR_ROOT environment variable to ZED_LAKE_ROOT (#2469)
• Revise the top-level Zed README to reflect reorganization of the repo and new/changed tools (#2461)
• Remove the -P flag from zq in favor of using from in the Zed language (#2491)
• Add casting of the net data type (#2493, #2496)
• zq now reads its inputs sequentially rather than the prior merged behavior (#2492)
• Extend the len() function to return the number of fields in a record (#2494)
• Remove the -E flag in zed commands that displayed time values as epoch (#2495)
• Add the Zed lake design README document (#2500, #2569, #2595, #2781, #2940, #3014, #3034, #3035)
• Fix an issue where escaping quotes caused a parse error (#2510)
• Fix an issue where multiple ZSON type definitions would be output when only the first was needed (#2511)
• Use less buffer when decoding ZSON (#2515)
• Allow aliases of all primitive types to be expressed in ZSON (#2519)
• Revert the "auto-fuse CSV" behavior originally added in #1908 (#2522)
• Add support for Git-style Zed lakes (#2548, #2556, #2562, #2563, #2564, #2566, #2571, #2577, #2580, #2616, #2613, #2738, #2763, #2806, #2808, #2811, #2816, #2860, #2861, #2931, #2944, #2954, #2960, #2976, #2994, #3007, #3013, #3020, #3023, #3024, #3026, #3030, #3031, #3039, #3046)
• Add support for reading JSON format input data via -i json (#2573, #2608)
• Remove the legacy approach for applying Zed types to NDJSON input, as this is now done via Zed shapers (#2587)
• Fix a Go client issue where ZNG marshal of unexported struct fields caused a panic (#2589)
• Show a warning rather than failing when an unset value tries to be cut to the root (#2591)
• Standardize -h usage in Zed CLI tools for showing help text (#2596, #2618)
• Fix an issue where type names that started with primitive type names caused parse errors (#2612)
• Colorize zson -Z output (#2621)
• Remove pcap-related code, as this functionality has been moved to Brimcap (#2632)
• The role previously performed by zqd is now handled by zed lake serve (#2629, #2722)
• Revise ZJSON to encode types and type values using JSON structure instead of ZSON type strings (#2526)
• this can now be used to reference the current top-level record (formerly ., which may be deprecated in the future) (#2650)
• Rework dataflow model and Zed compiler optimizations (#2669)
• Add initial explode operator that can break values from complex fields out into separate records (#2673)
• Fix an issue where including a particular time-typed field in a shaper script caused errors with shaping other fields (#2685)
• Silently discard duplicate fields when reading NDJSON records, which works around Suricata bug 4016 (#2691)
• Fix an issue where ZSON type values were output without parentheses (#2700)
• Swallow single-backslash-escaped / when reading NDJSON, which allows for reading default Suricata EVE output (#2697)
• Improve the error message shown when no Zed lake root is specified (#2701, #2739)
• Require on in join syntax (#2698)
• Add a typeunder() function that returns the concrete type underlying a named type (#2709)
• Improve ZNG scanner performance via multi-threading (#2678, #2682)
• Fix an issue where a shaper created a corrupt time-typed value from an invalid timestamp rather than rejecting it (#2705)
• Simplify keyword search by requiring := for assignment, == for comparison, and using matches for regex & glob match (#2692, #2744, #2773)
• Allow reading data from http:// and https:// targets (#2723, #2732)
• Support for arbitrary pool keys in Zed lakes (#2729, #2752)
• Add API docs for the Zed lake service (#2679)
• Support from file in Zed language in zq, which is particularly useful with join (#2753)
• Fix an issue where certain data could be queried successfully via zq but not if loaded into a Zed lake pool (#2755)
• Revise Python client docs to show double quotes during pip install, since Windows needs that (#2758)
• Fix an issue where a query was incorrectly parallelized by merging on the wrong key (#2760)
• Fix an issue where len() of a null array was evaluating to something greater than zero (#2761)
• Fix an issue where sort with no fields was ignoring alias types and nested fields when picking a sort field (#2762)
• Fix an issue where unexpected cut: no record found warnings were returned by zed lake query but not when the same data was queried via zq (#2764)
• Move and extend the Zeek interoperability docs (#2770, #2782, #2830)
• Create endpoints in the Zed lake service API that correspond to underlying Zed lake operations, and expose them via zapi commands (#2741, #2774, #2786, #2775, #2794, #2795, #2796, #2920, #2925, #2928)
• Fix an issue where zq would surface a syntax error when reading ZSON it had sent as output (#2792)
• Add an /events endpoint to the API, which can be used by clients such as the Brim app to be notified of pool updates (#2791)
• Simplify the ZSON enum type by removing the values from the list of symbols (#2820)
• Add Zed language documentation for the join operator (#2836)
• Fix an issue where reading ZNG input with more than 222 type definitions triggered a zng type ID out of range error (#2847)
• Have put only return the a referenced field is missing error on first occurrence (#2843)
• Fix an issue where a zed lake query triggered a send on closed channel panic (#2842)
• Allow casting to bool type (#2840)
• Fix an issue where zq would surface an error when reading ZST it had sent as output (#2854)
• Fix an issue where backend errors triggered by zapi query were not being surfaced (#2859)
• Have the Python client use the /query endpoint for the Zed lake (#2869)
• Minimize the amount of surrounding context shown when reporting parse errors (#2864)
• Field assignments in join now behave like cut instead of pick (#2868)
• Add more background/context to Zed top-level language README (#2866 #2878, #2901)
• Unify from, split, and switch syntax to the forms shown here (#2871, #2896)
• Shapers can now cast values of the null type to any type (e.g., arrays or records) (#2882)
• Fix an issue where join was failing to match on values of comparable types (e.g., string and bstring) (#2880, #2884)
• Shapers can now cast a value to a union type (#2881)
• Introduce alternate switch syntax (#2888, #3004)
• When fuse encounters a field with the same name but different types, it now creates one field of union type rather than separate, uniquely-named fields (#2885, #2886)
• Fix an issue where fuse would consume too much memory when fusing many types (#2897, #2899)
• Emphasize in the sort documentation that its output can be non-deterministic in the absence of an explicit field list (#2902)
• Remove the space separator before decorator in ZSON -z output (#2911)
• Fix an issue where handling of record alises caused a failure to shape Zeek NDJSON data (#2904)
• Fix an issue where posting garbage input data to a pool caused an HTTP 500 response (#2924)
• Fix an issue where reading a ZNG file and outputting as CSV caused a deadlock (#2929)
• In a from clause, range is now used instead of over to specify a range scan over a data source (#2943)
• Fix a Zed language issue with parsing parenthesized search terms (#2951)
• Column headers in -f table outputs now reflect the case of the field name rather than always being uppercase (#2964)
• Reserved words in the Zed language can now be used in more places (e.g., field name references) without risk of collisions that would require escaping (#2968)
• Zed CLI tools now send human-readable ZSON by default if output is to a terminal, otherwise binary ZNG (#2979, #2985)
• Temporary directories for spill-to-disk operations now are prefixed with zed-spill- rather than zq-spill- (#2980)
• The put operator keyword is now optional (e.g., can write x:=1 instead of put x:=1) (#2967, #2986, #3043)
• Fix an issue where a put on a nested record with an alias triggered a panic (#2990)
• Fix an issue where temporary spill-to-disk directories were not being deleted upon exit (#3009, #3010)
• Fix a ZSON issue with union types with alias decorators (#3015, #3016)
• The ZSON format has been changed such that integer type IDs are no longer output (#3017)
• Update the reference Zed shaper for Zeek (shaper, docs) to reflect changes in Zeek release v4.1.0 (#3021)
• Fix an issue where backslash escapes in Zed regular expressions were not accepted (#3040)
• The ZST format has been updated to work for typedef'd outer records (#3047)
• Fix an issue where an empty string could not be output as a JSON field name (#3054)

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim10 which provides the latest geolocation data (#2081)
• zql: Add shaping primitive functions cast(), fill(), crop(), and order(), along with fit() and shape() (#1984, #2059, #2073, #2033)
• zson: Read ZSON incrementally rather than all at once (#2031)
• zson: Tighten whitespace in ZSON -pretty=0 output (#2030)
• zql: Change parallel graph syntax to use split and => (#2037)
• zson: Add duration to the implied type list (#2039)
• zq: Fix an issue with rename where a subsequent count() would return no results (#2046)
• zq: Fix an issue where multiple alias typedefs were generated for the same type, causing a TZNG read failure (#2047)
• zson: Fix an issue with string scanning in the ZSON parser that caused the failure parse error: parsing string literal (#2048)
• zq: Fix an issue on Windows where - was not being treated as a way to read from stdin (#2061)
• zq: Add support in put for assigning to . and to nested fields (#2018)
• zson: Fix an issue where reading ZSON caused the failure parse error: mismatched braces while parsing record type (#2058)
• zson: Fix an issue where casting null values to string types caused invalid output (#2077)

Visit the Brim Download page to find the package for your OS platform.

NOTE - Beginning with this release, a subset of the source code in the github.com/brimsec/zq GitHub repository is covered by a source-available style license, the Polyform Perimeter License (PPL). We've moved the PPL-covered code under a ppl/ directory in the repository. The majority of our source code retains the existing BSD-3-Clause license.

The overwhelming majority of zq/zqd users and developers will not be impacted by this change, including those using zq/zqd in commercial settings. The use of the source-available Polyform Perimeter license prevents use cases like marketing a work as a "as-a-service" style offering for server components like zqd while using material covered under the PPL.

In general, we are making this change to ensure technology giants can't use the PPL-covered code to make replacement offerings of our projects. We believe users and developers should have access to the source code for our projects, and we need a sustainable business model to continue funding our work. Using the source-available Polyform Perimeter license on portions of the source code lets us realize both.

For more detail regarding licensing, see the CONTRIBUTING.md doc, and feel free to come talk to us on Slack if you have additional questions.

• zqd: Update Zeek pointer to v3.2.1-brim9 which provides the latest geolocation data (#2010)
• zqd: Update Suricata pointer to v5.0.3-brim1 which disables checksum checks, allowing for alert creation on more types of pcaps (#1975)
• zson: Update Zeek Interoperability doc to include current ZSON syntax (#1956)
• zq: Ensure the output from the fuse processor is deterministic (#1958)
• zq: Fix an issue where the presence of the Greek µ character caused a ZSON read parsing error (#1967)
• zqd: Fix an issue where Zeek events generated during pcap import and written to an archivestore were only visible after ingest completion (#1973)
• zqd: Change the logger configuration to output stacktraces on messages of level "warn" and higher (#1990)
• zq: Update performance results to include ZSON read/write (#1974)

Visit the Brim Download page to find the package for your OS platform.

• zq: Fix an issue where nested nulls caused a panic in CSV output (#1954)

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim8 which provides the latest geolocation data (#1928)
• zson: Allow characters . and / in ZSON type names, and fix an issue when accessing fields in aliased records (#1850)
• zson: Add a ZSON marshaler and clean up the ZNG marshaler (#1854)
• zq: Add the source field to the JSON typing config to prepare for Zeek v4.x weird events (#1884)
• zq: Add initial Z "shaper" for performing ETL on logs at import time (#1870)
• zq: Make all aggregators decomposable (#1893)
• zq/zqd: Invoke fuse automatically when CSV output is requested (#1908)
• zq: Fix an issue where fuse was not preserving record order (#1909)
• zar: Create indices when data is imported or chunks are compacted (#1794)
• zqd: Fix an issue where warnings returned from the /log/path endpoint were being dropped (#1903)
• zq: Fix an issue where an attempted search of an empty record caused a panic (#1911)
• zq: Fix an issue where a top-level field in a Zeek TSV log was incorrectly read into a nested record (#1930)
• zq: Fix an issue where files could not be opened from Windows UNC paths (#1929)

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim7 which provides the latest geolocation data (#1855)
• zq: Improve the error message shown when row size exceeds max read buffer (#1808)
• zqd: Remove listen -pprof flag (profiling data is now always made available) (#1800)
• zson: Add initial ZSON parser and reader (#1806, #1829, #1830, #1832)
• zar: Use a newly-created index package to create archive indices (#1745)
• zq: Fix issues with incorrectly-formatted CSV output (#1828, #1818, #1827)
• zq: Add support for inferring data types of "extra" fields in imported NDJSON (#1842)
• zqd: Send a warning when unknown fields are encountered in NDJSON logs generated from pcap ingest (i.e. Suricata) (#1847)
• zq: Add NDJSON typing configuration for the Suricata "vlan" field (#1851)

Visit the Brim Download page to find the package for your OS platform.

• zqd: Update Zeek pointer to v3.2.1-brim6 which provides the latest geolocation data (#1795)
• zqd: Update Suricata pointer to v5.0.3-brimpre2 to generate alerts for imported pcaps (#1729)
• zqd: Make some columns more prominent (moved leftward) in Suricata alert records (#1749)
• zq: Fix an issue where returned errors could cause a panic due to type mismatches (#1720, #1727, #1728, #1740, #1773)
• python: Fix an issue where the Python client did not generate an error when zqd was absent (#1711)
• zql: Allow the len() function to work on ip and net types (#1725)
• zson: Add a draft specification of the new ZSON format (#1715, #1735, #1741, #1765)
• zng: Add support for marshaling of time values (#1743)
• zar: Fix an issue where a couldn't read trailer failure was observed during a zar zq query (#1748)
• zar: Fix an issue where zar import of a 14 GB data set triggered a SEGV (#1766)
• zql: Add a new drop processor, which replaces cut -c (#1773)
• zql: Add a new pick processor, which acts like a stricter cut (#1773, #1788)
• zqd: Improve performance when listing Spaces via the API (#1779, #1786)

Visit the Brim Download page to find the package for your OS platform.

• zq: Update Zeek pointer to v3.2.1-brim5 which provides the latest geolocation data (#1713)
• zql: For functions, introduce "snake case" names and deprecate package syntax (#1575, #1609)
• zql: Add a cut() function (#1585)
• zar: Allow zar import of multiple paths (#1582)
• zar: Fix an issue where a bare word zar zq search could cause a panic (#1590)
• zq: Update Go dependency to 1.15 (#1547)
• zar: Fix an issue where zar zq yielded incorrect event counts compared to plain zq (#1588, #1602)
• zq: Fix a memory bug in collect() that caused incorrect results (#1598)
• zqd: Support log imports over the network (#1336)
• zq: Update performance results to reflect recent improvements (#1605, #1669, #1671)
• zq: Move Zeek & Suricata dependencies into package.json so Brim can point to them also (#1607, #1610)
• zql: Add support for aggregation-less group by (#1615, #1623)
• zqd: Run suricata-update at startup when Suricata pcap analysis is enabled (#1586)
• zqd: Add example Prometheus metrics (#1627)
• zq: Fix an issue where doing put of a null value caused a crash (#1631)
• zq: Add -P flag to connect two or more inputs to a ZQL query that begins with a parallel flow graph (#1628, #1618)
• zql: Add an initial join processor (#1632, #1642)
• zar: Fix an issue where consecutive timestamps caused seek index misses (#1634)
• zar: Fix an issue where time grouping was not working correctly for zar archives (#1650)
• zq/zql: Add support for ZQL comments, multi-line queries, and a -z flag for reading ZQL from a file (#1654)
• zqd: Automatically compact data via a background task (#1625)
• zq: Make ordered merge deterministic (#1663)
• zq: Fix a performance regression (#1672)
• zq: Fix an issue where the JavaScript and Go versions of ASTs could differ (#1665)
• zq: Fix an issue where a lone hyphen in an NDJSON value was output incorrectly (#1673)
• zq: Add an experimental writer for a new format called ZSON (#1681)
• zar: Fix an issue during import that could buffer too much data (#1652, #1696)
• zql: Add a network_of() function for mapping IP addresses to CIDR nets (#1700)
• zql: Add a docs example showing by grouping with non-present fields (#1703)

Visit the Brim Download page to find the package for your OS platform.

• zql: Add week as a unit for time grouping with every (#1374)
• zq: Fix an issue where a null value in a JSON type definition caused a failure without an error message (#1377)
• zq: Add zst format to -i and -f command-line help (#1384)
• zq: ZNG spec and zq updates to introduce the beta ZNG storage format (#1375, #1415, #1394, #1457, #1512, #1523, #1529), also adddressing the following:
  • New data type bytes for storing sequences of bytes encoded as base64 (#1315)
  • Improvements to the enum data type (#1314)
  • Special characters like . and @ may now appear in field names (#1291)
  • A set may now only support elements of a single type (#1220, #1515)
  • Remove the byte type from the spec in favor of uint8 (#1316)
  • New data type map, which is like set but the contents are key value pairs where only keys need to be unique and the canonical order is based on the key order (#1317)
  • First-class ZNG types (#1365)
  • New numeric data types float16 and float32 (not yet implemented in zq) (#1312, #1514)
  • New numeric data type decimal (not yet implemented in zq) (#1522)
• zq: Add backward compatibility for reading the alpha ZNG storage format (#1386, #1392, #1393, #1441)
• zqd: Check and convert alpha ZNG filestores to beta ZNG (#1574, #1576)
• zq: Fix an issue where spill-to-disk file names could collide (#1391)
• zq: Allow the fuse processor to spill-to-disk to avoid memory limitations (#1355, #1402)
• zq: No longer require _path as a first column in a JSON type definition (#1370)
• zql: Improve ZQL docs for aggregate functions and grouping (#1385)
• zql: Point links for developer docs at pkg.go.dev instead of godoc.org (#1401)
• zq: Add support for timestamps with signed timezone offsets (#1389)
• zq: Add a JSON type definition for alert events in Suricata EVE logs (#1400)
• zq: Update the ZNG over JSON (ZJSON) spec and implementation (#1299)
• zar: Use buffered streaming for archive import (#1397)
• zq: Add an ast command that prints parsed ZQL as its underlying JSON object (#1416)
• zar: Fix an issue where zar would SEGV when attempting to query a non-existent index (#1449)
• zql: Allow sort by expressions and make put/cut expressions more flexible (#1468)
• zar: Move where chunk metadata is stored (#1461, #1528, #1539)
• zar: Adjust the -ranges option on zar ls and zar rm (#1472)
• zq: Choose default memory limits for sort & fuse based on the amount of system memory (#1413)
• zapi: Fix an issue where create and find were erroneously registered as root-level commands (#1477)
• zqd: Support pcap ingest into archive Spaces (#1450)
• zql: Add where filtering for use with aggregate functions (#1490, #1481, #1533)
• zql: Add union() aggregate function (#1493, #1534)
• zql: Add collect() aggregate function (#1496, #1534)
• zql: Add and() and or() aggregate functions (#1497, #1534)
• zq: Fix an issue where searches did not match field names of records with unset values (#1511)
• zq: Fix an issue where searches were not reaching into records inside arrays (#1516)
• zar: Support microindexes created with a sorted flow of records in descending order (#1526)
• zapi: Allow zapi post of S3 objects (#1532)
• zar: Add the zar compact command for combining overlapping chunk files into single chunks (#1531)
• zar: Use chunk seek index for searching chunk data files (#1537)
• zq: Make timestamp output formatting consistent (#1550, #1551, #1557)
• zq: Update LZ4 dependency to improve performance (#1556)
• zq: Fix an issue where TZNG fields containing ] were treated as a syntax error (#1561)
• zar: Fix an issue where the zar import target size didn't take compression into account (#1565)
• zapi: Add a -stats option to zapi pcappost (#1538)
• zqd: Add a Python zqd API client for use with tools like JupyterLab (#1564)

Visit the Brim Download page to find the package for your OS platform.

• zq: Change the implementation of the union type to conform with the ZNG spec (#1245)
• zq: Make options/flags and version reporting consistent across CLI tools (#1249, #1254, #1256, #1296, #1323, #1334, #1328)
• zqd: Fix an issue that was preventing flows in nanosecond pcaps from opening in Brim (#1243, #1241)
• zq: Fix an issue where the TZNG reader did not recognize a bad record type as a syntax error (#1260)
• zq: Add a CSV writer (-f csv) (#1267, #1300)
• zqd: Add an endpoint for returning results in CSV format (#1280)
• zqd: Add an endpoint for returning results in NDJSON format (#1283)
• zapi: Add an option to return results as a JSON array (-e json) (#1285)
• zapi: Add output format options/flags to zapi get (#1278)
• zqd: Add an endpoint for creating/querying search indexes (#1272)
• zapi: Add commands zapi index create|find for creating/querying search indexes (#1289)
• pcap: Mention ICMP protocol filtering (-p icmp) in help text (#1281)
• zq: Point to new Slack community URL https://www.brimsecurity.com/join-slack/ in docs (#1304)
• zqd: Fix an issue where starting zqd listen created excess error messages when subdirectories were present (#1303)
• zql: Add the fuse processor for unifying records under a single schema (#1310, #1319, #1324)
• zql: Fix broken links in documentation (#1321, #1339)
• zst: Introduce the ZST format for columnar data based on ZNG (#1268, #1338)
• pcap: Fix an issue where certain pcapng files could fail import with a bad option length error (#1341)
• zql: Document the ** operator for type-specific searches that look within nested records (#1337)
• zar: Change the archive data file layout to prepare for handing chunk files with overlapping ranges and improved S3 support (#1330)
• zar: Support archive data files with overlapping time spans (#1348)
• zqd: Add a page containing guidance for users that directly access the root zqd endpoint in a browser (#1350)
• pcap: Add a pcap info command to print summary/debug details about a packet capture file (#1354)
• zqd: Fix an issue with empty records (#1353)
• zq: Fix an issue where interrupted aggregations could leave behind temporary files (#1357)
• zng: Add a marshaler to generate ZNG streams from native Go values (#1327)

Visit the Brim Download page to find the package for your OS platform.

• zq: Improve performance by making fewer API calls in S3 reader (#1191)
• zq: Use memory more efficiently by reducing allocations (#1190, #1201)
• zqd: Fix an issue where a pcap moved/deleted after import caused a 404 response and white screen in Brim (#1198)
• zqd: Include details on adding observability to the docs for running zqd in Kubernetes (#1173)
• zq: Improve performance by removing unnecessary type checks (#1192, #1205)
• zq: Add additional Boyer-Moore optimizations to improve search performance (#1188)
• zq: Fix an issue where data import would sometimes fail with a "too many files" error (#1210)
• zq: Fix an issue where error messages sometimes incorrectly contained the text "(MISSING)" (#1199)
• zq: Fix an issue where non-adjacent record fields in Zeek TSV logs could not be read (#1225, #1218)
• zql: Fix an issue where cut -c sometimes returned a "bad uvarint" error (#1227)
• zq: Add support for empty ZNG records and empty NDJSON objects (#1228)
• zng: Fix the tag value examples in the ZNG spec (#1230)
• zq: Update LZ4 dependency to eliminate some memory allocations (#1232)
• zar: Add a -sortmem flag to allow zar import to use more memory to improve performance (#1203)
• zqd: Fix an issue where file paths containing URI escape codes could not be opened in Brim (#1238)

Visit the Brim Download page to find the package for your OS platform.

• zqd: Publish initial docs for running zqd in Kubernetes (#1101)
• zq: Provide a better error message when an invalid IP address is parsed (#1106)
• zar: Use single files for microindexes (#1110)
• zar: Fix an issue where zar index could not handle more than 5 "levels" (#1119)
• zqd: Fix an issue where zapi pcappost incorrectly reported a canceled operation as a Zeek exit (#1139)
• zar: Add support for empty microindexes, also fixing an issue where zar index left behind empty files after an error (#1136)
• zar: Add zar map to handle "for each file" operations (#1138, #1148)
• zq: Add Boyer-Moore filter optimization to ZNG scanner to improve performance (#1080)
• zar: Change "zdx" to "microindex" (#1150)
• zar: Update the zar README to reflect recent changes in commands/output (#1149)
• zqd: Fix an issue where text stack traces could leak into ZJSON response streams (#1166)
• zq: Fix an issue where an error "slice bounds out of range" would be triggered during attempted type conversion (#1158)
• pcap: Fix an issue with pcapng files that have extra bytes at end-of-file (#1178)
• zqd: Add a hidden -brimfd flag to zqd listen so that zqd can close gracefully if Brim is terminated abruptly (#1184)
• zar: Perform zar zq queries concurrently where possible (#1165, #1145, #1138, #1074)

Visit the Brim Download page to find the package for your OS platform.

• zq: Move third party license texts in zq repo to a single acknowledgments.txt file (#1107)
• zq: Automatically load AWS config from shared config file ~/.aws/config by default (#1109)
• zqd: Fix an issue with excess characters in Space names after upgrade (#1112)

Visit the Brim Download page to find the package for your OS platform.

• zq: ZNG output is now LZ4-compressed by default (#1050, #1064, #1063, ZNG spec)
• zar: Adjust import size threshold to account for compression (#1082)
• zqd: Support starting zqd with datapath set to an S3 path (#1072)
• zq: Fix an issue with panics during pcap import (#1090)
• zq: Fix an issue where spilled records were not cleaned up if zq was interrupted (#1093, #1099)
• zqd: Add -loglevel flag (#1088)
• zq: Update help text for zar commands to mention S3, and other improvements (#1094)
• pcap: Fix an out-of-memory issue during import of very large pcaps (#1096)

Visit the Brim Download page to find the package for your OS platform.

• zql: Fix an issue where data type casting was not working in Brim (#1008)
• zql: Add a new rename processor to rename fields in a record (#998, #1038)
• zqd: Fix an issue where API responses were being blocked in Brim due to commas in Content-Disposition headers (#1014)
• zq: Improve error messaging on S3 object-not-found (#1019)
• zapi: Fix an issue where pcappost run with -f and an existing Space name caused a panic (#1042)
• zqd: Add a -prometheus option to add Prometheus metrics routes the API (#1046)
• zq: Update README and add docs for more command-line tools (#1049)

Visit the Brim Download page to find the package for your OS platform.

• zq: Fix an issue where the inferred JSON reader crashed on multiple nested fields (#948)
• zq: Introduce spill-to-disk groupby for performing very large aggregations (#932, #963)
• zql: Use syntax c=count() instead of count() as c for naming the field that holds the value returned by an aggregate function (#950)
• zql: Fix an issue where attempts to tail too much caused a panic (#958)
• zng: Readability improvements in the ZNG specification (#935)
• zql: Fix an issue where use of cut, put, and cut in the same pipeline caused a panic (#980)
• zql: Fix an issue that was preventing the uniq processor from working in the Brim app (#984)
• zq: Fix an issue where spurious type IDs were being created (#964)
• zql: Support renaming a field via the cut processor (#969)