Bot releases are visible (Hide)
Published by nwt almost 3 years ago
Visit the Brim Data download page to find the package for your platform.
Zed is distributed as a ZIP file of command line executables for each supported platform. To install, unpack with your platform's unzip utility. For example, on Linux:
curl -OL https://github.com/brimdata/zed/releases/download/v0.32.0/zed-v0.32.0.linux-amd64.zip
unzip zed-v0.32.0.linux-amd64.zip
If you've installed the Brim desktop app and want to work with its Zed lake from the command line, we recommend using the Zed executables included with the app to ensure compatibility. See this Brim wiki article for their location.
create_pool()
and load()
methods to the Python client (#3232)split
operator (#3230)exists()
function in favor of missing()
(#3225)iso()
function in favor of time()
(#3220)GET /pool
and GET /pool/{pool}
from the Zed lake service API (#3219)POST /query
, always send an array (#3207)SELECT ... GROUP BY ...
(#3193)fuse()
output deterministic (#3190)from
operator wiring logic (#3185)collect()
to handle heterogeneous types with a type union (#3176)join
operator to support the anti
join type (#3173)lake index create
output the details of the newly created rule (#3168)zed lake query -stats
output to ZSON (#3159)Published by nwt about 3 years ago
Visit the Brim Download page to find the package for your OS platform.
Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:
curl -OL https://github.com/brimdata/zed/releases/download/v0.31.0/zed-v0.31.0.linux-amd64.zip
unzip zed-v0.31.0.linux-amd64.zip
Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.
float32
primitive type (#3110)len()
support for bytes
, error
, and map types (#3136)range
queries on a lake containing records with a missing or null pool key (#3134)from ( pass => ...; )
(#3133)zed
from zng
(#3130)|{ key: value, ... }|
(#3111)zson_parse()
to parse_zson()
(#3092)zed lake index update
and zed api index update
commands (#3079, #3093)parse_uri()
function (#3080, #3084)from pool@branch:indexes
meta query (#3078)sort len(field)
produced incorrect output (#3045)POST /ast
and POST /search
from the Zed lake service API (#3065)drop
(#3064)Published by philrz about 3 years ago
Visit the Brim Download page to find the package for your OS platform.
Zed is distributed as a ZIP of command line binaries for each supported platform. To install, unpack with the unzip utility for your OS. For example, on Linux:
curl -OL https://github.com/brimdata/zed/releases/download/v0.30.0/zed-v0.30.0.linux-amd64.zip
unzip zed-v0.30.0.linux-amd64.zip
Note that if you've also installed the Brim desktop app, a set of Zed binaries is already unpacked as part of the app installation. If you intend to use Zed at the command line to work with data in the Zed lake that's behind by Brim, it is recommended to use the binaries there were unpacked with the app, since these have been tested with that app release. See this Brim wiki article for details regarding their location.
As you can see below, there's been many changes since the last Zed GA release! Highlights include:
The exhaustive set of changes is listed below. Come talk to us on Slack if you have additional questions.
join()
and split()
functions for use on strings (#2098)fuse
operator work on nested records (#2052)cut(.)
could cause a slice bounds out of range
panic (#2107)is()
, fields()
, and exists()
functions (#2131)cut
to the root would exit if the referenced field was missing from a record (#2121)put
to the root would panic on a non-record field (#2136)fuse()
aggregate function (#2115)switch
operator to allow branched processing (#2087, #2364, #2318, #2336)-I
option in zq
is now used for file includes (and allows multiple files), while -z
now used for compact ZSON output (#2180, #2208)sample
operator that returns an example value for a named field, or for each unique record type (#2200, #2211, #2623)this
or .
) an implicit argument to shape()
(#2199)zq
panic (#2206)summarize
operator as an explicit keyword before invoking aggregate functions (#2217, #2378, #2430, #2698)duration
type (#2194)join
to support inner
(now the default), left
, and right
variations (#2210)zson_parse()
function (#2242):
could not be read (#2240)const
references were not honored during query execution (#2260)unflatten()
function that turns fields with dot-separated names into fields of nested records (#2277)null
array element in a by
grouping caused a panic (#2310)±[hh][mm]
(#2297)shape()
(#2309)shape
operator, which is useful for cleaning up CSV inputs (#2327)time
and duration
types more flexible (#2334, #2442)null
values were not output consistently in a group-by aggregation (#2363)duration
format to be an extension of durations in Prometheus (#2358, #2371, #2381, #2396, #2405)missing()
, has()
, and nameof()
(#2393, #2708)zq
argument as a query if there are no additional arguments (#2382).
(#2407)in
with the map
data type (#2421)int64(123)
instead of 123:int64
) (#2427, #2438)-
(stdin) as a zapi
argument for loading data (#2435)zed
command with sub-commands like query
and api
, but shortcut commands (e.g., zq
, zapi
) still remain (#2450, #2465, #2466, #2463, #2624, #2620)ZAR_ROOT
environment variable to ZED_LAKE_ROOT
(#2469)-P
flag from zq
in favor of using from
in the Zed language (#2491)net
data type (#2493, #2496)zq
now reads its inputs sequentially rather than the prior merged behavior (#2492)len()
function to return the number of fields in a record (#2494)-E
flag in zed
commands that displayed time
values as epoch (#2495)-i json
(#2573, #2608)cut
to the root (#2591)-h
usage in Zed CLI tools for showing help text (#2596, #2618)zson -Z
output (#2621)zqd
is now handled by zed lake serve
(#2629, #2722)this
can now be used to reference the current top-level record (formerly .
, which may be deprecated in the future) (#2650)explode
operator that can break values from complex fields out into separate records (#2673)time
-typed field in a shaper script caused errors with shaping other fields (#2685)/
when reading NDJSON, which allows for reading default Suricata EVE output (#2697)on
in join
syntax (#2698)typeunder()
function that returns the concrete type underlying a named type (#2709)time
-typed value from an invalid timestamp rather than rejecting it (#2705):=
for assignment, ==
for comparison, and using matches
for regex & glob match (#2692, #2744, #2773)http://
and https://
targets (#2723, #2732)from file
in Zed language in zq
, which is particularly useful with join
(#2753)zq
but not if loaded into a Zed lake pool (#2755)pip
install, since Windows needs that (#2758)len()
of a null
array was evaluating to something greater than zero (#2761)sort
with no fields was ignoring alias types and nested fields when picking a sort field (#2762)cut: no record found
warnings were returned by zed lake query
but not when the same data was queried via zq
(#2764)zapi
commands (#2741, #2774, #2786, #2775, #2794, #2795, #2796, #2920, #2925, #2928)zq
would surface a syntax error when reading ZSON it had sent as output (#2792)/events
endpoint to the API, which can be used by clients such as the Brim app to be notified of pool updates (#2791)enum
type by removing the values from the list of symbols (#2820)join
operator (#2836)zng type ID out of range
error (#2847)put
only return the a referenced field is missing
error on first occurrence (#2843)zed lake query
triggered a send on closed channel
panic (#2842)bool
type (#2840)zq
would surface an error when reading ZST it had sent as output (#2854)zapi query
were not being surfaced (#2859)/query
endpoint for the Zed lake (#2869)join
now behave like cut
instead of pick
(#2868)from
, split
, and switch
syntax to the forms shown here (#2871, #2896)null
type to any type (e.g., arrays or records) (#2882)join
was failing to match on values of comparable types (e.g., string
and bstring
) (#2880, #2884)union
type (#2881)switch
syntax (#2888, #3004)fuse
encounters a field with the same name but different types, it now creates one field of union
type rather than separate, uniquely-named fields (#2885, #2886)fuse
would consume too much memory when fusing many types (#2897, #2899)sort
documentation that its output can be non-deterministic in the absence of an explicit field list (#2902)-z
output (#2911)from
clause, range
is now used instead of over
to specify a range scan over a data source (#2943)-f table
outputs now reflect the case of the field name rather than always being uppercase (#2964)zed-spill-
rather than zq-spill-
(#2980)put
operator keyword is now optional (e.g., can write x:=1
instead of put x:=1
) (#2967, #2986, #3043)put
on a nested record with an alias triggered a panic (#2990)union
types with alias decorators (#3015, #3016)Published by philrz over 3 years ago
Visit the Brim Download page to find the package for your OS platform.
cast()
, fill()
, crop()
, and order()
, along with fit()
and shape()
(#1984, #2059, #2073, #2033)-pretty=0
output (#2030)split
and =>
(#2037)duration
to the implied type list (#2039)rename
where a subsequent count()
would return no results (#2046)parse error: parsing string literal
(#2048)-
was not being treated as a way to read from stdin (#2061)put
for assigning to .
and to nested fields (#2018)parse error: mismatched braces while parsing record type
(#2058)null
values to string types caused invalid output (#2077)Published by philrz over 3 years ago
Visit the Brim Download page to find the package for your OS platform.
NOTE - Beginning with this release, a subset of the source code in the github.com/brimsec/zq GitHub repository is covered by a source-available style license, the Polyform Perimeter License (PPL). We've moved the PPL-covered code under a ppl/
directory in the repository. The majority of our source code retains the existing BSD-3-Clause license.
The overwhelming majority of zq/zqd users and developers will not be impacted by this change, including those using zq/zqd in commercial settings. The use of the source-available Polyform Perimeter license prevents use cases like marketing a work as a "as-a-service" style offering for server components like zqd while using material covered under the PPL.
In general, we are making this change to ensure technology giants can't use the PPL-covered code to make replacement offerings of our projects. We believe users and developers should have access to the source code for our projects, and we need a sustainable business model to continue funding our work. Using the source-available Polyform Perimeter license on portions of the source code lets us realize both.
For more detail regarding licensing, see the CONTRIBUTING.md doc, and feel free to come talk to us on Slack if you have additional questions.
fuse
processor is deterministic (#1958)Published by philrz almost 4 years ago
Visit the Brim Download page to find the package for your OS platform.
Published by philrz almost 4 years ago
Visit the Brim Download page to find the package for your OS platform.
.
and /
in ZSON type names, and fix an issue when accessing fields in aliased records (#1850)source
field to the JSON typing config to prepare for Zeek v4.x weird
events (#1884)fuse
automatically when CSV output is requested (#1908)fuse
was not preserving record order (#1909)/log/path
endpoint were being dropped (#1903)Published by philrz almost 4 years ago
Visit the Brim Download page to find the package for your OS platform.
listen -pprof
flag (profiling data is now always made available) (#1800)Published by philrz almost 4 years ago
Visit the Brim Download page to find the package for your OS platform.
zqd
was absent (#1711)len()
function to work on ip
and net
types (#1725)time
values (#1743)couldn't read trailer
failure was observed during a zar zq
query (#1748)zar import
of a 14 GB data set triggered a SEGV (#1766)drop
processor, which replaces cut -c
(#1773)pick
processor, which acts like a stricter cut
(#1773, #1788)Published by philrz almost 4 years ago
Visit the Brim Download page to find the package for your OS platform.
cut()
function (#1585)zar import
of multiple paths (#1582)zar zq
search could cause a panic (#1590)zar zq
yielded incorrect event counts compared to plain zq
(#1588, #1602)collect()
that caused incorrect results (#1598)package.json
so Brim can point to them also (#1607, #1610)suricata-update
at startup when Suricata pcap analysis is enabled (#1586)put
of a null value caused a crash (#1631)-P
flag to connect two or more inputs to a ZQL query that begins with a parallel flow graph (#1628, #1618)join
processor (#1632, #1642)-z
flag for reading ZQL from a file (#1654)network_of()
function for mapping IP addresses to CIDR nets (#1700)by
grouping with non-present fields (#1703)Published by philrz almost 4 years ago
Visit the Brim Download page to find the package for your OS platform.
week
as a unit for time grouping with every
(#1374)null
value in a JSON type definition caused a failure without an error message (#1377)zst
format to -i
and -f
command-line help (#1384)zq
updates to introduce the beta ZNG storage format (#1375, #1415, #1394, #1457, #1512, #1523, #1529), also adddressing the following:
bytes
for storing sequences of bytes encoded as base64 (#1315)enum
data type (#1314).
and @
may now appear in field names (#1291)set
may now only support elements of a single type (#1220, #1515)byte
type from the spec in favor of uint8
(#1316)map
, which is like set
but the contents are key value pairs where only keys need to be unique and the canonical order is based on the key order (#1317)float16
and float32
(not yet implemented in zq
) (#1312, #1514)decimal
(not yet implemented in zq
) (#1522)fuse
processor to spill-to-disk to avoid memory limitations (#1355, #1402)_path
as a first column in a JSON type definition (#1370)ast
command that prints parsed ZQL as its underlying JSON object (#1416)zar
would SEGV when attempting to query a non-existent index (#1449)put
/cut
expressions more flexible (#1468)-ranges
option on zar ls
and zar rm
(#1472)sort
& fuse
based on the amount of system memory (#1413)create
and find
were erroneously registered as root-level commands (#1477)where
filtering for use with aggregate functions (#1490, #1481, #1533)union()
aggregate function (#1493, #1534)collect()
aggregate function (#1496, #1534)and()
and or()
aggregate functions (#1497, #1534)zapi post
of S3 objects (#1532)zar compact
command for combining overlapping chunk files into single chunks (#1531)]
were treated as a syntax error (#1561)zar import
target size didn't take compression into account (#1565)-stats
option to zapi pcappost
(#1538)zqd
API client for use with tools like JupyterLab (#1564)Published by philrz about 4 years ago
Visit the Brim Download page to find the package for your OS platform.
union
type to conform with the ZNG spec (#1245)-f csv
) (#1267, #1300)-e json
) (#1285)zapi get
(#1278)zapi index create|find
for creating/querying search indexes (#1289)-p icmp
) in help text (#1281)zqd listen
created excess error messages when subdirectories were present (#1303)fuse
processor for unifying records under a single schema (#1310, #1319, #1324)bad option length
error (#1341)**
operator for type-specific searches that look within nested records (#1337)zqd
endpoint in a browser (#1350)pcap info
command to print summary/debug details about a packet capture file (#1354)Published by philrz about 4 years ago
Visit the Brim Download page to find the package for your OS platform.
zqd
in Kubernetes (#1173)cut -c
sometimes returned a "bad uvarint" error (#1227)-sortmem
flag to allow zar import
to use more memory to improve performance (#1203)Published by philrz about 4 years ago
Visit the Brim Download page to find the package for your OS platform.
zqd
in Kubernetes (#1101)zar index
could not handle more than 5 "levels" (#1119)zapi pcappost
incorrectly reported a canceled operation as a Zeek exit (#1139)zar index
left behind empty files after an error (#1136)zar map
to handle "for each file" operations (#1138, #1148)zar
README to reflect recent changes in commands/output (#1149)-brimfd
flag to zqd listen
so that zqd
can close gracefully if Brim is terminated abruptly (#1184)zar zq
queries concurrently where possible (#1165, #1145, #1138, #1074)Published by philrz about 4 years ago
Visit the Brim Download page to find the package for your OS platform.
~/.aws/config
by default (#1109)Published by philrz about 4 years ago
Visit the Brim Download page to find the package for your OS platform.
zqd
with datapath set to an S3 path (#1072)zq
was interrupted (#1093, #1099)-loglevel
flag (#1088)zar
commands to mention S3, and other improvements (#1094)Published by philrz about 4 years ago
Visit the Brim Download page to find the package for your OS platform.
rename
processor to rename fields in a record (#998, #1038)pcappost
run with -f
and an existing Space name caused a panic (#1042)-prometheus
option to add Prometheus metrics routes the API (#1046)Published by philrz over 4 years ago
Visit the Brim Download page to find the package for your OS platform.
c=count()
instead of count() as c
for naming the field that holds the value returned by an aggregate function (#950)tail
too much caused a panic (#958)cut
, put
, and cut
in the same pipeline caused a panic (#980)uniq
processor from working in the Brim app (#984)cut
processor (#969)Published by philrz over 4 years ago
Visit the Brim Download page to find the package for your OS platform.
cut
processor documentation (#924)Published by philrz over 4 years ago
Visit the Brim Download page to find the package for your OS platform.
Time.trunc()
(#842)every X
to use a computed groupby key (#893)cut
processor to emit any matching fields (#899)