A jq clone focussed on correctness, speed, and simplicity
MIT License
Bot releases are visible (Hide)
This release finishes the work started in jaq 1.4 towards making jaq being able to process non-JSON values. In particular, a vast majority the filters in jaq's core library can now process any kind of value that satisfies certain conditions; see the corresponding PR. Currently, this is mostly interesting for people who want to use jaq as an API.
Since about two months, the compilation of jaq via cargo install jaq
failed. This happened because a dependency of jaq called ariadne
(responsible for printing compilation errors in jaq) broke semantic versioning. Unfortunately, despite the issue being known upstream, at the time of writing there has been no action towards resolving this problem. This put jaq in an awkward position and was somewhat ironic: the library responsible for reporting compilation errors caused compilation errors.
Eventually, I decided to resolve this problem by replacing ariadne
. After having evaluated a number of alternative code span printing libraries, such as miette
and codespan-reporting
, I wrote a new library called codesnake
. Starting from this release, codesnake
is in charge for formatting code that is printed during compilation errors in jaq. One direct benefit of this change is that compilation errors are now also colored on the jaq playground.
At this point, I would like to thank @zesterer for having created and maintained ariadne
. Without it, the error reporting in jaq would not be close to what it is today.
This release corrects escaping of special characters in strings; in particular, newlines tabs, and quotes ("
) were not correctly escaped. For example, for the filter "\n" | tojson
, jaq 1.4 yields "\"\n\""
, whereas jaq 1.5 yields "\"\\n\""
(the difference being an additional backslash before n
). Among others, this shows up in string interpolation ("\(.)"
), tostring
, and tojson
.
Apart from ariadne
, this release also removes another dependency, namely colored_json
. This makes it possible to print output values without first converting them to a format that colored_json
can interpret (serde_json::Value
). This should favourably influence the performance of printing output values. In the long run, this will allow to remove jaq's dependency on serde, further contributing to shorter compilation times and smaller binary size.
Furthermore, when using --color never
, compilation errors are now also printed without colors (#177).
Full Changelog: https://github.com/01mf02/jaq/compare/v1.4.0...v1.5.0
Published by github-actions[bot] 5 months ago
Work on this release has been sponsored by the NLnet foundation, to whom I would like to express my deep gratitude.
This release brings an exciting new addition: the jaq playground! Unlike its counterpart, the jq playground, the jaq playground runs jq filters on your computer and does not send data to a server to process it there. This makes it also suitable to run filters that yield an infinite number of outputs. For more details, see here.
The other big change is that when jaq is used as a library, you can now use it to process custom types of values. That makes it possible to adapt jaq to process non-JSON data, similarly to fq. See ValT
for a description of the functions that every value type has to provide. (Thanks to @wader for providing feedback!)
In this version, the core and standard libraries of jaq (which provide named functions, such as length
) still only provide functionality for JSON, but I plan to also change this in the future.
--indent
and --tab
by @chutz https://github.com/01mf02/jaq/pull/153
indices
, index
, and rindex
by @kklingenberg https://github.com/01mf02/jaq/pull/158 https://github.com/01mf02/jaq/pull/165
range(from; upto; 0)
https://github.com/01mf02/jaq/pull/170
aarch64-apple-darwin
(Apple Silicon) by @amitksingh1490 https://github.com/01mf02/jaq/pull/166
jaq-interpret
tests pass on 32-bit platforms by @nekopsykose https://github.com/01mf02/jaq/pull/159
Full Changelog: https://github.com/01mf02/jaq/compare/v1.3.0...v1.4.0
Published by github-actions[bot] 9 months ago
This release makes jaq a lot lazier. In particular, several core filters, such as debug
, are now evaluated as late as possible https://github.com/01mf02/jaq/pull/131, and filters passed to path expressions are now also executed lazily https://github.com/01mf02/jaq/pull/150. That means that many filters which did not terminate before now terminate!
Thanks go especially to @kklingenberg for implementing several new functionalities!
a * b
by @kklingenberg https://github.com/01mf02/jaq/pull/130
env
filter and $ENV
variables by @kklingenberg https://github.com/01mf02/jaq/pull/128
.a.[]
https://github.com/01mf02/jaq/pull/145
group_by
terminates as soon as there is an error https://github.com/01mf02/jaq/pull/129
last(f)
filter now returns no output instead of null
when f
yields no output. This is to make it consistent with first(f)
, which has the same behaviour. 51d3f513e6fa2cf41bdf92cd27f7cb45448dd601--compact
to --compact-output
to match jq https://github.com/01mf02/jaq/pull/148
to_entries
and paths
now return objects without sorting keys, like jq 484dd270a084ee7e6f233c781f96444264af530b''
) now yields an error message instead of crashing, thanks to a new ariadne
release --- thanks to @zesterer!foreach
used to crash with a stack overflow at the end, for example when executing foreach limit(1000000; repeat(1)) as $x (0; .+$x)
. This is now not longer the case. 854d22cadfa4c8fa18d53a658a5da5cb139bbd80Full Changelog: https://github.com/01mf02/jaq/compare/v1.2.0...v1.3.0
Published by github-actions[bot] 11 months ago
jaq is a jq clone with an emphasis on correctness, speed, and simplicity.
This release brings initial support for tail-recursion optimisation. For example, the filter
def repeat(f): def rec: f, rec; rec; repeat(1)
yielded a stack overflow after several values in jaq before. Now, jaq will actually yield an infinite stream of ones.
As a result, several filters have been rewritten to leverage tail-recursion optimisation, in particular repeat
and paths
. The filters recurse
, while
, and until
, which previously had to be implemented as native filters, are now implemented by definition, which significantly simplifies the code base.
Furthermore, a few filters are now executed more lazily, including array construction ([...]
) and reduce
/foreach
/for
. For example, the filters
def f: 1, [f]; limit(1; f)
and
def f: f; limit(1; foreach (0, f) as $x (1; .))
yielded a stack overflow in jaq before. Now, both just return 1, like jq.
Finally, as cherry on the icing, jaq now implements the range/3
filter and range/2
supports more maximal elements. For example, range(1; 10; 2)
yields 1, 3, 5, 7, 9
, and range(0; infinite)
now yields the (infinite) stream of all natural numbers.
Full Changelog: https://github.com/01mf02/jaq/compare/v1.1.2...v1.2.0
Like jaq 1.1.1, this is a maintenance release without new features.
It decreases the minimal supported Rust version (MSRV) to allow building jaq on Rust versions as low as 1.64 (released Sept. 22, 2022).
Some parts of jaq, like the interpreter and the core library, now build even with Rust 1.63.
Published by github-actions[bot] 12 months ago
jaq 1.1 did not build on Rust 1.62 anymore due to some new versions of dependencies.
Therefore, jaq 1.1.1 downgrades a few dependencies so that it builds at least on Rust 1.65.
This is now automatically verified by a GitHub action.
Apart from this, this release contains no new functionality.
Full Changelog: https://github.com/01mf02/jaq/compare/v1.1.0...v1.1.1
Published by github-actions[bot] 12 months ago
jaq is a jq clone with an emphasis on correctness, speed, and simplicity.
The largest change in jaq 1.1 is the support for filter arguments in recursive filters. This means that you can now write something like:
def repeat(f): f, repeat(f); repeat(1, 2)
However, unlike in jq, this will currently fail with a stack overflow after a certain number of values (about 12,000 on my machine). To address this issue, tail-call optimisation is necessary, which is planned to be supported in jaq 1.2.
Furthermore, several expressions that were evaluated strictly are now evaluated lazily, in particular cartesian products and objects.
That means that some filters that would previously never yield any input, now can yield an infinite number of inputs.
The API of jaq 1.1 is fully backwards compatible with the API of jaq 1.0.
Full Changelog: https://github.com/01mf02/jaq/compare/v1.0.0...v1.1.0
Published by github-actions[bot] about 1 year ago
jaq is a jq clone with an emphasis on correctness, speed, and simplicity.
jaq 1.0 brings lots of features compared to the last stable release (0.10).
A short summary with examples:
def walk(f): def rec: (.[]? |= rec) | f; rec;
def foo($v; f): ...
@uri "https://www.google.com/search?q=\(.search)"
if . < 0 then - . end
try . * 10 catch "NaN"
For more details on these changes, see the alpha, beta, and gamma release information.
Given that this is an 1.0 release, I will try to maintain a stable API.
Among others, future improvements that should be possible within the current API include optimisation of tail recursive calls (for better recursion performance) as well as parallel execution.
Thanks again to all contributors since the last stable release!
Full Changelog: https://github.com/01mf02/jaq/compare/v1.0.0-gamma...v1.0.0
Published by github-actions[bot] about 1 year ago
This release brings support for string interpolation, including custom formatters (thanks to @kklingenberg). That means that you can now write something like @uri "https://www.google.com/search?q=\(.search)"
and have it interpreted like in jq.
Furthermore, like in the upcoming jq 1.7, else
branches in if-then-else expressions can now be omitted.
This version is planned to be the last one before 1.0. No new features are supposed to be integrated into 1.0, only bug fixes.
From this release on, binaries are automatically built. It would be great if someone with an Apple system could test it on their machine.
--nul-output
flag, following jq https://github.com/01mf02/jaq/pull/108
sub
branching https://github.com/01mf02/jaq/pull/110
Full Changelog: https://github.com/01mf02/jaq/compare/v1.0.0-beta...v1.0.0-gamma
Published by 01mf02 about 1 year ago
This release includes a large number of new filters, mostly for:
Furthermore, it adds support for the try f catch g
syntax (@kklingenberg, https://github.com/01mf02/jaq/pull/100), and strings can now be multiplied with integers to repeat them (@nouritsu, https://github.com/01mf02/jaq/pull/86).
The CLI supports the option --nul-output
(@baod-rate, https://github.com/01mf02/jaq/pull/91) and matches jq’s exit code with --exit-status
when there is no output (@baod-rate, https://github.com/01mf02/jaq/pull/92).
Under the hood, the API has changed considerably in order to accommodate for the changes implied by native filters. Most notably, the jaq-interpret crate (containing the interpreter, i.e. the heart of jaq) was split off from jaq-core, and the jaq-syn crate (containing only data types for the syntax) was split off from jaq-parse.
Thanks again to @baod-rate and @kklingenberg for their tremendous efforts!
Full Changelog: https://github.com/01mf02/jaq/compare/v1.0.0-alpha...v1.0.0-beta
Published by 01mf02 about 1 year ago
This release fixes a critical bug that can occur when binding multiple variables. Furthermore, it uses a newer version of the regex crate to allow for a simpler installation.
Full Changelog: https://github.com/01mf02/jaq/compare/v0.10.0...v0.10.1
Published by 01mf02 over 1 year ago
This release adds a lot of new features to jaq:
def foo($v; f): ...
--arg
) can be referred to from definitions nowwalk(f)
, group_by(f)
, unique
, unique_by(f)
, paths
--run-tests
)Thanks are due especially to @passcod for having implemented native filters. Users that use jaq as part of their own programs via the API can now implement filters as Rust functions and expose them to jaq. The API is still subject to change, see https://github.com/01mf02/jaq/pull/70.
Given the extent of these new features, I think that the time is ripe for a 1.0 release.
However, this is an alpha release because especially recursive/nested functions have not yet received the amount of testing that I would expect from a 1.0 release.
As I will be going this week on a cycling trip that will last about two months, I will not be available during this time to address issues myself. However, it is my hope that this time allows users to test the new functions and to report issues (and in the optimal case submit PRs to fix the problems), in order to allow for a more polished 1.0 release later.
Full Changelog: https://github.com/01mf02/jaq/compare/v0.10.0...v1.0.0-alpha
Published by 01mf02 over 1 year ago
jaq is a jq clone with a focus on correctness, speed, and simplicity.
jaq 0.10.0 improves JSON parsing speed, supports regular expression filters and walk(f)
, and improves compatibility of foreach
syntax and arguments handling.
Details:
test
, scan
, match
, capture
, splits
, sub
, gsub
are now supported.(?<name>exp)
is not supported when installing jaq by cargo install jaq
;cargo build
, cargo install --path jaq
or cargo install --git https://github.com/01mf02/jaq
.regex-syntax
, does not support the (?<name>exp)
syntax at the time of writing, so jaq currently uses a patched version of this crate, which is however not considered by cargo install jaq
. In the future, the (?<name>exp)
syntax is likely to be included in regex-syntax
, which will make the (?<name>exp)
syntax available also when installing jaq via cargo install jaq
.foreach
, for
: In jq 0.9, the semantics of foreach xs as $x (init; f)
were changed such that it yielded the output of init
, unlike jq. Following some discussion, the suggestion by @pkoppstein was implemented that restored the previous semantics of foreach
, while introducing a new filter for xs as $x (init; f)
that yields the output of init
. More information can be found in the README.--slurpfile
and --rawfile
options are now supported. Furthermore, arguments (such as passed by --arg a b
) are now bound to $ARGS
.Full Changelog: https://github.com/01mf02/jaq/compare/v0.9.0...v0.10.0
Published by 01mf02 almost 2 years ago
This release greatly improves performance of file reading by memory mapping. On one benchmark, for example, this change decreased runtime by about 27%. Pass input files as arguments to profit from this; i.e., use jaq 'filter' file.json
instead of jaq 'filter' <file.json
.
Several filters now behave more like in jq, for example assignment (=
) and arithmetic update-assignment (+=
, -=
, ...).
Updates using f as $x | g
or if f then g else h
on the left-hand side are now evaluated correctly if f
yields multiple values.
recurse
, reduce
, and foreach
are now evaluated more lazily.
BREAKING CHANGE: foreach xs as $x(init; f)
does now also output the outputs of init
, and the syntax foreach xs as $x(init; f; proj)
is not supported anymore. This makes the behaviour of foreach
easier to describe and reason about, greatly simplifies the implementation and makes foreach
behave more like reduce
. See the README for details.
Users of jaq-core
(the library powering jaq
) might be happy to know that Filter
s are now Send
and Sync
. That means that it is now possible to execute a filter on multiple threads, processing multiple values in parallel. The jaq
application itself does not make use of this (yet).
Full Changelog: https://github.com/01mf02/jaq/compare/v0.8.2...v0.9.0
Published by 01mf02 about 2 years ago
This release adds the --raw-input
/ -R
command-line option to jaq.
This is particularly useful to try the newly included Brainfuck interpreter (based on @itchyny's work) in examples/bf.jq
. :)
Under the hood, jaq tries to avoid more clones when destructuring or updating arrays or objects.
For example, this makes [range(1000000) | [.] | .[]] | length
about 6% faster.
Furthermore, expressions such as f[]
, which consist of some filter directly followed by a path, are now evaluated differently.
In particular, f
is now evaluated lazily, whereas before, it was evaluated strictly.
This makes it possible to write a Fibonacci sequence generator as
[0,1] | recurse([.[1], add])[0]
which previously would not have terminated.
Full Changelog: https://github.com/01mf02/jaq/compare/v0.8.1...v0.8.2
Published by 01mf02 about 2 years ago
The main feature of this release is greatly improved parsing speed (012c9c58a35ee6b41fdca691f74fef5571493074).
Apart from this, there are three new filters suggested by @pkoppstein, namely debug
, finites
, and normals
.
Furthermore, the performance of join
has increased vastly.
Last but not least, the command-line interface has been updated to use Clap 4.
Full Changelog: https://github.com/01mf02/jaq/compare/v0.8.0...v0.8.1
Published by 01mf02 about 2 years ago
The largest addition in this release is support for complex assignments: Previously, the left-hand side of assignments in jaq were restricted to simple path expressions, such as .[]
, .a
. Now, jaq also supports assignments with more general left-hand sides, such as if .. then .. else .. end
, recurse(..)
and several others. This enables filters such as (.posts[] | select(.author == "stedolan") | .comments) |= . + ["terrible."]
(taken from the jq manual --- I do not have such strong feelings towards Stephen Dolan's posts ^^).
This release also adds support for several new filters:
inputs
, input
: These filters make it now possible to process input lazily with state. This effectively makes jaq an impure programming language --- but impurity is restricted to modifying the stream of input values.foreach xs as $x (init; update)
: This is a powerful tool, in particular in conjunction with inputs
. For example, foreach inputs as $x (null; .+$x)
builds a cumulative sum of the input elements, lazily...
: shorthand for recurse(.[]?)
while
, until
: thanks to @capezotteSmaller changes include:
--color
"\""
.Full Changelog: https://github.com/01mf02/jaq/compare/v0.7.0...v0.8.0
Published by 01mf02 over 2 years ago
This version adds a bunch of new command-line options:
-c
/ --compact
: output JSON compactly (thanks to @alex-ozdemir!)-i
/ --in-place
: replace input files by their output--arg
: set contents of variables from the command-lineFurthermore, the try operator (?
) is now supported. In a nutshell, f?
yields those values for which f
does not return an error.
Full Changelog: https://github.com/01mf02/jaq/compare/v0.6.0...v0.7.0