murex

The previous releases have brought significant advancements to Murex's syntax but at the cost of longer gaps between releases. So the 4.1.x versions will be shorter releases but focusing on bug fixes. The 4.1.x release notes will be appended to murex.rocks changelog and available on Github releases too

v4.1.6140

Breaking Changes

None

Features

• readline: blinking cursor added to fuzzy find. This is just a very minor UI change to make it easier to tell where keyboard input would be updating

• git autocomplete now includes git aliases

Bug Fixes

• merge operator (<~) should unmarshal strings

• scalars available in all assigns (eg +=, -=, etc)

• lazy assigns allowed when strict types is set to false. Meaning merge operations and += (et al) will work even if the left scalar is unset

• suppress !!! style warnings upon ^C

• strip : colon from command name when command string is passed from the pre-parser (#601)

The previous releases have brought significant advancements to Murex's syntax but at the cost of longer gaps between releases. So the 4.1.x versions will be shorter releases but focusing on bug fixes. The 4.1.x release notes will be appended to murex.rocks changelog and available on Github releases too

v4.1.5200

Breaking Changes

None

Features

• Merge into operator added: ~>

• f: new flag i -- deprecates ? flag due to ? being a glob pattern. However ? will remain supported

• $PREFIX variable passed to Dynamic and DynamicDesc functions in autocomplete -- includes the partial term (read more)

• user@ completion support for ssh

• Updated autocomplete docs (read more)

• command line is included in error message when error running inside another module

• errors within other modules now include the command line string to make debugging even easier

• lambdas success/fail criteria is now driven by the exitnum rather than stdout data type. This allows it to be much more robust. In addition $. is now mutable

Bug Fixes

• f improved help message (previously some items displayed code numbers instead of descriptions)

• readline: hint text flicker removed

The previous releases have brought significant advancements to Murex's syntax but at the cost of longer gaps between releases. So the 4.1.x versions will be shorter releases but focusing on bug fixes. The 4.1.x release notes will be appended to murex.rocks changelog and available on Github releases too

v4.1.4200

Breaking Changes

None

Features

• Merge operator added: <~

Bug Fixes

• readline: fixed out of bounds errors when calculating unicode cell positions

• readline: fixed panic raised when vim key p is used on an empty buffer

• readline: fixed bug where vim mode became unavailable / esc didn't work as expected. This was due to autocompletion find (^f, ^r) being launched before tab completions and state not being handled correctly

The previous releases have brought significant advancements to Murex's syntax but at the cost of longer gaps between releases. So the 4.1.x versions will be shorter releases but focusing on bug fixes. The 4.1.x release notes will be appended to murex.rocks changelog and available on Github releases too

v4.1.3000

Breaking Changes

None

Features

• lambda support added to piped indexes, ie example -> [{ lambda }], to mirror existing @example[{ lambda }] functionality

• --autocomplete flag added to read

Bug Fixes

• application specific environmental variables (eg <env:FOO=BAR>) were not getting set

• cast now checks if it is a method. Previously was piping regardless

• fixed by where nested maps would overwrite when altered with --merge. Now they merge as expected

• alter panics are caught issue (#536)

The previous releases have brought significant advancements to Murex's syntax but at the cost of longer gaps between releases. So the 4.1.x versions will be shorter releases but focusing on bug fixes. The 4.1.x release notes will be appended to murex.rocks changelog and available on Github releases too

v4.1.1100

Breaking Changes

• Any code that expects formap values to be strings despite that value being a different primitive from the originating object. However this type of code would be an anti-pattern so we shouldn't expect much, if any, code written like this

Features

• formap values are now data type aware. You no longer need to cast values. Keys are still string only and the intention is to keep them this way

Bug Fixes

• improved error messages around a few internal APIs

• micro optimisations in string parsing (minor order to operations, now converting points at a slightly later stage. This adds zero impact on code complexity and almost zero code refactoring but results in slightly less memory getting copied about)

The previous releases have brought significant advancements to Murex's syntax but at the cost of longer gaps between releases. So the 4.1.x versions will be shorter releases but focusing on bug fixes. The 4.1.x release notes will be appended to murex.rocks changelog and available on Github releases too

Breaking Changes

None

Features

No new features

Bug Fixes

• code references in error messages are cropped to terminal width

• error handling around multiple chained but incompatible tokens (#566)

Minor release: improved error messages raised when using variables

This release sees significant improvements for use with non-latin characters in both the interactive prompt and shell scripting. It introduces new syntax to make working with structured data even easier than before. As well as new data types and smoother user experience.

Breaking Changes:

• $PWD is no longer a str. If you need to parse $PWD as a string then you should out: $PWD or use ${pwd}

• . is now a valid bareword character for variables. This will break any instances in code that contain something like out $hello. however you can now use out $(hello). syntax for safer string infixing

Features:

• dot notation added for variables which allow accessing and writing to object properties. eg $json.array.index

• added support variable encapsulation via $() to avoid variable barewords from leaking into other values when infixed. eg $(foobar) is the same as $foobar

• variable names can now be Unicode, eg $(比如) = "举手之劳就可以使办公室更加环保，比如，使用再生纸。"

• Unicode wide characters are properly handled in readline

• new data type path that is an object

• new data type paths

• support for lambdas added

• readline [ctrl]+[f] now supports word matching as well as regexp

• readline [ctrl]+[f] now supports glob filters (eg *.txt) too

• $ENV and $GLOBAL objects (deprecates the need for global and export)

• improved error messages in expressions

• scalars can be used as assignees in expressions, eg $foo = "bar"

• added -i flag to ssh autocompletion

• readline prompts are now buffered to reduce cursor flickering

Non-User Facing / Maintenance Changes:

• backend support added for data types to be dynamic objects

Bug Fixes:

• better wide character support in auto completion suggestions

• function parameters default and comment fields are no longer order dependent

• better string parsing for barewords inside arrays and objects

• ahead of time hint caching now done on demand per hint page rather than against every executable upon murex start up

• new lines should be ignored in %{}

• bareword true and false should be converted to boolean in %{}

• autocomplete suggestions are now dedup'ed when delayed results return similar suggestions to fast results

• various preview pane bug fixes

• support for ANSI escape sequences in experimental (disabled by default) TTY buffer

• correction to count docs

This release includes mostly bug fixes and new experimental features which are opt into. To enable all experimental features, set the environmental variable MUREX_EXPERIMENTAL to any value. Or you can enable specific features individually via config

Breaking Changes:

• None

Features:

• default statement added to switch (this will deprecate catch, albeit catch will remain supported for compatibility)

• Option to change start directory: config: set shell start-directory $path (where $path is the working directory you wish to start the shell in)

• onCommandCompletion event no spawns a pty. This allows capturing of stdout while still presenting a tty to the executing process

• EXPERIMENTAL: preview files. This can be enabled via config: set shell preview-enabled true

• EXPERIMENTAL: preview images. This can be enabled via config: set shell preview-images true

• EXPERIMENTAL: tty buffer. This can be enabled via config: set shell tty-buffer-enabled true

• ctrl+r has been tweaked to make it more user friendly

• Long overdue SIGWINCH signal handler to reflow after terminal resized

• Lots of improvements to documentation

Non-User Facing / Maintenance Changes

• none

Bug Fixes:

• re-added columns data-type to fix jobs builtin

• added bang alternatives to the glob exception list

• added missing support for && tokens in expressions

• ![] can now produce empty maps

• fixed regression bug in file pipe mutex

• fixed missing bounds check in statement parser

This is a major release that brings a significant number of changes and improvements, including a complete overhaul of the parser. Backwards compatibility is a high priority however these new features bring greater readability and consistency to shell scripting. So while the older syntax remains for compatibility, it is worth migrating over to the newer syntax for all new code being written

Breaking Changes:

• Optional builtin removed: bson. This was disabled by default and likely never used. So it has been removed to reduce murex's testing footprint. It can easily be re-added if anyone does actually use it

• Optional builtin removed: coreutils. This was actually enabled by default for Windows builds. However rewriting Linux/UNIX coreutils for Windows support is a massive project in its own right and with the maturity of WSL there's almost no reason to run murex on "native Windows". So the coreutils builtin has been dropped to allow us to focus on the responsibilities of the shell

Features:

• Support for expressions, eg 5 * 5 or foobar = $foo + "bar", etc. This syntax can be used directly or specified specifically via the expr builtin

• New syntax sugar for creating JSON objects: %{ foo: bar }

• New syntax sugar for creating JSON arrays: %[ foo bar ]

• New syntax sugar for creating strings: %() (this is preferred over the, now deprecated, feature of parenthesis quotes)

• Ranges supported in [] (@[ is now deprecated)

• Support for multiline comments: /# comment #/. It is unfortunate this differs from C-style comments (/* comment */) but this has to be the case because /* is ambiguous for barewords in shells: is it a path and glob or a comment? Where as /# isn't a common term since # is already a comment

• If any processes pass null as a data type across the pipe, it will be ignored. This solves the problem where functions that don't write to STDOUT would still define the data type

• Config option auto-glob renamed to expand-glob, and now enabled by default

• Globbing exclusion list. This allows you to avoid annoying prompts when parameters shouldn't be expanded as globs by the shell (eg when using regular expressions). This can be managed via shell expand-glob-unsafe-commands option in config

• @g removed. It is no longer needed with expand-glob enabled by default

• New builtin: continue: skip subsequent processes in an iteration block and continue to next iteration

• New builtin: break: exit out of a block of code (eg in an iteration loop)

• Additional syntax for index ([): *1: 1st row, *A: 1st column

• New alias: help -> murex-docs. This brings murex a little more inline with Bash et al

• pre-cache-hint-summaries now enabled by default after testing has demonstrated it doesn't have nearly as expensive footprint as first assumed

• Hitting TAB when nothing has been typed in the REPL will suggest past command lines

• ^ autocompletion added

• getfile writes to disk if STDOUT is a TTY

• mkarray (eg ja) now writes an integer array if range is integers. eg ja: [1..3]. This change wouldn't affect a since that outputs as list of strings (for streaming performance reasons) rather than a data type aware document

• debug (method) output tweaked

• Improved error messages in a number places

• Revamped README / website landing page

Non-User Facing / Maintenance Changes

• Minimum Go version supported is now 1.17.x

• Main parser completely rewritten

• ReadArray API now requires a context.Context

• egrep references changed to grep -E to work around GNU grep deprecating support for egrep

• Added marshallers for boolean, null

• Variables.GetValue() now errors instead of returns nil when no variable set

• Additional tests. So many new tests added

• Lots of code refactoring

Bug Fixes:

• regexp wasn't erroring if nothing was matched

• readline: fixed deadlock

• append and prepend now type aware (no longer converts all arrays into string arrays)

• foreach was setting variables as strings rather than honoring their original data type

• yarn autocompletion errors should be suppressed

• spellcheck missing break causing more occasionally incorrect instances of underlined words

• config wasn't passing data type when executing blocks via eval

• debug wasn't setting data type when used as a function

• macro variables don't re-prompt when the same variable is used multiple times

This is a bug fix release:

• readline: fixed deadline in autocomplete
• suppress yarn errors while building yarn autocomplete suggestions

This is a bug fix release:

• https://github.com/lmorg/murex/issues/466

This release mainly focuses on refinements in performance and usability, rather than introducing new features

Breaking Changes:

• None

Features:

• File index caching for faster completions (eg on mechanical drives). This happens upon shell start up (except on Darwin for security reasons) and after every cd

• Relaxed alias syntax. It now supports whitespace between the = statement (like set and let do)

• New aliases added for common methods which work against lists/arrays. Type list.{TAB} to see the complete list of methods

• Added support for alternative separator values in struct-keys

• Hint summaries are now cached, to improve performance on slower machines

• Opt in support to pre-cache hint summaries. This will also auto-disable appending colon to command suggestions in autocomplete. This is not enabled by default because it could negatively affect performance on some slower machines and is more of an aesthetic improvement. You can enable it by adding config: set shell pre-cache-hint-summaries true to your murex profile

• New autocompletions for yay (ArchLinux)

• Some minor performance improvements

• website: updates to docs

Bug Fixes:

• readline: max field size adjusted for autocompletes so more completions are visible at one time

• readline: all cropped autocompletions now provide an uncropped version in the hint text field

• readline: fix bug with history not getting written

• OS signals (^c et al) are now re-registered after each exec (fixes bug where some programs would "steal" ^c, breaking support for those signals upon subsequent execs)

• Fixed regression bug in man page parsing

• Added check for zero-length string used as a process name (this shouldn't ever happen under normal circumstances however closing this bug allows for a managed error rather than the code panicking)

• readline: Prompt environment reset on each loop (this change is to future-proof against potential bugs)

• Fixed race condition bug when some of the tests are run concurrently (this bug only existed in the unit tests but sometimes caused those tests to fail)

• Added CI tests for Go v19.x

Bug fix for range variables in v2.10

This release brings a few minor improvements and bug fixes rather than big new headline features.

Breaking Changes:

• None

Features:

• Added support for ranges to be used as variables. eg

f: +f -> set v
echo: @v[2..]

• Complete re-write of f. It is now the tool I had always intended when I first created that builtin

• g now supports being run as a method (eg to filter items from a file list)

• rx now supports being run as a method (eg to filter items from a file list)

• Updated autocompletes for git

• Updated autocompletes for terraform

Minor Changes:

• Added support for profiling

• Removed unused data types

• Dropped profile/ prefix from the module string for modules and profiles (as seen in FileRef)

Bug Fixes:

• Lots of improvements to how || and && are used in other runmode's, try, and trypipe

• autoglob prompt now has a mutex wrapped around it to prevent concurrent processes sending the prompt haywire

• Spellchecker no longer underlines partial words

• Better removal of \x16 and other non-printable characters leaking into readline

This release focuses on testing and REPL usability improvements but also
includes updates several new run modes to make error handling easier in
larger scripts.

Breaking Changes:

• None

Features:

• >> redirect (append to file) operator added. This is just syntactic sugar for -> >>. To redirect and overwrite/truncate a file use |> rather than >

• Support for automatic globbing in the commandline. This has to be enabled via config: set shell auto-glob true and you will get prompted for each glob. Any globs inside blocks will be ignored and this only works on the interactive prompt (ie not in scripts or background processes) -- these limitations are not a bug but by design

• Dynamic autocomplete suggestions are now cached for 5 seconds by default. This cache can be changed via the CacheTTL directive in autocomplete

• Additional scoping for try and trypipe. You can define the scope as being an entire function or a module. If set for an entire module then any functions, autocompletions, etc inside that module will inherit that try or trypipe mode automatically. These are set via the new builtin runmode

• New reserved variable $PWD (this was just a global variable previously)

• Additional flags added to range, @[

• New builtin, man-get-flags. This exposes murex's internal man page parser to the command line scripts

• Binaries now produced for ARM64 Windows

Minor Changes:

• The order in which namespaces are looked up has been altered since privates should be resolved before functions

• Updated autocomplete for homebrew, brew

• New autocomplete for AWS CLI, aws

• New autocomplete for yarn, yarn

• murex unit tests should report progressively rather than everything upon completion

• Minimum recommended version of Go has been raised to 1.15. This is due to some bugs in dependencies when running tests against v1.13 and v1.14. murex will technically compile and work on them but it not worth my time debugging why the CircleCI tests fail for those versions given their age

• Vendor directory removed. This isn't really needed now support is focused on newer versions of Go

• Website: New 404 error

• Website: search bar added

• The start of a massive push towards upping the test coverage

• Lots of improvements to the docs

Bug Fixes:

• FileRef module string for functions forked from the REPL changed to murex/shell to bring it inline with the naming convention of other modules

This release comes with a number of experimental but stable features that might eventually become standard practice. The features are there to use if you with but adjacent from the older code so there is zero risk in updating to this version.

Breaking Changes:

• None

Features:

• New data-type jsonc (Concatenated JSON). This is an experimental parser for reading multiple JSON documents in a single file or stream. The expectation is that it will eventually replace jsonl (jsonlines) and possibly even the default json data-type

• Autocomplete can now split on =. However this is an experimental feature

• New autocomplete directive: Goto. This is an experimental way to add a little more structure to autocomplete definitions without resorting to using dynamic code everywhere

• autocomplete directive FlagValues now supports two values: "*" and "". These define default properties regardless of the flag used

Minor Changes:

• Unmatched records in index ([) when parsing a table now produces the same configurable behavior (erroring by default but which can be disabled) as when using index against an object

• Autocompletion suggestions for yarn

• Data-type field in function is now optional

• Lots of improvements to the docs

Bug Fixes:

• $MyVar[[/path/to/element]] now behaves the same when used infixed as it did when used as a function

Breaking Changes:

• zero length arrays returned from subshells (eg echo @{g this-file-does-not-exist}) should fail by default, like unset variables. This is enabled by default but can be disabled via config: set proc strict-arrays false

• autoglob should fail if it doesn't match any results. eg @g echo this-file-does-not-exist.*

• Subshells should fail parent command when used inside try and trypipe blocks. eg try { echo ${false} }

Features:

• function now supports defining parameters

• Added support fro && eg do-something && do-something-else for chaining successful commands

• Added support for || eg do-something || do-something-else for chaining unsuccessful commands

• Added support for writing to the terminal emulator's titlebar via config: set shell titlebar-func { block }

• titlebar-func can also be written to your tmux window title via config: set shell tmux-echo true.

• New reserved variable: $HOSTNAME

• New reserved variables: $1 (and upwards) which correlates to the scope's parameter index. $1 is the functions first parameter. $2 is the second. $13 is the thirteenth.

• New reserved variable: $0 which contains the function name

• New event: onCommandCompletion (this is experimental and subject to change in the next release)

• Macro variables. eg echo Hello ^$name will prompt the user to enter a name. Macro variables are only support in the REPL

• read now supports flags (eg default option, etc) to allow for a better experience in shell scripting

Minor Changes:

• You can now overwrite onKeyPress events. This no longer produces an error forcing you to remove the old event before adding the new one

• Autocompletion suggestions shouldn't be sorted is results include paths (improves the ordering of autocompletion suggestions)

• Autocompletion suggestions for openagent builtin

• Autocompletion suggestions for hashtags

• Test counts re-added to website

• Windows should show file extensions by default in autocompletion results

Bug Fixes:

• Fix inverted logic on forceTTY: config: get proc force-tty false set by default, which then proxies STDERR and prints them in red

• ctrl+c (^c) now currectly sends SIGTERM rather than just ending the child process

• Better handling of SIGTERM

• SIGTSTP isn't working. Switched to SIGSTOP when (^z) doesn't

• Fix panic in event onFilesystemChange where fileRef is not getting passed correctly

• Fix panic in event onFilesystemChange where path is zero length string

• Some improvements to variable previews in the REPL

• count should check if it is a method

• AST cache now checked more regukarly. This is to reduce the risk of memory leaks during fuzz or load testing

• murex-docs still referred to len builtin. That should be changed to count

• Lots of fuzzing added -- a few edge case bugs discovered

Breaking Change:

• strict-vars is enabled by default. This means if a variable doesn't exist / hasn't been instantiated then your command will fail. You can disable this by putting config: set proc strict-vars false inside any affected functions and/or in your ~/.murex_profile

Features:

• select builtin now added to the default build. This built in will stream tabulated data into an in memory sqlite3 database and take SQL arguments to query that data. Thus allowing you to inline SQL queries as part of your native command line. This addition will add complications for anyone building from source on Windows. If you are a Windows user you are recommended to use WSL or, if you desperately need to use murex natively on Windows, download the appropriate per-compiled binary from the downloads page
• When used in WSL, any Windows mounts included in $PATH will now be subject to the same inclusion rules as Windows file systems when murex is used in Windows. This means only executables (ie files with .exe, amongst other, extensions are present). This has dramatically cleaned up autocomplete suggestions when running murex in WSL.
• New autocomplete directive: FileRegexp. Use this if you want to return only files where the file name matches a defined pattern (eg only .txt files: \.txt$)
• HTTP client (as used by get, getfile, post, open, etc) supports additional logic with dynamically guessing the correct data type to assign to STDOUT where the Content-Type hasn't been mapped. Any Content-Type with a +json suffix will be assumed to be json data type
• getfile output has been revamped. If STDOUT is a TTY then you'll now see a progress bar
• .tfstate (Terraform state files) have been added to supported JSON extensions
• ~/.ssh/config hosts have been added to SSH (et al)'s autocomplete suggestions
• murex-package: reload now only reloads packages, not .murex_preload nor .murex_profile`
• website tweaks: updated INSTALL doc, CSS fixes
• debug message added to warn when fork FIDs are being killed

Non-user facing changes (internal changes to the murex code base):

• Additional tests
• Older function definitions replaced with the newer (correct) way to define functions and methods. This allows for greater discovery in the interactive terminal
• Root unit tests finish quicker

Bug fixes:

• HTTP client (as used by get, getfile, post, open, etc) wasn't breaking down the Content-Type header correctly. This only affected unknown MIME types where murex had to guess the correct data type to assign to STDOUT. This is an edge case bug.
• while, when used without a separate parameter for the conditional logic, was buffering the STDOUT for each loop and outputting it after that iteration had ended. while now uses a "tee" stdio writer which means it can stream STDOUT while the iteration is in progress

Features:

• alter now supports --sum where structures are merged and numeric values are added together

• New builtin count. This has deprecated len however len will stick around as an alias for backwards compatibility

• New operators added to let: +=, -=, /=, *=

• New builtin addheading for adding headings to lists

• Compiled profile will now always execute even if murex flags set to ignore the modules/user profile. This is so that aliases like len get set and thus murex can still move forward with changes to builtins but without breaking backwards compatibility

• autocomplete now passes ISMETHOD variable to dynamic completions so those dynamic completions are aware if a command requesting auto-completion suggestions is being invoked as a method (mid-pipeline) or function (start of a pipeline)

• Index, [, now supports inlining element, [[, lookups. eg [ a b [/map/c] ]

• Dynamic auto-completions that include @IncFiles or @IncDirs will now automatically append files and/or directories to their auto-completion suggestions

• New autocomplete directives: IncExeAll (like IncExePath but includes builtins, functions, aliases), IncManPage (include results from the man page parser -- usually suppressed when autocomplete config is set)

• Disabled 'Loading default profile' message -- this was always pretty redundant but now that the compiled profile is being loaded all the time (eg murex -c 'command' or when called in a shebang), it's also now ugly too

• select now supports passing a file in the FROM syntax. eg select * FROM ./example.csv. The caveat here is this breaks currently auto-complete on column names

• select now supports multiple tables using either named pipes (eg select * FROM <table1>, <table2>) or variables (eg select * FROM \$table1, \$table2) passed in the FROM syntax. Variables should be escaped and you cannot mix and match between named pipes, file names nor variables. You can use any number of tables from 1 to 2^63-1 (64bit systems) or 1 to 2^31-1 (32bit systems). Which should be more than enough ;)

• config option for select to define default output data type where multiple tables are imported

• Lots of new and updated documentation!

Non-user facing changes (internal changes to the murex code base):

• open functions can now be called by other functions to take advantage of auto-typing and auto gunzip etc.

• tmp.Close() should return err. This isn't a bug but it might catch future bugs

• LazyLogging created to speed up writing tests against data structures

• utils/List package created to handle list / array / map functions. Also makes testing more complex routines easier

Bug fixes:

• Regression bug fixed where prepend was invoking append

• streams.ReadCloser not setting context

• parameters.StringArray() should copy values instead of a pointer to ensure the underlying parameters are immutable