Help support quality software by contributing via Patreon or Github Sponsors.

v3.0.2 (2024-01-15)

This is a another patch release with some necessary fixes. The most important fixes:

• [sort] a better fix for maintaining sort ordering on sheet copies #2190 #2254

• [cmdpalette] only first 10 suggestions should have shortcut keys #2242

• [input] draw help sidebar on top of sheet/updater #2241

• [sidebar] grab user's keystroke of sidebar-toggle for helpstring #2250

• [tests] add "test" extras for installing PyPI packages needed to run tests

• [cli] add --guides option alias to open the guide index

• [shell] clean up DirSheet sidebar

Please see the CHANGELOG for the full list of changes.

[Help support quality software by contributing via Patreon or Github Sponsors.]

v3.0.1 (2024-01-07)

This is a patch release with some necessary fixes. The most important fixes (and the reasons a patch release was triggered):

• [dup-selected] dup-selected should unselect all rows in copied sheet #2225
• [cmdpalette] Enter executes first row, if partially typed out #2219
• [expr] fix KeyError crash with invalid inputs in expr for Python 3.12 #2179
• [test] update unit tests to use packaged sample.tsv and benchmark.csv #2218

Other fixes and minor improvements:

• [color] add options.color_longname to use instead of color_keystrokes for command longnames #2219
• [cmdpalette] add sidebar for longname and aggregator palette #2219
• [guide] add show-command-info to display command info for a keystroke #2228
• [test] add assert-expr and assert-expr-row to evaluate Python expressions, and assert result is truthy

Please see the CHANGELOG for the full list of changes.

v3.0 (2023-12-30)

This is a major release.

There have been several major improvements (sidebar, help panels, command palette), and a great many smaller improvements and bugfixes: over 1600 commits in the year since v2.11.

To put it to a word, 1.x was a library (apps welcome), 2.x was a platform (plugins welcome), and 3.x will be more like a product (apps and plugins included).

Highlights

• The sidebar shows status messages as well as context-dependent help panels.
• Help and guides in more and more places: try vd -P open-guide-index.
• Command palette: Press Space and fuzzy search for commands to execute.
• Themes: Try vd --theme=light or vd --theme=asciimono.
• The kitchen sink: most third-party plugins and loaders (including vdplus) are released with VisiData in the same package.

Upgrading from 2.x to 3.0

Most things should just work.

• Run vd2to3.vdx to convert the internal list of macros from tsv to jsonl.
• Rename {options.input_history}.jsonl to input_history.jsonl (the option is deprecated and the name is no longer configurable).

There have been a few API changes that might require updates to plugins, or code in e.g. .visidatarc.

Changelog

Please see the CHANGELOG for the full list of changes.

@QuLogic kindly let us know that there was an API change in Python 3.12's Path module. This needed a shipped patch of VisiData to address, and so we made a callout to the community for what else could be snuck into this patch.

What resulted is a tidy little collection of fixes. Stand-out stars are @daviewales and @midichef for their user-experience improvements. @daviewales has been improving the feel of the chooser, the output of the fixed-width saver and the appearance of the Frequency Table, while @midichef has been tweaking the graphs and improved the display of json columns.

All together, these contributed to a release that feels better than its predecessor, with a fairly minimal changelog.

As usual, the Changelog contains the full list of changes!

v2.11 (2023-01-15)

I’m happy to announce a new maintenance release 2.11. This will be the last release before some big, exciting changes introduced in 3.0. If no significant issues come up in the next couple of weeks, this will be packaged and released for Debian.

The most significant change is that VisiData 2.11 drops support for Python 3.6 and adds support for Python 3.11.

2.11 also includes several other small improvements and bug fixes, of which I’ll highlight two:

• dup-sheet (") now carries over attributes of columns added by add-column (helpful!)
• We added a new "concat" jointype. This behaves similar to "append", but keeps first sheet type and columns. This isn't the final word on this, but it addresses a particular usecase that seems important to include.

Please see the CHANGELOG for the full list of changes.

Hi data friends!

This release primarily fixes startup issues related to importlib_metadata and automatic loading of plugins (#1529). Other changes:

• added -N/--nothing option to disable loading visidatarc or any addons
• #1431 fix system clipboard on windows (thanks @daviewales)
• #1527 Enter opens file on DirSheet
• #1532 allow vd to be truly idle
• Other bugs fixed: #1541 #1547 #1546 #1452 #1431 #1471

Please see the CHANGELOG for the full list of changes.

Learn how to update your VisiData installation here.

v2.10.1 (2022-09-12)

This release has a few minor improvements and several bugfixes. Notably, macros and repeat in 2.10 were not working, and the Guix build was broken also. These are fixed.

Some of the other improvements:

• option to disable histogram with disp_histlen or disp_histogram set to 0 or empty string
• add guard-sheet-off command to unguard sheet (thanks @hanfried for PR #1517)
• add zo (open-cell) command to open file or url from path in current cell
• fixed bugs: #1499 #1494 #1513 #1510 #804
• fewer than 50% of commits (13/28) in this release were authored by Saul

Also there are new decorators, @Extensible.before and @Extensible.after, to make it easier to extend existing functions.

For a complete list of changes, see the [CHANGELOG](https://github.com/saulpw/visidata/blob/develop/CHANGELOG.md#v2101-2022-09-12).

The main new feature in v2.10 is autoloading of installed modules with a visidata.plugins entry_point specified in setup.py, e.g. from vdsql:

    entry_points={'visidata.plugins': 'vdsql=vdsql'}

This release also includes a few dozen minor improvements and bugfixes; see the CHANGELOG for more information.

Quick release in the morning after the previous one!

Thanks to @jsvine for letting us know so soon.

urllib3 is now an optional dependency. If you have v2.9 installed, please upgrade to v2.9.1.

This release has about 6 months of development in it, including many bugfixes, improvements, and a few experimental features. Here are the highlights:

• new confirm dialog window (press y to accept; Enter no longer required)
• new command addcol-window bound to w to create a new column with a list of values "around" the current row
• new formatter attribute on column (and options.disp_formatter) to set the format for both display and text saving. Can be generic, python, json (and it's also extensible)
• support for standard paths for config file and support dirs (XDG and otherwise)
• new loaders for Apache Arrow (IPC and streaming) and parquet formats
• browse remote .zip URLs and extract files from them without downloading the whole archive
• by popular demand, we've decided to invert the default scroll wheel direction! If you liked how it was before, you can set options.scroll_incr = 3
• and a couple things to make it easier for Windows users

There's a lot to love in this release of VisiData; the above is only a partial list. See the official CHANGELOG for more details.

This is a pretty small release of mostly bugfixes, which still has a few changes that merit a minor version bump.

The all-stars of this release are @ajkerrigan and @geekscrapy.

@ajkerrigan found, debugged, and fixed a Python 3.10 compatibility bug. Before v2.8, you could see many warnings related to COLOR-1 in the status message.

@geekscrapy contributed 7 PRs for v2.8. Thanks to his PRs, VisiData now gracefully skips blank lines in json files, and will not attempt to load plugins which failed to install.

The ever-vigilant @geekscrapy found a bug with Enter on the helpmenu #1196.

This is a small enough fix for a very important feature.

A bunch of improvements, bugfixes, and a couple of new loaders.

Improvements

• [movement] bind Home/End to go-top/go-bottom (thanks @geekscrapy #1161)
• [api] add vd.urlcache as alias for urlcache global (thanks @geekscrapy for PR #1164)
• [plugins] do not continue installation if main package fails pip install (thanks @geekscrapy for PR #1194)
• [plugins] allow for plugin records without SHA256; warn if absent (thanks @geekscrapy for PR #1183)
• [load_lazy] do not load subsheets, if sheet.options.load_lazy is True (thanks @geekscrapy for PR #1193)
• [save] confirm when save_foo function does not exist and saver fallsback to options.save_filetype (reported by @geekscrapy #1180)
• [save] options.save_filetype default now 'tsv'
• several cosmetic improvements

Loaders

• [lsv] add lsv filetype for simple awk-like records (requested by @fourjay #1179)
• [ods] add odf filetype for Open Document Format spreadsheets
• [xlsx] add extra columns (cellobject, fontcolor, fillcolor) if options.xlsx_meta_columns (default False) (thanks @hoclun-rigsep for PR #1098)
• [sqlite] allow query/insert (no modify/delete yet) for WITHOUT ROWID tables (requested by @stephancb #1111)

Bugfixes

• [savers compression formats] fix corruption when saving to compression formats (#1159)
• fix "ModuleNotFoundError: no module named 'plugins'" error on startup (#1131 #1152)
• [windows] fix issue with Enter key on Windows (reported by @hossam-houssien #1154)
• [draw] fix multiline rows by making height fixed for all rows (reported by @geekscrapy #916)
• [DirSheet] fix bug where fix key column sheets (e.g. DirSheet, SqliteIndexSheet) keycols were not being saved in batchmode (reported by @geekscrapy #1181)
• [async] make sure all threads started on sheet are cancelable (reported by @geekscrapy #1136)
• [AttrDict] fix bug with setting value on nested AttrDict
• [dup-X-deep] fix error with async_deepcopy (thanks @pstuifzand for fix)
• [join] fix 'inconsistent-keys' issue when joining between XlsxSheet with typed columns and CsvSheet with untyped columns (reported by @davidwales #1124)
• [sqlite] handle sqlite column names with spaces (thanks @davidskeck for PR #1157)
• [sqlite] use options.encoding and options.encoding_errors for decoding of sqlite db text (reported by @WesleyAC #1156)
• [xlsx] add handling for EmptyCell instances (thanks @hoclun-rigsep for PR #1121)
• [xlsx] gate sheet name cleaning on options.clean_names (reported by @davidwales #1122)
• [macos] fix bindings for Option+key
• [random-rows] fix import (reported by @geekscrapy #1162)
• [save-selected] better default save filename (reported by @geekscrapy #1180)
• [save] fix bug where saving multiple sheets to a single non-embeddable format did not result in fail (reported by @geekscrapy #1180)
• [slide] fix Shift slide-down and Shift slide-up with arrow keys (reported by @a-y-u-s-h #1137)
• [replay] fix replay where join-sheets operation hangs (reported by @agjohnson #1141)
• [undo] no more KeyError when Undoing modifications (reported by @geekscrapy #1133)
• [unfurl-col] fix unfurl-col on cells containing exceptions (reported by @jsvine #1171)

In preparation for the VisiData workshop at Strangeloop 2021, this patch was released with several minor bugfixes.

Features with fixes include:

• sysopen-row
• fixed-format saver
• DirSheet's copy-files
• sqlite loader's exception handling

On top of this, we have updated the homebrew and personal debian repositories to contain the most recent version of VisiData.

We appreciate everyone who has mentioned VisiData in several threads on HackerNews and reddit, recently. Talking about tools you love in public spaces, means a lot to everyone who dedicates time to work on them.

Happy VisiDating!

v2.6 (2021-09-19)

VisiData v2.6: now with menus!

This release introduces a new[0] hierachical menu system for discovering and executing commands.

Press Ctrl+H to open the help menu, arrows and Enter to move around. Or use the mouse.

Here's a video demo of this new menu system, including a sample workflow within VisiData for finding which commands aren't in the menu (using the mouse almost entirely).

Also in this release are several performance improvements, feature enhancements, and bugfixes. See the CHANGELOG for full details.

[0] Back in v1.1 (2018), VisiData had an experimental one-line menu system, but it was removed in v1.3 (6 months later), as it wasn't feeling right.

v2.5 (2021-07-08)

First off, some non-technical notices:

Since 2017 when VisiData was started, I've created a number of plugins and tools that take advantage of the VisiData "platform".
These have historically been separate from VisiData, but it became a fair amount of mental overhead tracking a dozen different repos when making changes to the VisiData core repo. So a little while ago I migrated some of my older projects, like vsh and vgit, into the saulpw/visidata repo, aiming at a "mono-repo". But they never really got integrated, because they didn't feel right either: some of them were wild experiments, some I don't want to release as open source, some I don't want to provide the level of support for, and some simply don't belong in the visidata repo (like galcon).

So now I'm migrating towards a "duo-repo" approach. The saulpw/visidata repo can remain focused on providing a reliable and flexible platform, and a separate repo, "vdplus", is where I put all my VisiData plugins and extras. (This vdplus repo is private and not open-source, but Patreon subscribers can get access to it and use it if they want.)

Secondly, the core VisiData repo now requires submitters to e-sign a Copyright Assignment Agreement (CAA). The core repo will always be open-source, but as the author I should have the legal authority to release it under a different license if I choose.

Finally, our IRC channel has moved from Freenode to libera.chat; come visit us on the new #visidata!

And now for the actual software changes:

Interface Changes

• quitguard ux improvement

  • [quitguard] confirm quit/reload only if sheet modified (references #955, #844, #483; thanks @jvns, @frosencrantz)
  • not enabled by default yet

• [quit] add Shift+Q/quit-sheet-free to quit and free associated memory (thanks @cwarden)

• New major keybinding!

  • x is now "cut" (copy and delete), like in vim.
  • In v2.3 the delete family was changed to delete without copying to the clipboard (previously it acted like "cut").

• [options] save to foo.visidatarc from OptionsSheet (thanks @njthomas #958)

  • note that you can't save directly to .visidatarc (call it a security feature).

Minor Improvements

• [splitwin] push sheet in empty pane iff splitwin
• [movement] bind Ctrl+Left/Right to go-left/right-page (thanks @davidwales #1002)
• [unfurl] add options.unfurl_empty to include row for empty list/dict (thanks @frosencrantz #898)
• [regex expand] deprecate expand_col_scanrows; standardize on options.default_sample_size (thanks @jsvine)
• [canvas] add options.disp_canvas_charset to change displayed chars (thanks @albert-ying #963)
  • @albert-ying convinced me to make the canvas character set configurable. With options.disp_canvas_charset = ' o', any points on the graph become an o, which is hopefully big enough to see. You can set this to use different characters, just make sure that the first character is a space. This option can also be configured more precisely if you want to vary the character by how many/which points are present (like the braille characters).
• [join] extend join columns now writable (and correct)

Loader-specific Improvements

• [sqlite] rename or drop tables from the sqlite index sheet (edit the name or delete the row and then z Ctrl+S to commit as usual)
• [DirSheet] add y/gy to copy file(s) to given directory
• [zstd] support loading zstd-compressed files (thanks @lxcode #971)

Bugs fixed (34): #758 #895 #896 #898 #946 #952 #955 #958 #959 #961 #962 #963 #964 #966 #969 #971 #974 #975 #979 #982 #983 #987 #991 #1002 #1003 #1010 #1011 #1011 #1012 #1016 #1018 #1029 #1032 #1033

See the CHANGELOG for the comprehensive list of bugfixes and changes.

v2.4 (2021-04-11)

This release, hot on the heels of the 2.3 release last week, is primarily to address some bugs that were found in 2.3.
Originally it was slated to be a patch release (2.3.1), but the removal of options.use_default_colors with the fix for #939 and the semantics of splitwindow felt sufficient to warrant a minor version bump, and so we tossed the few other improvements that were sitting on the develop branch.

Improvements

• [color] remove options.use_default_colors (thanks @lxcode #939)

  • options.color_default can now have both fg and bg
  • other color options which do not specify fg or bg will use the missing component from color_default
  • to use terminal default colors, set options.color_default=""

• [splitwindow] stabilize sheet stack associations

  • Shift+Z pushes 'under sheet' (if any) onto other stack
  • Shift+Z does not swap panes anymore
  • g Tab changes which stack is in the top vs bottom panes
  • options.disp_splitwin_pct cannot be sheet-specific

• [status] show nSelectedRows on rstatus

Bugfixes

• [pyobj-show-hidden] grab visibility lvl from sheet specific option (thanks @frosencrantz #947)
• [splitwin] prevent flickering-on-full-window
• [splitwin] if top sheet quit, keep bottom sheet in bottom pane

Loader specific bugfixes

• [gzip] fix progress bar when opening gzip (thanks @geekscrapy #925)
• [http] fix loading files from url without specifying filetype
• [sqlite] use TABLE_XINFO for hidden/virtual columns (thanks @dotcs #945)
• [sqlite] reload perf: load without pre-counting rows (which required 2nd full table scan)
• [vds] save typed values instead of formatted display values (thanks @frosencrantz #885)
• [xlsx] stringify "header" row values for column names (thanks @davidwales #921)

This release adds a memory sheet and improves on several other features. Major changes are listed below. Please see the CHANGELOG for the full list of changes.

1. Memory sheet, for recording values usable in expressions. View the memory sheet with open-memory (Alt+Shift+M), add current cell with Alt+M, add single aggregated value with z+. Also reference the current clipboard value as clipval. #912

2. The split pane feature has been revamped. Now there is one sheet stack per pane, and sheets do not jump between panes. New sheets open in the current pane. Send the current sheet to the other pane with Shift+Z (same command as invoking splitpane in the first place). Ctrl+^ swaps sheets in the current pane. See the pane column on the Sheets Sheet (pane 0 means "not on any sheet stack", i.e. it's been quit). #894

3. Colors now may have a background color in addition to the fg color and attributes. Add e.g. "on blue" or "on 214" to a color to set the background color. There is still the (curses) limit of 256 active colors shown at one time. (This is one of several changes in 2.3 necessary to use the new darkdraw plugin.) #435

4. minor additions

• new command: jump-first (bound to g Ctrl+^) to push the bottom sheet (used repeatedly will cycle through sheets in the current pane).
• Ctrl+X within Ctrl+X (pyobj-expr) drops into a Python REPL.

5. changes to options and commands

• options.use_default_colors now defaults to True (was False).
• delete commands no longer move deleted values to clipboard #895
• zd and gzd (delete-cells) set the value to options.null_value instead of always using None.

This is a small patch.

There are two bugfixes for setcol-fill (see #884), and a typo fix in the manpage.

Additionally, there is now a packaged visidata.1, so that man visidata will also bring up the VisiData manpage, when the manpage is successfully installed.

VisiData v2.2 has been packaged for Debian, and is slated to be part of the bullseye distribution. I am hoping I can sneak v2.2.1 in there, also.

Happy data-sleuthing!

This release adds new features and changes some existing behavior. Please see the CHANGELOG for the complete list of changes and bugfixes.

1. The interpretation of "global" vs. "sheet-specific" CLI options has changed.

In v2.0 and v2.1, options given on the command-line were "sheet-specific" and only applied to files on the command-line, and -g (--global) had to be specified to make them apply to sheets created after launch. Now, options are "global" by default (affecting all sheets which don't have a sheet-specific override), and -n indicates that subsequent CLI options are sheet-specific. The -g option is now basically a no-op (though it can be given after -n (--nonglobal) to return to the default behavior after specifying some sheet-specific options). This fixes a few things that users (including myself) were consistently misinterpreting or getting frustrated with, and hopefully does not affect anyone adversely.

2. New feature: persistent input history

By popular demand, VisiData can now store user-provided input values so they persist between sessions. This is currently disabled by default, as it may have surprising privacy or security implications in some environments. Enable persistent input history by adding the following to visidatarc:

options.input_history='previnputs'

3. New type: floatlocale

The VisiData float type uses the Python builtin float, which only parses numbers in "international" notation. To parse locale-specific numbers (which use e.g. a decimal comma instead of a decimal point), there is now the floatlocale type, which parses numbers based on the LC_NUMERIC environment variable (which must be set before starting vd). A separate type was created because locale-based parsing was measured as 20x slower than the base float type. The type-floatlocale and type-floatlocale-selected (on ColumnsSheet) commands were added without keybindings; users who need this functionality will probably want to add the following to their visidatarc:

bindkey('%', 'type-floatlocale')
bindkey('g%', 'type-floatlocale-selected')

4. Option change: Fancy Chooser now disabled by default

The "fancy chooser", a tabular menu with helpstrings for choosing e.g. aggregators and jointypes, is now disabled by default. Choosing an aggregator or jointype now (again) uses the status/input line, and requires entering the value manually. To invoke the fancy chooser, use Ctrl+X. If the fancy chooser is preferred by default, set options.fancy_chooser=True.

5. New format: .vds (VisiData Sheet)

This new single-file format can store typed data and column information from multiple sheets. Use g Ctrl+S (save-all or save-selected) to invoke the multisheet saver, and then specify a filename with a .vds extension. When you load the file into vd, the Column typing and formatting will be the same!

6. New loader (and saver): geojson

Thanks @dracos for PR #876 implementing geojson loading and saving!

7. Several other improvements and bugfixes