Custom plugins for https://github.com/saulpw/visidata/
MIT License
Custom plugins for https://github.com/saulpw/visidata/
Note: This plugin's functionality has been migrated into VisiData core. If you're using VisiData v2.12dev or higher, you no longer need this plugin!
Browse S3 with an interface like a console-based file explorer:
Use glob-matching to focus your search:
Via pip:
pip3 install visidata
There is a comprehensive guide to various installation methods
here. I prefer using
pipx for the main install, and pipx inject
to add plugin
dependencies. Choose whichever install method works best for you, as long as you install VisiData
2.0 or higher.
This plugin can be installed using VisiData's built-in plugin framework, or manually.
vd
Hit <Space>
, type open-plugins
and hit <Enter>
to open the plugins sheet.
Scroll to the vds3
plugin and hit a
to add it.
Wait for installation to complete, then exit VisiData.
Be sure that your ~/.visidatarc
file contains the line:
import plugins
pip3 install s3fs
vds3.py
to a plugins
subdirectory inside your VisiData directory (by default,~/.visidata
):mkdir -p ~/.visidata/plugins
cd path/to/visidata-plugins
cp plugins/vds3.py ~/.visidata/plugins
~/.visidatarc
file:import plugins.vds3
Because this plugin builds on top of s3fs and boto3, it takes advantage of standard AWS CLI configuration settings.
Be sure that the AWS CLI is installed and configured to point to your desired AWS account.
vd 's3://my-bucket/path/to/file.json.gz'
vd 's3://'
vd 's3://my-bucket'
vd 's3://my-bucket/path'
When browsing a bucket, VisiData will behave like a file explorer:
Enter
: Opens the current file or directory as a new sheet. g+Enter
: Open all selected files and
directories.
q
's behavior is unchanged (closes the current sheet), but while browsing a bucket it effectively
becomes the "go up one level" command.
vd 's3://my-bucket/**/*.csv.gz
Since glob-matching can return results from multiple "directories" (S3 prefixes), the glob results sheet will display full object names rather than imitating a navigable directory hierarchy.
Open an S3 path:
vd 's3://my-bucket'
vd 's3://my-bucket/path'
Hit ^V
to toggle support for S3 versioning. When enabled, there will be an additional Latest?
column along with a Version ID
column that is hidden by default. Previous versions can be opened
with Enter
or g+Enter
as usual.
From an S3 directory listings, select multiple objects and use &
to join object contents into a
single sheet. This uses the native VisiData join functionality under the hood, so the available join
types match those described in VisiData's join documentation.
This plugin's behavior can be tweaked with the following options:
vds3_glob
(Default: True
): Enable glob matching for S3 paths. Glob-matching will only kick in
for path names which contain glob patterns (*
, ?
, [
or ]
). However, it's possible to have S3
keys which contain those characters. In those cases, set this to False
to explicitly disable
glob-matching.
vds3_endpoint
(Default: None
): Specify a custom S3 endpoint. This can be useful for local
testing, or for pairing this plugin with S3-compatible endpoints such as MinIO, Backblaze B2, etc.
Note: This sample ~/.visidatarc
snippet defines local S3 endpoints to be used when specific
AWS profiles are active. It assumes that if the moto
or localstack
AWS CLI profiles are active,
you have a local moto server or
localstack S3 service running on a specific port. For
any other AWS profile it falls back to the AWS default endpoint. A block like this can help you
naturally switch between endpoints based on context, rather than requiring command line switches.
profile_endpoint_map = {
'localstack': 'http://localhost:4572',
'moto': 'http://localhost:3000',
}
options.vds3_endpoint = profile_endpoint_map.get(os.environ.get('AWS_PROFILE'))
Options can be configured directly in a ~/.visidatarc
file:
options.vds3_glob = False
Or specified at runtime:
vd --vds3-glob false 's3://my-bucket/file[?].json'
VisiData also supports changing options from the Options sheet inside the application. Jeremy Singer-Vine's tutorial is a helpful reference for that.
This plugin is in a "minimally viable" state - focused on basic S3 read operations. Reading directly from S3 into pandas/dask dataframes is not currently supported, nor is writing to S3.
This plugin adds a pair of column-level convenience functions (from_entries
and to_entries
),
which are similar to their jq
counterparts. As of
this writing, they're most useful for helping to break out tags from AWS API responses. For that
specific case, this custom keybinding is a handy shortcut that composes with VisiData's existing
"expand column" logic:
Sheet.addCommand(
"gz{",
"expand-tags",
"expand_cols_deep(sheet, [sheet.colsByName['Tags'].from_entries()], cursorRow)"
)
In that scenario, assume we have a Tags
column whose data looks like this:
[
{"Key": "Environment", "Value": "production"},
{"Key": "Name", "Value": "my-project"}
]
from_entries()
turns that into this:
{
"Environment": "production",
"Name": "my-project"
}
And VisiData's expand_cols_deep()
function (bound by default to (
) breaks that into
Tags.Environment
and Tags.Name
columns, so each tag becomes a first-class VisiData column.
The kvpairs
plugin is not currently included in VisiData's plugin framework. It can be installed
manually by copying kvpairs.py to your local ~/.visidata/plugins
directory
and including import plugins.kvpairs
in your ~/.visidatarc
file.
VisiData's vfake plugin provides interactive access to some common Faker functionality. The extra bits in vfake_extensions are some personal customizations. They skew heavily toward AWS and are probably most useful as a reference/inspiration for other vfake customizations.
This plugin won't be included in VisiData, and probably shouldn't be added manually as-is either. If
you find any pieces of vfake_extensions.py useful, transplant them
into your own ~/.visidatarc
file or personal plugin collection inside ~/.visidata/plugins
.
VdCustomProvider
could be a helpful reference if you have a need to define your own custom Faker
generator functions for use with vfake.
The autofake
functionality can save a lot of time if you repeatedly generate fake data for values
that follow predictable patterns.
For cells that contain long strings, it can sometimes be easier to pass the value into an external
viewer rather than relying on VisiData's line wrapping. This plugin supports arbitrary external
commands. Unless otherwise specified it relies on the PAGER
environment variable, and defaults to
less
.
Option 1: Include the contents of vpager.py in your ~/.visidatarc
file.
Option 2:
~/.visidata/plugins
directory.import plugins.vpager
to ~/.visidata/plugins/__init__.py
import plugins
to ~/.visidatarc
if it is not there alreadyDefine keybindings or options in your ~/.visidatarc
file. Examples:
# Use spacebar as a custom command prefix, and shuffle other bindings
# around to accommodate that.
BaseSheet.unbindkey(' ')
vd.allPrefixes.append(' ')
vd.bindkeys[':'] = {'BaseSheet': 'exec-longname'}
BaseSheet.bindkey(' ;', 'split-col')
# Tell `open-cell-pager` to invoke `bat` rather than $PAGER
vd.options.vpager_cmd = '/usr/bin/env bat --paging=always'
# Space+Enter: Open a cell's values with the default viewer - `vpager_cmd`
# if it's defined, otherwise $PAGER.
#
# Space+m: Open a cell's value in `glow` for prettier rendered markdown
# in the terminal.
BaseSheet.bindkey(" Enter", "open-cell-pager")
BaseSheet.addCommand(" m", "open-cell-markdown", "cursorCol.pageValue(cursorRow, cmd='glow -p')")
Navigate to a cell with a long value, and use the View
--> Open cell with
menu to open that cell
with an external program.
If you've defined custom keys for the open-cell-*
group of commands, use those instead of menus.
VisiData is a multi-threaded curses application, which can trip up some traditional console-based debugging tools. For example, vanilla pdb is a terrible fit for VisiData - the output is all over the place.
This plugin adds a --debugger
option, initially supporting the following debuggers:
Since the latter two wrap pdb, they will automatically use pdb++ if it's installed.
Install a supported debugger via pip, then run VisiData with the --debugger
option:
vd --debugger pudb sample_data/benchmark.csv
VisiData should immediately trigger a breakpoint. The behavior here varies by debugger:
telnet 127.0.0.1 4444
or nc 127.0.0.1 4444
from anotherOnce the debugger is active, you can start poking around right away or continue execution with c
.
At that point, the debugger will set up an event handler for the interrupt signal. This plugin binds
z^C
(z, Ctrl-C
) as an interrupt keybinding, so that becomes your interactive "break on demand"
shortcut.
^L
binding to redraw the screen is helpful.VisiData's split window feature enables interesting use cases like displaying a data set and frequency table simultaneously, or a master list of records and a child view of details. In that second case, it can be useful to keep focus in the child/detail view while navigating up and down in the parent view. This little plugin sets up keybindings for that.
~/.visidata/plugins
directory.import plugins.parent_navigation
to ~/.visidata/plugins/__init__.py
import plugins
to ~/.visidatarc
if it is not there alreadyThis plugin adds commands but does not define its own keyboard shortcuts for them, since those are
a matter of personal preference and the risk of collisions is high. Instead, you can define your own
shortcuts in ~/.visidatarc
. For reference, mine look like this:
TableSheet.bindkey(ALT + "j", "next-parent-row")
TableSheet.bindkey(ALT + "k", "prev-parent-row")
FreqTableSheet.bindkey(ALT + "j", "zoom-next-freqrow")
FreqTableSheet.bindkey(ALT + "k", "zoom-prev-freqrow")
FreqTableSheet.bindkey(ALT + ENTER, "zoom-cur-freqrow")
So j
and k
move in the current sheet, but with Alt
they move in a parent sheet instead.
Enter
) or cell (z
+ Enter
)next-parent-row
and prev-parent-row
commands to navigate a parent sheet, refreshingJMESPath is a query language for JSON data. This plugin adds VisiData commands to add columns or select rows based on JMESPath expressions.
~/.visidatarc
~/.visidata/plugins
directory.import plugins.vd_jmespath
to ~/.visidata/plugins/__init__.py
import plugins
to ~/.visidatarc
if it is not there alreadyThis plugin adds commands but does not define its own keyboard shortcuts for them, since those are
a matter of personal preference and the risk of collisions is high. Instead, you can define your own
shortcuts in ~/.visidatarc
. For reference, mine look like this:
from visidata import BaseSheet, vd
# Use space as a prefix key rather than to execute a command by name.
vd.bindkeys[':'] = {'BaseSheet': 'exec-longname'}
vd.allPrefixes.append('Space')
# Define JMESPath commands by adding a custom prefix to the built-in
# addcol/select/unselect commands.
BaseSheet.bindkey(' =', 'addcol-jmespath')
BaseSheet.bindkey(' |', 'select-jmespath')
BaseSheet.bindkey(' \\', 'unselect-jmespath')
Inside a sheet containing JSON data:
addcol-jmespath
adds a new column by evaluating a given expression against each rowselect-jmespath
and unselect-jmespath
toggle row selection based on an expressionPlease open an issue for any bugs, questions or feature requests. Pull requests welcome!