datasette

Documentation improvements, bug fixes and support for SpatiaLite 5.

• The SQL editor can now be resized by dragging a handle. (#1236)
• Fixed a bug with JSON faceting and the __arraycontains filter caused by tables with spaces in their names. (#1239)
• Upgraded httpx dependency. (#1005)
• JSON faceting is now suggested even if a column contains blank strings. (#1246)
• New datasette.add_memory_database() method. (#1247)
• The Response.asgi_send() method is now documented. (#1266)
• The official Datasette Docker image now bundles SpatiaLite version 5. (#1278)
• Fixed a no such table: pragma_database_list bug when running Datasette against SQLite versions prior to SQLite 3.16.0. (#1276)
• HTML lists displayed in table cells are now styled correctly. Thanks, Bob Whitelock. (#1141, #1252)
• Configuration directory mode now correctly serves immutable databases that are listed in inspect-data.json. Thanks Campbell Allen and Frankie Robertson. (#1031, #1229)

Support for cross-database SQL queries and built-in support for serving via HTTPS.

• The new --crossdb command-line option causes Datasette to attach up to ten database files to the same /_memory database connection. This enables cross-database SQL queries, including the ability to use joins and unions to combine data from tables that exist in different database files. See Cross-database queries for details. (#283)
• --ssl-keyfile and --ssl-certfile options can be used to specify a TLS certificate, allowing Datasette to serve traffic over https:// without needing to run it behind a separate proxy. (#1221)
• The /:memory: page has been renamed (and redirected) to /_memory for consistency with the new /_internal database introduced in Datasette 0.54. (#1205)
• Added plugin testing documentation on Using pdb for errors thrown inside Datasette. (#1207)
• The official Datasette Docker image now uses Python 3.7.10, applying the latest security fix for that Python version. (#1235)

• Fixed a bug where ?_search= and ?_sort= parameters were incorrectly duplicated when the filter form on the table page was re-submitted. (#1214)

The two big new features in this release are the _internal SQLite in-memory database storing details of all connected databases and tables, and support for JavaScript modules in plugins and additional scripts.

For additional commentary on this release, see Datasette 0.54, the annotated release notes.

The _internal database

As part of ongoing work to help Datasette handle much larger numbers of connected databases and tables (see Datasette Library) Datasette now maintains an in-memory SQLite database with details of all of the attached databases, tables, columns, indexes and foreign keys. (#1150)

This will support future improvements such as a searchable, paginated homepage of all available tables.

You can explore an example of this database by signing in as root to the latest.datasette.io demo instance and then navigating to latest.datasette.io/_internal.

Plugins can use these tables to introspect attached data in an efficient way. Plugin authors should note that this is not yet considered a stable interface, so any plugins that use this may need to make changes prior to Datasette 1.0 if the _internal table schemas change.

Named in-memory database support

As part of the work building the _internal database, Datasette now supports named in-memory databases that can be shared across multiple connections. This allows plugins to create in-memory databases which will persist data for the lifetime of the Datasette server process. (#1151)

The new memory_name= parameter to the Database class can be used to create named, shared in-memory databases.

JavaScript modules

JavaScript modules were introduced in ECMAScript 2015 and provide native browser support for the import and export keywords.

To use modules, JavaScript needs to be included in <script> tags with a type="module" attribute.

Datasette now has the ability to output <script type="module"> in places where you may wish to take advantage of modules. The extra_js_urls option described in Custom CSS and JavaScript can now be used with modules, and module support is also available for the extra_body_script() plugin hook. (#1186, #1187)

datasette-leaflet-freedraw is the first example of a Datasette plugin that takes advantage of the new support for JavaScript modules. See Drawing shapes on a map to query a SpatiaLite database for more on this plugin.

Code formatting with Black and Prettier

Datasette adopted Black for opinionated Python code formatting in June 2019. Datasette now also embraces Prettier for JavaScript formatting, which like Black is enforced by tests in continuous integration. Instructions for using these two tools can be found in the new section on Code formatting in the contributors documentation. (#1167)

Other changes

• Datasette can now open multiple database files with the same name, e.g. if you run datasette path/to/one.db path/to/other/one.db. (#509)
• datasette publish cloudrun now sets force_https_urls for every deployment, fixing some incorrect http:// links. (#1178)
• Fixed a bug in the example nginx configuration in Running Datasette behind a proxy. (#1091)
• The Datasette Ecosystem documentation page has been reduced in size in favour of the datasette.io tools and plugins directories. (#1182)
• The request object now provides a request.full_path property, which returns the path including any query string. (#1184)
• Better error message for disallowed PRAGMA clauses in SQL queries. (#1185)
• datasette publish heroku now deploys using python-3.8.7.
• New plugin testing documentation on Testing outbound HTTP calls with pytest-httpx. (#1198)
• All ?_* query string parameters passed to the table page are now persisted in hidden form fields, so parameters such as ?_size=10 will be correctly passed to the next page when query filters are changed. (#1194)
• Fixed a bug loading a database file called test-database (1).sqlite. (#1181)

Alpha release. Release notes in progress.

• Improved support for named in-memory databases. (#1151)
• New _internal in-memory database tracking attached databases, tables and columns. (#1150)
• Support for JavaScript modules. (#1186, #1187)

Datasette has an official project website now, at https://datasette.io/. This release mainly updates the documentation to reflect the new site.

• New ?column__arraynotcontains= table filter. (#1132)
• datasette serve has a new --create option, which will create blank database files if they do not already exist rather than exiting with an error. (#1135)
• New ?_header=off option for CSV export which omits the CSV header row, documented here. (#1133)
• "Powered by Datasette" link in the footer now links to https://datasette.io/. (#1138)
• Project news no longer lives in the README - it can now be found at https://datasette.io/news. (#1137)

• Fix for error caused by combining the _searchmode=raw and ?_search_COLUMN parameters. (#1134)

• Show pysqlite3 version on /-/versions, if installed. (#1125)
• Errors output by Datasette (e.g. for invalid SQL queries) now go to stderr, not stdout. (#1131)
• Fix for a startup error on windows caused by unnecessary from os import EX_CANTCREAT - thanks, Abdussamet Koçak. (#1094)

• Fixed bug where static assets would 404 for Datasette installed on ARM Amazon Linux. (#1124)

• Generated columns from SQLite 3.31.0 or higher are now correctly displayed. (#1116)
• Error message if you attempt to open a SpatiaLite database now suggests using --load-extension=spatialite if it detects that the extension is available in a common location. (#1115)
• OPTIONS requests against the /database page no longer raise a 500 error. (#1100)
• Databases larger than 32MB that are published to Cloud Run can now be downloaded. (#749)
• Fix for misaligned cog icon on table and database pages. Thanks, Abdussamet Koçak. (#1121)

• Documentation on Testing plugins now recommends using datasette.client. (#1102)
• Fix bug where compound foreign keys produced broken links. (#1098)
• datasette --load-module=spatialite now also checks for /usr/local/lib/mod_spatialite.so. Thanks, Dan Peterson. (#1114)

This release includes a number of changes relating to an internal rebranding effort: Datasette's configuration mechanism (things like datasette --config default_page_size:10) has been renamed to settings.

• New --setting default_page_size 10 option as a replacement for --config default_page_size:10 (note the lack of a colon). The --config option is deprecated but will continue working until Datasette 1.0. (#992)
• The /-/config introspection page is now /-/settings, and the previous page redirects to the new one. (#1103)
• The config.json file in Configuration directory mode is now called settings.json. (#1104)
• The undocumented datasette.config() internal method has been replaced by a documented .setting(key) method. (#1107)

Also in this release:

• New plugin hook: database_actions(datasette, actor, database), which adds menu items to a new cog menu shown at the top of the database page. (#1077)
• datasette publish cloudrun has a new --apt-get-install option that can be used to install additional Ubuntu packages as part of the deployment. This is useful for deploying the new datasette-ripgrep plugin. (#1110)
• Swept the documentation to remove words that minimize involved difficulty. (#1089)

And some bug fixes:

• Foreign keys linking to rows with blank label columns now display as a hyphen, allowing those links to be clicked. (#1086)
• Fixed bug where row pages could sometimes 500 if the underlying queries exceeded a time limit. (#1088)
• Fixed a bug where the table action menu could appear partially obscured by the edge of the page. (#1084)

• Improvements to the new Binary data documentation page.

A new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy.

New visual design

Datasette is no longer white and grey with blue and purple links! Natalie Downe has been working on a visual refresh, the first iteration of which is included in this release. (#1056)

Plugins can now add links within Datasette

A number of existing Datasette plugins add new pages to the Datasette interface, providig tools for things like uploading CSVs, editing table schemas or configuring full-text search.

Plugins like this can now link to themselves from other parts of Datasette interface. The menu_links(datasette, actor) hook (#1064) lets plugins add links to Datasette's new top-right application menu, and the table_actions(datasette, actor, database, table) hook (#1066) adds links to a new "table actions" menu on the table page.

The demo at latest.datasette.io now includes some example plugins. To see the new table actions menu first sign into that demo as root and then visit the facetable table to see the new cog icon menu at the top of the page.

Binary data

SQLite tables can contain binary data in BLOB columns. Datasette now provides links for users to download this data directly from Datasette, and uses those links to make binary data available from CSV exports. See Binary data for more details. (#1036 and #1034).

URL building

The new datasette.urls family of methods can be used to generate URLs to key pages within the Datasette interface, both within custom templates and Datasette plugins. See Building URLs within plugins for more details. (#904)

Running Datasette behind a proxy

The base_url configuration option is designed to help run Datasette on a specific path behind a proxy - for example if you want to run an instance of Datasette at /my-datasette/ within your existing site's URL hierarchy, proxied behind nginx or Apache.

Support for this configuration option has been greatly improved (#1023), and guidelines for using it are now available in a new documentation section on Running Datasette behind a proxy. (#1027)

Smaller changes

• Wide tables shown within Datasette now scroll horizontally (#998). This is achieved using a new <div class="table-wrapper"> element which may impact the implementation of some plugins (for example this change to datasette-cluster-map).
• New debug-menu permission. (#1068)
• Removed --debug option, which didn't do anything. (#814)
• Link: HTTP header pagination. (#1014)
• x button for clearing filters. (#1016)
• Edit SQL button on canned queries, (#1019)
• --load-extension=spatialite shortcut. (#1028)
• scale-in animation for column action menu. (#1039)
• Option to pass a list of templates to .render_template() is now documented. (#1045)
• New datasette.urls.static_plugins() method. (#1033)
• datasette -o option now opens the most relevant page. (#976)
• datasette --cors option now enables access to /database.db downloads. (#1057)
• Database file downloads now implement cascading permissions, so you can download a database if you have view-database-download permission even if you do not have permission to access the Datasette instance. (#1058)
• New documentation on Designing URLs for your plugin. (#1053)

• New load_template(template, request, datasette) plugin hook. (#1042)
• New debug-menu permission. (#1068)

• New colour scheme and improved visual design, courtesy of Natalie Downe. (#1056)
• scale-in animation for column action menu. (#1039)
• Wide tables now scroll horizontally. (#998)
• Option to pass a list of templates to .render_template() is now documented. (#1045)
• New datasette.urls.static_plugins() method. (#1033)
• BLOB column values can now be downloaded directly from the Datasette UI. (#1036)
• .csv exports now link to direct BLOB downloads. (#1034)
• datasette -o option now opens the most relevant page. (#976)
• datasette --cors option now enables access to /database.db downloads. (#1057)
• Database file downloads now implement cascading permissions, so you can download a database if you have view-database-download permission even if you do not have permission to access the Datasette instance. (#1058)
• New documentation on Designing URLs for your plugin. (#1053)
• New navigation menu plus a menu_links(datasette, actor) plugin hook to customize it. (#1064)
• table_actions(datasette, actor, database, table) plugin hook for the new table actions menu. (#1066)

• Fixed a bunch of issues relating to the base_url setting. (#1023)
• New datasette.urls URL builder for plugins, see Building URLs within plugins. (#904)
• Removed --debug option, which didn't do anything. (#814)
• Link: HTTP header pagination. (#1014)
• x button for clearing filters. (#1016)
• Edit SQL button on canned queries, (#1019)
• --load-extension=spatialite shortcut. (#1028)

• Fixed another bug introduced in 0.50 where column header links on the table page were broken. (#1011)

• Fixed a bug introduced in 0.50 where the export as JSON/CSV links on the table, row and query pages were broken. (#1010)

The key new feature in this release is the column actions menu on the table page (#891). This can be used to sort a column in ascending or descending order, facet data by that column or filter the table to just rows that have a value for that column.

Plugin authors can use the new datasette.client object to make internal HTTP requests from their plugins, allowing them to make use of Datasette's JSON API. (#943)

New Deploying Datasette documentation with guides for deploying Datasette on a Linux server using systemd or to hosting providers that support buildpacks. (#514, #997)

Other improvements in this release:

• Publishing to Google Cloud Run documentation now covers Google Cloud SDK options. Thanks, Geoffrey Hing. (#995)
• New datasette -o option which opens your browser as soon as Datasette starts up. (#970)
• Datasette now sets sqlite3.enable_callback_tracebacks(True) so that errors in custom SQL functions will display tracebacks. (#891)
• Fixed two rendering bugs with column headers in portrait mobile view. (#978, #980)
• New db.table_column_details(table) introspection method for retrieving full details of the columns in a specific table, see Database introspection.
• Fixed a routing bug with custom page wildcard templates. (#996)
• datasette publish heroku now deploys using Python 3.8.6.
• New datasette publish heroku --tar= option. (#969)
• OPTIONS requests against HTML pages no longer return a 500 error. (#1001)
• Datasette now supports Python 3.9.