docspell

🎉 Changelog • Documentation • Milestone 🎉

• French translation (#1440): The UI is now available in French! Thanks to @jgirardet.
• Link Items (#528): Link items together directly (without using custom fields) and see all related items quickly on the search page.
• Import mails recursively (#1260): The scanmailbox task can now go through folders recursively. Thanks to @Moerfi666 and @seijikun.
• Fulltext search via PostgreSQL (#424): Another backend for full-text search was added that is backed by PostgreSQL's text search feature. You can now choose between SOLR and PostgreSQL for fulltext search.
• More file backends (#1379): Next to storing the files in the database, you can now choose to store them in a S3 compatible storage or use the filesystem directly.
• Flat zip upload (#1366): An option has been added to the upload metadata that will unpack zip files and process each entry separately (instead of treating a zip file as a single item).
• Fix to upload large files (#1339): Uploaded files are not loaded entirely into memory allowing to upload large files.
• Fix previously published db migration (#1454, #1436): Unfortunately, there was an db migration in the last release that caused problems on some installations. A fix for this migration has been added.
• Fix build to run on windows (#1449, #1461): Building docspell on windows has been improved by @arittner.

For some of the new features, you need the new version of dsc.

🎉 Changelog • Documentation • Milestone 🎉

• Refactor file identifiers, as preparation for different file storage backends (#1380)
  • PLEASE NOTE this release contains a DB migration that changes all file identifiers in the database. It is really very much recommended to do a backup of the database before updating!
• Allow background tasks to return results that are published via events (#1362)
• Users can specify a time zone in user settings (#1062)
• Improve item selection (#1422)
  • item selection is now stored globally and kept until you explicitly clear the selection
  • items can be selected in detail view and the search menu shows an entry to quickly see all items currently selected
• Change logging format and backend
  • The logging format has changed again from logfmt to the "classic" one
  • the configuration allows to use logfmt, json or two "classic" formats
  • logback has been removed! If you used a custom logback.xml before, I'm sorry this is not possible anymore. There are two settings in the config file for now to control logging: logging.minimum-level and logging.format.
• Pdf preview not updated (#1210)
  • some browsers (e.g. safari and some mobile browsers) don't update the ui when the src attribute of the embed element changes
  • for the "fallback view", an iframe is now used - thus for mobile it should work now. Safari on desktop would require to change the respective setting
• Several ui improvements
  • More cards per row for large screens (#1401)
  • more space below direction dropdown (#1400)
  • fix input field save-while-typing (#1340, #1299)
  • improves detail view on small screens (#1420)
  • sort tags by group and then name (#1396)
  • fix bug presenting an empty tag category in search menu (#1402)
  • wrap long text in dashboard tables (#1357)
  • typo fixes (#1387, #1433)
• Remove tools package from release (#1421)

🎉 Changelog • Documentation • Milestone 🎉

There is now a dashboard page and notification and channels can be maintained separately.

• Dashboards: There is a new starting page showing a dashboard. Dashboards can be customized (#1294)
• UI-settings can be specified per collective and then be overriden by user settings (#838)
• Managing notifications channels: Instead of giving channel data with each notification hook, they can now be managed separately. This is more convenient, because they can be reused for different notification hooks and periodic queries. Notifications can be associated to multiple channels. (#1293)
  • Please note that some (small) manual effort is recommended when upgrading: The channel data from all your current notifications is copied into a separate data set. This can create duplicates in the channels if you had notifications with equal channel data. In order to remove these duplicates, first change all notifications to a known channel and afterwards you can delete the obsolete ones.
• Replace bundled pdf viewer with pdfjs default viewer. (#1304)
• Fixes the query used in a periodic query, that had returned trashed items (#1323)
• Fixes UI bugs where some data was not initialized in the view (#1317, #1324)
• Fix sorting tags in search menu (#1318)
• Reworked website using tailwindcss

🎉 Changelog • Documentation • Milestone 🎉

This release includes bookmarking queries as the main feature.

• Bookmark queries and use it in searches. (#1175) Also the periodic query form is updated to allow using a bookmarks.
• Customize message for periodic queries (#1258). You can now add your own sentence to the beginning of the message notifying you for a list of items.
• Allow to configure a priority for Gotify notifications (#1277)
• Improve handling tags (#960). When choosing a tag by the dropdown the catgory is shown next to a tag and the options can be constrainted by clicking a category bubble.
• Fix timezone in docker image (#1234)
• Fix adding high-dpi images that would cause an out-of-memory error when generating previews (#1183).
• Fix tearing down and initializing pubsub table to allow changing app ids (#1251)
• Fix switching between tile and list view when "full width" preview is enabled (#1261)
• Disable "group by month" if there are no groups (#1255)
• Fix deleting periodic queries (button was not working) (#1257)

🎁 🎄 🎁 🎅🏼 🎄 🩹 Changelog 1 Changelog 2 • Documentation • Milestone 🩹 🎁 🎄 🎁 🎅🏼 🎄

The last last release this year is a bug fix release for #1229 😵‍💫

Here the changelog for 0.30.0:

• Add a list view for items, allowing to toggle between tile and list view (#526)
• Generic notification system: It is a start. A few events are available with the idea to add more when needed. Get notified about certain events via gotify, matrix or e-mail. A generic periodic query has been added (in addition to the notify-due-items task). (#848, #1174)
• Update stanford-corenlp and add Spanish to the supported language set for NLP and Hungarian to the standard set of languages.
• Fix to update job execution count when a job is canceled (#1182)
• Change the log format to logfmt and remove all ansi color sequences

🎁 🎄 🎁 🎅🏼 🎄 Changelog • Documentation • Milestone 🎁 🎄 🎁 🎅🏼 🎄

The last release this year comes with a nice notification component and an item list view.

• Add a list view for items, allowing to toggle between tile and list view (#526)
• Generic notification system: It is a start. A few events are available with the idea to add more when needed. Get notified about certain events via gotify, matrix or e-mail. A generic periodic query has been added (in addition to the notify-due-items task). (#848, #1174)
• Update stanford-corenlp and add Spanish to the supported language set for NLP and Hungarian to the standard set of languages.
• Fix to update job execution count when a job is canceled (#1182)
• Change the log format to logfmt and remove all ansi color sequences

Changelog • Documentation • Milestone

This release contains mainly fixes, a little feature and some changes "under the hood".

• Show number of waiting jobs in the top bar (#1069). This introduces some changes under the hood, for example: while previously the restserver was notifying job executors about new jobs, it will now also receive messages from the job executors. This requires a new setting (see below).
• Hide sidebar by default on mobile (#1169)
• Improve scanmailbox form (#1147)
• Improve input of an e-mail address (#987)
• Fix e-mail import for certain files (#1140)
• FIx uploading files with non-ascii filenames (#991)

Changelog • Documentation • Milestone

You can now share subsets of documents with anyone! Here the summary

• Share items (#446). Allows to create shares, public cryptic links, to a subset of your documents that can be shared with other people (who don't need an account). It is possible to search inside the shared subset. Shares have a lifetime and can be password protected.
• Support encrypted PDFs (#1074). When importing PDF files, the protection layer (usually for signed PDFs) is being removed in order to process it. The config file and collective settings can now define a list of passwords that are being used when trying to decrypt encrypted PDFs.
• Use environment variables to configure Docspell instead of a config file (#1121). This is mainly intended when running via docker or other similar tools. Note that settings that accept list as its values are not yet supported.
• Try to detect the best way to render PDFs (#1099). Mobile browsers need a fallback for rendering PDFs, but desktop browsers can do it much better natively. The user settings allow to decide how to render a PDF or to let docspell detect it.
• Filter possible values in search menu based on current results (#856). This removes options that would only yield empty results from the dropdowns.
• Fix search in documenation (#1120)

Changelog • Documentation • Milestone

This release focuses on features for authentication.

• Allow external authentication providers via OpenID Connect. Now you can integrate Docspell into your SSO solution. Using keycloak, for example (or other such tools) users can be maintained elsewhere, like in an LDAP directory. (#489)
• Adds two-factor authentication using TOTPs. If you don't want to setup an external authentication provider (which is another tool to maintain), you can use the builtin TOTP support to have two-factor authentication. (#762)
• Improvements when querying documents (#1040)
• Changed the underlying code for storing and loading files. This is a preparation to allow different storage backends for files in the future (maybe the filesystem or s3). (#1080)
• The license has changed from GPLv3+ to AGPLv3+ (#1078)
• Fixes a bug in the "check for updates" task that was added in the last release (#1068)
• Reduces the length of the startup command, which makes tools like ps much more readable and allows now to start docspell on Windows (untested, though ;-)) (#1062)
• Fixes merging items, where sent mails were not copied to the target item. (#1055)
• Fixes and improves deleting users. Now all their data is also removed and it is shown what that would be. (#1060)

Changelog • Documentation • Milestone

Hi all! This release comes with some nice features that have been in the queue for a long time. Here is a quick list:

• Add ability to merge items (#414). You can now select multiple items and merge them all into one. The first item in the list is the target item, all others are deleted after a successful merge. The webapp allows to reorder this list, of course.
• Add option to only import attachments of e-mails (#983).
• Improve Manage Data page by sorting the tables (#965, #538)
• Allow wildcard searches in queries using attach.id (#971). Before you would have to specify the complete id. This is inconvenient when using from the command line client.
• Add Hebrew to the document languages (#1027, thans @wallace11). Please note, that the SOLR support is very basic for this language.
• Add a periodic task to check for Docspell updates (#990). Let's you check periodically for new versions of docspell. It uses an existing user account and its mail settings to send an e-mail.
• Show the link to an item and its attachments as a QR code in item details (#836). This might be useful when you want to attach this link to physical devices.
• The search menu highlights the sections that contain active filters (#966)
• Safe deletion of items (#347). When deleting items, they are now marked as deleted and can therefore be restored. A periodic job will really delete them from the database eventually.
• Improves German translation (#985, thanks @monnypython)
• Remove the scripts in tools/ since these are now obsolete. The new command line interface covers these features now. Note that the docker images are also NOT built anymore. The directory still exits and is still a place for scripts and little tooling around docspell.
• Fixes a regression where the browser would not display the pdf (#975)
• Fixes the health checks in the docker setup (#976)
• Fixes an issue with text extraction for Japanese documents where numbers were extracted as special unicode points (#973). This only affects the docker setup, when not using the docker images you need to setup tesseract to use different training data for Japanese.

The dsc tool has also been improved, thanks to @seijikun.

Please see the Changelog for more details, the http api changed and the config file for joex has new stuff in it.

Ths is a fixup release to make fulltext search work again. Sorry for the inconvenience!

Please refer to the 0.25.0 release page.

Changelog • Documentation • Milestone

Here is 25th release of Docspell. This tiime with a few bug fixes and some smaller additions. The main thing in this release is the upgrade of several core libraries that essentially affects every piece of the software. Then, finally, a "real" command line interface is introduced (in a separate repository).

• Introducing a new CLI tool (#345) that replaces all the shell scripts from the tools/ directory! https://github.com/docspell/dsc
• UI changes:
  • year separators are now more prominent (#950)
  • fixes a bug in the item counter in detail view when an item is deleted (#920)
  • German translation improvements (#901)
  • The number of selected files is shown in upload page (#896)
• The created date of an item can now be used in queries (#925, #958)
• Setting tags api has been improved (#955)
• Task for converting pdfs is now behind the admin secret (#949)
• Task for generating preview images is now behind the admin secret (#915)
• Respond with 404 when the source-id is not correct (#931)
• Update of core libraries (#890)
• Add Japanese to the list of document languages. Thanks @wallace11 for helping out (#948, #962)
• Fix setting the folder from metadata when processing a file and allow to specifiy it by name or id (#940)
• Fixes docspell config file in docker-compose setup (#909)
• Fixes selecting the next job in the job executor (#898)
• Fixes a bug that prevents uploading more than one file at once (#938)

Please see the Changelog for more details.

Changelog • Documentation • Milestone

This time a translation of the Web-UI in German is included and the docker build was overhauled. The releases are now build and tested using Java 11.

• Rework Docker setup. Images are now provided for different architectures and have a new home now. The images are now built via a github-action from the official packages of each release. (#635, #643, #840, #687)
• Translation of the UI into German (thanks to @monnypython for proof reading and applying lots of corrections!) (#292, #683, #870)
• Improve migration of SOLR (#604)
  • The information whether solr has been setup, is now stored inside SOLR. This means when upgrading Docspell, all data will be re-indexed.
• Add --exclude and --include options to the consumedir.sh script (#885)
• Improved documenation of the http api (#874)
• Removed unused libraries in the final packages to reduce file size a bit (#841)
• Bug: Searching by tag category was broken when using upper case letters (#849)
• Bug: when adding a boolean custom field, it must be applied immediatly (#842)
• Bug: when entering a space in a dropdown the menu closes (#863)
• Bug: Some scripts didn't work with earlier versions of jq (#851)
• Bug: The source form was broken in that it didn't load the language correctly (#877)
• Bug: Tag category options were wrongly populated when narrowing tags via a search (#880)

Breaking Changes

The docker setup changed significantly. Please see the changelog for more information.

Changelog • Documentation • Milestone

This release enables deleting multiple files at once of an item. It also changes how user settings are stored. Additionally several bugs in the ui and server have been fixed.

• Feature: Central user settings (#565): user settings have been stored in the browser but are now stored at the server. This means that all settings are now shared across all devices.
  The ui provides a migration button: goto user profile -> UI settings. If you have stored settings in the browser but not on the server, a button appears that pushes the settings to the server.
  Note: the button is only there if there are no settings at the server. So if you want to migrate, don't set the theme or click on other things that are persistet in the settings before doing the migration!
• Feature: Delete multiple attachemnts at once, thanks to @stefan-scheidewig (#626): multiple attachments on an item can be deleted with a single click
• Feature: Make consumedir-cleaner run on windows, thanks to @JaCoB1123 (#809)
• Bug: More work externalizing strings (#784, #760): many more strings have been externalized for being translated; also dates are now externalized, too
• Bug: Better anonymous upload page (#758): the upload page for anonymous users shouldn't show a form to provide any metadata
• Bug: Tag category color (#835): the input field to specify colors for tag categories didn't show the category name
• Bug: Search in names (#822): a bug in the webui sent a broken query to the server, making the "search in names" field unusable
• Bug: Fulltext only search broken (#823): the fulltext only search didn't only consult the solr index, but also the database, making it a lot slower and presenting the results not in the order returned by solr.
• Bug: Ui switches to logged in state on auth failure (#814)
• Bug: Broken search summary when tag has no category (#759)

Please see the cangelog for some more details. The milestone page lists all issues for this release.

Changelog • Documentation • Milestone

This release fixes some annoying bugs and prepares the web-ui to be translated into other languages. For actual translating them, I ask for your help. If you are interested, there is a detailed post about how to start with it in the docs.

• Refactor webui to prepare for localisation (#726)
• Add names to user defined tasks for better documentation (#712)
• Fixes some ui bugs:
  • scrollbar position (#722)
  • other minor ui related fixes (#746)
• Removed deprecated api endpoints, fixing #482
• Fixes bug where items are already shown in the ui, although still in processing (#719)
• Switch to github actions for ci (#748)
• Fixes a bug in the new query language (#754)
• Fix counters for categories in the search menu (#755)

Thanks to everyone showing interest in docspell and dedicating time by opening issues, testing and providing ideas!

Please see the cangelog for some more details. The milestone page lists all issues for this release.

Changelog • Documentation • Milestone

The main feature of this release gives a very flexible way of searching for documents, using a query.

• Add a query language to provide a flexible way for searching
  • the search form now is translated into a search query
  • allows to search for items not in a folder or not in a specific folder (#628, #379)
  • Allows for range searches in custom fields (#540)
  • And more! See the documentation
• Add a use attribute to all metadata, to be able to exclude them from suggestions (#659)
• Allow to submit items for reprocessing via the UI (#365)
• Add Latvian language (#679)
• Scrollbars are back! (#681, #677)
• The ds.sh script was changed to inform the processing jobs to also check for duplicates (#653)
• Docker setup now uses again a fixed hostname (#608)
• Moving the unit tests to MUnit (#672)
• Remove the old UI code (#636)
• Fix date extraction for English (#665)
• Fix bug when reading contacts from extracted text (#709)
• Fix bugs when reading mails (#691, #678)
• Fix a bug that wouldn't show an error message when entering bad characters in the register form (#663)
• Fixes a typo in the user settings menu (#654, thanks @ChristianKlass)

Thanks to everyone showing interest in docspell and dedicating time by opening issues, testing and providing ideas!

Please see the cangelog for some more details. The milestone page lists all issues for this release.

Changelog • Documentation • Milestone

This release comes with a completely new UI, including (finally!) a dark theme 🚀 🎊 . In addition many smaller features and bug fixes are included.

• New WebUI including a litght and dark theme.
  • The old ui can still be used, as a fallback. It is required to install a Browser Addon for this to add a request header. Check this for firefox or this for chromium. Then add the request header Docspell-Ui with value 1.
  • Experience on mobile devices is greatly improved
• The following UI issues are only included in the new UI:
  • foldable sections in search and multi-edit menu (#613, #527)
  • show current item in detail view (#369)
  • fixed some ui issues regarding processing logs (#363)
  • scrollbar fix (#600)
• Allow a person to be correspondent, concerning or both (#605)
• Add a short-name field to the organization (#560)
• Add a description field to the equipment (#633)
• Allow to specify a language for a source url (#651). This can be used to define upload urls per document language.
• Trim whitespace for certain fields (#539)
• A different docker entrypoint for the consumedir script was added that supports polling (thanks @JaCoB1123, #603, #624)
• Fix duplicate suggestions (#627)
• Fix reading mails with empty headers (#606)
• Fix suggesting person that doesn't belong to the suggested organization (#625)
• Cleanup registered nodes periodically (#618)

Please see the cangelog for some more details. The milestone page lists all issues for this release.

Changelog • Documentation • Milestone

This release comes with major improvements to the text analysis module. It is now much more configurable, has improved results and can learn tags from all categories. Additionally, more languages for document processing have been added and it's now easier to add more. Please open an issue if want more languages to be included.

• text analysis improvements (#263, #570)
  • docspell can now learn from all your tag categories
  • the detection for correspondents/concerned entities has been improved by using the classifier for this, too - all text analysis steps are now configurable that makes it possible to adapt it better to your data and machine.
  • The docs have been updated with some details here and here.
• more languages (#488)
  • Adds: Spanish, Italian, Portuguese, Czech, Dutch, Danish, Finnish, Norwegian, Swedish, Russian, Romanian
• scan mailbox change (#576)
  • The change from last version (#551) has been moved behind a flag in the "scan mailbox settings". Please review your scan mailbox tasks in your user settings.
  • The scan mailbox settings form view has been organized into tabs, as it grew too large for a single form.
• nix tools package fixed (#584)
  • If you are using docspell tools package for nix, it has now been fixed in that all scripts are available. They are now all prefixed by ds- (except the ds script)
• fix deleting organization (#578)
  • Due to the new relationship of a person to an organization, deleting an organization with references a person was not possible. This is now fixed.
• base url fix (#579)
  • The baseurl setting is optional, but when specified it was required to omit a trailing slash. This is now fixed in that it is always rendered without the trailing slash to the client, no matter what is in the config
• tag category case sensitive search fix (#568)
  • This was a bug introduced by the last release. When tag categories can now be spelled upper- or lower-case. In 0.18.0 you had to spell them lowercase, otherwise the search doesn't work.
• adds a workaround for mails that don't specify their used charset (#591)

Breaking Changes

• The joex configuration changed around text analysis. If you had some custom settings there, please review these wrt the new default config.
• When using the nix package manager: the tools package renamed the scripts to be better distinguishable, since they all end up in $PATH. They are now prefixed by ds-.
• The path of the consumedir script changed in the consumedir docker image
• The settings of the scan-mailbox task has been extended by another flag. If you were relying that all mails (even those excluded by a subject filter) where moved away, you need to check your scan-mailbox task settings.

Please see the cangelog for some more details. The milestone page lists all issues for this release.

Changelog • Documentation • Milestone

This release comes with many UI improvements and features as well as bug fixes. The screencasts on the website have been updated.

• Feature: Results summary and updated tag count (#496, #333)
  • A search summary can be displayed that shows the overall result count and to each custom field with a numeric type (number or money) small statistics like sum, average and max/min values. This is useful when you track your expenses on invoices or receipts.
  • This additional ui element can be enabled/disabled in your ui settings.
  • The result summary is now also used to update the tag counts in the search menu according to the current results.
• Feature: password reset (#376, docs)
  • Adds a new route for admins to reset the password of a user
  • Admin users are those with access to the config file, the endpoint requires to supply a secret from the config file.
  • A bash script is provided for more convenient access.
  • Note this also moves the re-create index endpoint behind the same secret! See below.
• Feature: custom fields clickable (#514)
  • The item detail view allows to click on tags to quickly find all tagged items. This now works for custom fields, too.
• Feature: scroll independently (#541)
  • The search menu can scroll now independent from the main area containing the item cards.
• Improvement: improve attachment selection (#396)
  • When selecting an attachment, it shows its preview to the name instead of the name only
• Improvement: wildcard search for custom date fields (#550)
  • Searching for custom field values allows to use a wildcard * at beginning or end. This is also enabled for date-fields.
• Improvement: joex memory (#509)
  • Joex currently requires a lot of memory to hold the NLP models. After idling for some time, which can be configured and defaults to 15 min, the NLP model cache is cleared. This reduces memory load and makes it possible for the JVM to give it back to the OS.
  • This is supposed to relieve memory consumption when idling only. However, whether it is reclaimed by the OS depends on the JVM and its settings. To observe it early, use the G1GC garbage collector. This is enabled by default for JDK11. So it is recommended to use JDK11 (which is used in the docker images).
• Improvement: allow scaling joex with docker-compose, thanks @bjeanes (#552)
  • This allows to easily start multiple joex containers via docker-compose
• Improvement: allow to connect with gmail via app specific passwords (#520)
  • Imap settings have been extended to be able to specify if a OAuth2 should be used or not.
  • Before, OAuth2 was the default when the server has advertised it. This has been changed now, which means you need to adapt your IMAP settings if you currently use OAuth2
• Fix: provide multiple possible date suggestions for English documents (#561)
• Fix: add missing language files to joex docker image (#525)
• Fix: fix a bug that occurs when processing is restarted (i.e. after a crash) (#530)
• Fix: fix a bug in the ui where the mail connection field was not correctly updated (#524)
• Fix: fix bug when importing mails with an applied filter (#551)

Breaking Changes

• Rest Server config:
  • If you specify the fulltext-search.recreate-key, you need to change your config. Delete it and use the secret now for the new setting admin-endpoint.secret.
• routes
  • The route to drop and recreate the fulltext search index has been moved. It is now at /admin/fts/reIndexAll. The secret must now provided as http header and not in the url.
• collective settings:
  • The imap settings have a new flag which indicates whether OAuth2 auth mechanism should be prefered. This is false by default. If you have used it with OAuth2 (like with gmail) you need either set this flag to true manually or use an application specific password.

Please see the cangelog for some more details. The milestone page lists all issues for this release.

Changelog • Documentation

I shouldn't have said that 0.17.0 would be the last release of this year… An unfortunate bug has made it into the previous release that makes the webapp near unusable. Therefore this release, containing only the fix for #508 (compared to 0.17.0).

I'm sorry for the inconvenience!

Please see the 0.17.0 release page for more info.