hoarder

This is a dummy hotfix release for 0.13.0 just to fix the missing docker images for linux/arm64. No changes in the app itself.

Welcome to the 0.13.0 release of Hoarder! This release is packed with a lot of new features to better organize your bookmarks. This includes: Nested lists, tag merging, a new suggested cleanups page, and more! Crawler also now takes screenshots and downloads the banner images locally. The CLI is now more powerful giving you ability to do mass modifications (e.g. archiving/adding to lists) until the mutli-select feature is developed.

Note: You probably want to read the Upgrading section for that release. No breaking changes, but some nice recommendations are mentioned.

Announcements

• The android app has been finally released on the play store (link)! Thanks a lot to all the beta testers during that time! Go check it out (and maybe give it a review while you're there 😉). The iOS app is still stuck in apple's review queue unfortunately, so you still need to use the beta version for now.
• The new chrome/firefox extensions allow you to quickly add tags and lists to your newly saved bookmarks!

New Features

• Nested lists are here! When you're creating a new list, you can choose a parent for that list and it'll be nested under it. The All lists page has been revamped to support showing the list tree.
• Merging tags is now a thing.
• A new Cleanups page that will suggest to you ways to cleanup your collection. As a start, it now detects potentially duplicate tags and suggests merging them. Currently, the algorithm is very simple and will detect duplicates if they have different cases (e.g. Chess and chess), and also tags that differ by at most one character (e.g. web-devopment and Web development). The inference job now is also fixed to avoid creating those duplicates in the first place.
  • In the future, tag duplication detection will be able to suggest merging semantically equivalent tags.
• The crawler now by default downloads the banner images. This is to protect against bookmarked links going away, but also to get around rate limits when you have multiple bookmarks from the same website (e.g. github). This will result into a larger storage space utilization. If you don't care about that, you can disable by setting CRAWLER_DOWNLOAD_BANNER_IMAGE=false.
• The crawler also will now take screenshots of the website it crawls and allows you to see them in the preview page. The screenshots will also be used as a fallback banner for links that don't have an image. Again, this will use some extra storage, so if you don't want screenshots, you can disable them with CRAWLER_STORE_SCREENSHOT=false.
• You can now edit lists to change its name or update its parent.
• You can now see (and manage) all the bookmarks a list is in from the new Manage Bookmarks drop down option.
• You can now delete tags from the All tags page. The All tags page will also show you the "unused" tags and gives you a quick way to mass delete them.
• The CLI got a lot of new commands. You can now update bookmarks from the CLI (e.g. to archive or favourite them), and you can add or remove bookmarks from lists. You can have a look on the updates docs page here. Also, you can now get the CLI directly from NPM using npm install -g @hoarderapp/cli. The CLI is very useful for now to do mass actions on your bookmark while we develop the multi-select feature https://github.com/MohamedBassem/hoarder-app/issues/84.

Fixes

• When you preview a bookmark (using the maximize button), the address bar of the browser will now show a unique link for that bookmark.
• The crawler now can correctly find the images in amazon links.
• The search indexing worker was in some rare cases incorrectly reporting success while it actually failed. This is now fixed.

Screenshots

The new Cleanups page

Viewing and mass deleting unused tags

Nested lists

The new extension capabilities

Quick tag deletion

Viewing the screenshots

Upgrading

• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

After Ugprading:

• Now that we're taking screenshots for pages, you'll want to tell chrome to hide scrollbars by adding --hide-scrollbars to the command section under the chrome container. Check the update docker compose (here) for reference on how to do it.
• Apparently, meillisearch collects analytics by default unless you opt out. Add MEILI_NO_ANALYTICS=true in the environment section of the meillisearch container if you don't want that. Check the update docker compose (here) for reference on how to do it. Thanks for the report and fix @vivekmiyani!
• The release introduced the ability to download banner images and take screenshots from webpages. Those will be applied only on newly crawled pages. if you want to do the same for your old bookmarks, you can re-crawl all the links from the admin panel by pressing on the Recrawl All Links (Without Inference). Make sure you click the (Without inference) button otherwise you'll be re-running automatic tagging which can be expensive.

All commits

• docker: Attempt to improve build times by changing cache backend by @MohamedBassem in #5d805e2
• fix(inference): Attempt to reuse existing identical tags by @MohamedBassem in #b4c7de2
• feature: Allow users to delete all unused tags in one go by @MohamedBassem in #7d163f2
• fix: Fix error when merging empty tags by @MohamedBassem in #0b02f94
• fix: Change tag listing API to return unused tags by @MohamedBassem in #5aabbec
• fix: Re-index affected bookmarks when one of their tags gets updated by @MohamedBassem in #4efcac7
• feature(crawler): Allow increasing crawler concurrency and configure storing images and screenshots by @MohamedBassem in #a91aff2
• feature: A new cleanups page to suggest ways to tidy up your bookmarks by @MohamedBassem in #9dace18
• feature(web): Add ability to rename, merge and fast delete tags. Fixes #105 (#125) by @MohamedBassem in #d07f2c9
• docs: Update docs to include the links to the android app by @MohamedBassem in #da6df7c
• fix(web): Fix hydration error in the bookmark preview page by @MohamedBassem in #92cff68
• ui(web): Show bookmark URL when maximizing a bookmark by @MohamedBassem in #ec15d20
• docs: Update the CLI docs to reference the published NPM package by @MohamedBassem in #4804f98
• style(cli): Change all help messages to lowercase by @MohamedBassem in #ac2f78f
• build(cli): Prepare for publishing CLI to npm by @MohamedBassem in #3352a3e
• feature(cli): Add ability to manipulate lists, tags and update bookmarks by @MohamedBassem in #af0cf9c
• fix(crawler): Better extraction for amazon images by @MohamedBassem in #5dac180
• fix(docs): Update the social image from the generic docusaurus image by @MohamedBassem in #ab92969
• feature(web): Change the AddList modal to allow adding and removing bookmarks from lists by @MohamedBassem in #88b92d8
• fix(workers): Increase robustness of search worker and add extra logging. Fixes #118 by @MohamedBassem in #c1baada
• fix(workers): Set a modern user agent and update the default viewport size by @MohamedBassem in #cca81ca
• release(extension): Release version 1.2.0 by @MohamedBassem in #3d1eeef
• feature(extension): Allow adding tags and lists to newly hoarded bookmarks by @MohamedBassem in #77b1aba
• fix(web): Fix missing link card image in list view by @MohamedBassem in #0e26095
• ui(extension): Use shadcn and better dark mode support by @MohamedBassem in #7ddcb5f
• fix(mobile): Add a bigger button to collable the nested list by @MohamedBassem in #5f599f2
• fix: Update docker-compose.yml (#119) by @Rosin1 in #cbdaf56
• docker: Disable meilisearch analytics (#117) by @vivekmiyani in #9b9cdcf
• fix: Ensure that downloaded asset images are from the allowed content types by @MohamedBassem in #12c682b
• fix: Use next/image for serving bookmark images by @MohamedBassem in #e12fe02
• feature: Allow recrawling bookmarks without running inference jobs by @MohamedBassem in #4c589d4
• feature: Download images and screenshots by @MohamedBassem in #4402e6f
• mobile: Bump mobile version to 1.5 by @MohamedBassem in #b4a13ce
• fix(mobile): Fix the color of the collapsible triangle to respect dark mode by @MohamedBassem in #0c9a07e
• fix(mobile): Fix flicker on sign in page by @MohamedBassem in #4f9b6c8
• feature(mobile): Add support for nested listed on mobile by @MohamedBassem in #5ae6694
• feature: Nested lists (#110). Fixes #62 by @MohamedBassem in #e0999f7
• refactor: Move the shared types to the shared package by @MohamedBassem in #deba31e
• mobile: Bump mobile version to 1.4 by @MohamedBassem in #81fc4c4
• mobile: Add dark mode support by @MohamedBassem in #c46482c

Another small hotfix for the 0.12.0 release.

Fixes:

• Preferred layout was getting lost when the browser is closed and reopened. This is fixed in this release.

Commits:

• fix: Fix user local settings cookie to have max age as by default it's a session cookie. Fix #104 by @MohamedBassem in bb44ebc

This is a small hotfix for the 0.12.0 release.

• fix: Fix slice call in the content truncation logic which was resulting in excessive usage of context tokens. Fixes #94 by @MohamedBassem in #4629dac

The bug fixed in this release resulted into mistakingly truncating the first 1500 words in large contents during tag inference (instead of taking only the first 1500 words). This had two side effects for bookmarks with a lot of content (> 1500 words):

1. Due to the truncation of the first 1500 words, you might have got some sub optimal tags for content with >1500 words.
2. For bookmarks with huge content size, you might have got inference failures due to hitting the context limits of the models you're using. And for the ones that were under the context size, they might have taken longer and used more credits than intended.

If this is a problem for you, you can trigger a re-index for the affected bookmarks.

Welcome to the 0.12.0 release of hoarder! This release is special because we have our first full feature contribution by @AhmadMuj. Welcome to the contributors list @AhmadMuj! If you're interested in helping us develop hoarder, you can join the new #development channel on hoarder's discord.

New Features

• Raspberry Pi support is now here!
• You can now upload pdfs and get them indexed and automatically tagged thanks to @AhmadMuj!
• You can now change the layout of the bookmark list. We're starting with three layouts:
  • Masonry which was the layout used before that release.
  • Grid where every card has a fixed height resulting in a strict grid.
  • List which shows all bookmarks under each other in a list (screenshot attached later in the release notes).
• Previously, you were only allowed to upload items from the homepage. Now you can upload directly in a list, tag or even the favourited/archived list.
• Previously, the admin panel only allowed you to recrawl all bookmarks. @AhmadMuj added support for only crawling the failed ones (and showed some stats about them there as well).
• The default OpenAI image model got upgraded to the new gpt-4-turbo which should be more robust and accurate while costing the same.
• You can now add titles to bookmarks (and edit existing ones). Uploaded assets (images and pdfs) now automatically get their filename as a title. A new bookmark action bar got added to the bookmark preview page.

Changes in Behavior

• Previously, archived items were hidden from almost everywhere except the archived list which made it not very different from just deleting the item. Starting from this release, archived items are only hidden from the homepage but will continue appearing in lists, search, etc. This is more aligned with other bookmark managers (e.g. pocket). With this change, you can now use archiving to achieve Inbox Zero on your homepage by archiving the items that no longer require your attention or articles that you've already read.

Fixes

• Crawling robustness:
  • Default crawling timeout increased from 30s to 60s and default navigation timeout increase from 10s to 30s. Both can now be controlled with the env variables CRAWLER_JOB_TIMEOUT_SEC and CRAWLER_NAVIGATE_TIMEOUT_SEC respectively. You might want to increase them further if you're running hoarder on a low-perf machine or have a slow internet connection.
  • Crawling jobs are now retried 5 times before giving up on them.
  • Crawler error messages are now more descriptive to help debug issues.
• You no longer need to refresh the All tags page when tags are changed.
• When a user is deleted, their on disk assets were not correctly deleted. This is now fixed.
• Fixed error when adding notes with a colon in the first word.

Screenshots of the new features

The new action bar in the preview page along with the new support for editing titles.

The new List view and its toggles.

PDF support

Upgrade

• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

All Commits

• feature: Add title to bookmarks and allow editing them. Fixes #27 by @MohamedBassem in #81e0b28
• fix(web): Fix some JS console errors in the preview page by @MohamedBassem in #5c9acb1
• feature(web): Allow changing the bookmark grid layout (#98) by @MohamedBassem in #4f17ea6
• fix: Differentiate between pending in db and in redis in admin job stats by @MohamedBassem in #cf0df0e
• feature: Recrawl failed links from admin UI (#95) by @AhmadMuj in #95cf8f4
• fix: Increase default navigation timeout to 30s, make it configurable and add retries to crawling jobs by @MohamedBassem in #238c296
• feature: Add PDF support (#88) by @AhmadMuj in #be622e5
• feature(inference): Upgrade the default vision model to the new gpt-4-turbo by @MohamedBassem in #2806701
• fix(crawler): Skip validating URLs in metascrapper as it was already being validated. Fixes #22 by @MohamedBassem in #a9242a5
• fix: Delete the API key on logout from phone or extension by @MohamedBassem in #f696d33
• fix(web): Fix the toggle theme button allowing clicks from anywhere in the dropdown item by @MohamedBassem in #5ab6c33
• fix: Fix type in the useAddBookmarkToList hook by @MohamedBassem in #cae543c
• feature(web): Allow uploading directly into lists/tags. Fixes #69 by @MohamedBassem in #fe13408
• refactor(web): Move the tooltip provider to the root of the app by @MohamedBassem in #994691b
• feature: Extract hook logic into separate package and add a new action bar in bookmark preview by @MohamedBassem in #79d61be
• fix(workers): Increase default timeout to 60s, make it configurable and improve logging by @MohamedBassem in #4491831
• fix(docker): Remove BUILDPLATFORM from Dockerfile by @MohamedBassem in #e0bb1fc
• fix: Refresh the all tags page automatically when a tag is modified by @MohamedBassem in #044659f
• ui(web): Add a tooltip explaining what archived means by @MohamedBassem in #4cf9908
• fix: Only consider text as URL in editor card if it has an http protocol. Fixes #86 by @MohamedBassem in #bc6ee2e
• feature: Change archived meaning to only mean removed from homepage by @MohamedBassem in #a81c394
• fix: Delete user's assets when the user is deleted by @MohamedBassem in #91b088d
• build: Increment android app versionCode by @MohamedBassem in #8af95fe

This is a small hotfix release for v0.11.0.

5ef2efe fix: Incorrect counts for tags in the AllTags page. Fixes #78

Welcome to the v0.11.0 release of Hoarder. This packs a bunch of new stuff and fixes.

New Features

• There's now a CLI tool to enable mass manipulations for your bookmarks. You can use it, for example, to mass import bookmarks from your previous tool. Check its docs here, and you can check the initial version of importing bookmarks docs here.
• You can now attach notes to your bookmarks.
• You can now delete tags in the tag view page.
• The All Tags page now differentiates between manually attached tags and AI attached tags.
• You can specify the language of the auto generated tags with INFERENCE_LANG=<lang> env variable. It currently defaults to english. Thanks @COOHU-Kr for the suggestion!
• The web UI now allows uploading multiple images at the same time. Webp support also got added.
• Clicking ⌘ + E or (ctrl + E) will now focus on the new bookmark editor. Thanks @chanunnaki for the idea!
• Current server version is now shown in the admin panel. It'll also check github API to know if there are new version. You can disable the new version check with DISABLE_NEW_RELEASE_CHECK=true.
• In case you missed it, the android app is almost there (under google's mercy). If you can't wait, you can download the APK from this issue

Fixes

• Search results are now sorted by relevance.
• In some rare instances, the crawler was getting stuck parsing some pages preventing any other links from getting crawled. There's now a 30s timeout to solve this problem.
• Fix the sharp is missing error in the web container logs.
• When logging out, people were getting redirected to localhost:3000 by mistake. Add NEXTAUTH_URL=<server_url> to your env file to fix that. The installation docs has been updated.

Upgrade

• [Optional but hightly recommended] The initial docker compose was using an old chrome version (v100). Bump the chrome container version to v123 by changing the chrome container image:
  • from: gcr.io/zenika-hub/alpine-chrome:100
  • to: gcr.io/zenika-hub/alpine-chrome:123
• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

Some screenshots from the release

The new All Tags page

Attaching notes to bookmarks

Server version in admin panel

The new CLI

Mostly a lot of minor fixes:

• Fix: Attempt to increase the reliability of the ollama inference by working around some of the errors in the ollama package.
• Fix: There was a limit of 2k chars on notes. This is no longer there.
• Fix: When a bookmark is deleted in the search page, it wasn't getting removed from the grid. This is now fixed.
• feature: In the editor card, you can now save the bookmark by pressing Ctrl + Enter as well as the Cmd + Enter. This should be more intuitive to windows users.
• feature: You can now upload webp images.
• feature: Maximum asset size was 4MB. Now it's configurable with the MAX_ASSET_SIZE_MB env variable. This variable still defaults to 4MB.
• ui: Red destructive color looks slightly better now in dark mode.

Main features:

• Local LLM support via ollama. Check the configuration docs for how to set it up.

Other minor fixes:

• Drop CORS restrictions on the API because it is actually intended to be used cross origins).
• Fix the "Favourites" typo in the sidebar.
• Allow specifying a specific redis database index to use instead of the default 0 one.
• Document the DISABLE_SIGNUPS flag.

It's announcement day 🎉🎉

• The browser now runs in its own container which resulted into a smaller overall image sizes.
• We have a new logo. This release updates the logo everywhere.
• Allow showing demo mode creds in sign in page.
• Minor UI fixes

• web: Add night mode support
• docker: Run chrome in its own container

• A new mobile app got introduced
• Now it's possible to hoard images and get them auto tagged
• The bookmark preview page got revamped
• An inline editor card to replace the nav bar buttons
• Fix flicking in search which makes feel much smoother now. Also reduced the debounce duration for faster reaction time.
• Added missing indicies to the db.
• The bookmark grid is now paginated.
• A new revamped tags editor
• A new landing page at https://hoarder.app
• Add support for removing items from lists.
• We're now publishing both macos and linux docker images

feature: Preview page for the stored content of links
fix: Use puppeteer adblocker to block cookies notices
fix: Teach openai to ignore cookie consent popups
feature: Initial attempt at making the app a PWA
feature: Add an 'All Lists' page
feature: Show a loading indicator when tags are still being fetched
feature: Make the first user an admin automatically

feature: Add support for lists
feature: Add full text search support
feature: Add a server admin page

Mostly minor UI improvements

fix: Prevent openai from inferring tags from the prompt itself
fix: Delete URL on successful bookmark
fix: Fix the look of the grid view and allowing notes to have variable size
fix: Fix phantom links in tag pills
fix: Change the colors of the tag pill
fix: Open links in new tab

Main feature: Add support for storing markdown notes

fix the docker image to run the migrations at startup

feat: Add support for adding/removing tags from an existing bookmark

feature: Add support for adding/removing tags

fix: Tag filtering and sorting
fix: Remove the separator line from the sign in page when no providers are configured