0.14.0

Welcome to the 0.14.0 release of Hoarder! I want to welcome our two new contributors @kamtschatka who added a bunch of quality of life improvements and @lilacpixel who fixed a bunch of cosmetic issues in Hoarder! I also want to thank @Collectathon who published Hoarder's template on unraid which was a very popular request from unraid users!

Note: You probably want to read the Upgrading section for that release. No breaking changes, but some recommendations are mentioned.

Announcements

• Hoarder is now available on the unraid store thanks to @Collectathon! The docs now include an unraid installation section here.
• The iOS app has finally been accepted in the app store (link). Thanks a lot to all the beta testers during that time! You can still continue using TestPilot if you want the latest versions fasters, but you can also now switch to the prod version if you want stability.

New Features

• Link deduping is here. When adding a link that you've hoarded before, it won't be re-added and instead you'll be prompted to go to your original link. When adding it from the browser extension, the existing bookmark's tags and lists will be shown allowing you to alter them if needed.
  • Note: Unless you're on the latest mobile version, a deduped link will be silently ignored (without any visual indications). Upgrade to the latest version to get a message telling you that it already exists.
• To quickly merge tags, @kamtschatka added support for dragging and dropping tags on each others to merge them together! To aid with the merging, you can now sort the tags in the "All tags" page by name instead of count.
• You can now batch add multiple links to the bookmark editor card (one per line) and hoarder will import them in one go.
• Hoarder can now take full page screenshots (instead of only the top of the page) thanks to @kamtschatka. This is disabled by default as it consumes a lot of space, but can be enabled by setting CRAWLER_FULL_PAGE_SCREENSHOT=true.
• In the list management dialog, you can now quickly archive a bookmark after adding it to lists. This is useful to those who follow Inbox Zero for their homepage.
• Hoarder is now using openai's new model gpt-4o by default for image tagging. It's faster and 50% cheaper!

Fixes

• Local tagging with Ollama is now more robust thanks to @kamtschatka refactoring the prompt a bit so that model doesn't lose context on what the instructions actually are.
• Bookmarks that were created in the same second (usually as part of a mass import) were sometimes skipped between pages. This is now fixed.
• A lot of cosmetic touches by @lilacpixel:
  • Clicking on images in the bookmark list now opens the bookmark preview popup.
  • Clicking on an image in the preview popup will open it in full quality in a new browser tab.
  • Long titles used to overflow in modals and in the bookmark grid. This is also fixed by @lilacpixel.
  • The bookmark preview popup's input fields now have full width to better show the available tags.
• The tags in the tags editor are now sorted lexicographically.

Screenshots

Link Deduping

New Archive Button in "Manage Lists" Dialog

Multi-URL import

Drag and Drop tag merging

Untitled.mov

Upgrading

• The repo got moved to its own organization, and the docker images unfortunately didn't get moved. which means you might want to update the docker images to the new urls. I'll keep updating the old images for the foreseeable future though. In your docker-compose file, change:
  • ghcr.io/mohamedbassem/hoarder-web to ghcr.io/hoarder-app/hoarder-web
  • ghcr.io/mohamedbassem/hoarder-workers to ghcr.io/hoarder-app/hoarder-workers
• To improve chrome's stability for large pages, add --disable-dev-shm-usage to the existing list of arguments of the chrome container.

To upgrade:

• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

All Commits

• build: Attempt to fix docker permission error - @MohamedBassem in eac54e6
• docker: Update docker images to point to the new repo URLs - @MohamedBassem in 6839289
• docs: Change references from the old repo URL to the new one - @MohamedBassem in 1506ff0
• fix(web): Stop pre-loading all the bookmark lists in the bookmark grid - @MohamedBassem in 8274d07
• feature(web): Add an archive button to list management dialog - @MohamedBassem in f99f4c0
• fix(web): Sort the tags in the TagsEditor of the preview dialog. Fixes #171 - @MohamedBassem in bfcf0a4
• fix: Fix missing bookmarks during pagination if they got created in the same second. Fixes #140 - @MohamedBassem in e8b4775
• feature(web): Allow adding multiple URLs at once #158 (#167) - @kamtschatka in d1ad84b
• ui(web): Set tag input field in preview modal to 100% width to fix overflow issues on select dropdown (#165) - @lilacpixel in cb62db7
• feature(inference): Improve ollama tagging (#162) - @kamtschatka in fdc27d3
• docs: Add documentation for installing hoarder on unraid. Fixes #81 - @MohamedBassem in eb218ce
• fix: Add the missing bookmark URL index migration - @MohamedBassem in c9dc23f
• fix(web): Simplify the logic for tag drag and dropping - @MohamedBassem in e5fd9ee
• feature(web): Improve merging of tags by simple drag and drop #144 (#154) - @kamtschatka in 1fee129
• build: An empty commit to fix vercel's build - @MohamedBassem in 6eea671
• feature(inference): Upgrade the default vision model to the new gpt-4o-2024-05-13 which is faster and 2x cheaper - @MohamedBassem in 168aed0
• feature(web): Clicking an image in the preview modal will open it in a new tab (#159) - @lilacpixel in c0aa6d5
• ui(web): Fixed overflow issues for very long titles (#152) - @lilacpixel in 747efa5
• feature(crawler): Allow connecting to browser's websocket address and launching the browser on demand. This enables support for browserless - @MohamedBassem in 39025a8
• feature(worker): Allow configuring redis with a password - @MohamedBassem in f64a5f3
• fix(crawler): Add --disable-dev-shm-usage to chrome container - @MohamedBassem in 9b5ef3a
• feature(mobile): Show server address in the settings - @MohamedBassem in 8521580
• ui(mobile): Add horizontal margins to the bookmark cards - @MohamedBassem in e6c9ef4
• tests: Add tests for bookmark deduplication - @MohamedBassem in ecfcba5
• feature: Take full page screenshots #143 (#148) - @kamtschatka in d33be14
• feature: Clicking preview for image assets brings up the details modal (#147) - @lilacpixel in cbc8dde
• Development Instructions are incomplete/not working (fixes #145) (#146) - @kamtschatka in 1e317d9
• landing: Add google and ios badges to the landing page - @MohamedBassem in edf5fdb
• readme: Add the iOS app store link to the readme and docs - @MohamedBassem in 7f3e482
• feature: Dedup links on creation. Fixes #49 - @MohamedBassem in 32b5a02
• build: Add iOS and android builds to github actions - @MohamedBassem in 02ef4bf
• fix(cli): Remove limit from array length when listing bookmarks. Fixes #135 - @MohamedBassem in 2ed134e
• build: Add a workflow to auto publish the CLI to npm - @MohamedBassem in 79777bd
• fix(cli): Allow bookmark listing to fetch all pages instead of only the first one. Fixes #135 - @MohamedBassem in 27f45d5
• fix: Allow zod to coerce the crawler timeout from strings. Fixes #134 - @MohamedBassem in 81e49fa

0.13.1

This is a dummy hotfix release for 0.13.0 just to fix the missing docker images for linux/arm64. No changes in the app itself.

0.13.0

Welcome to the 0.13.0 release of Hoarder! This release is packed with a lot of new features to better organize your bookmarks. This includes: Nested lists, tag merging, a new suggested cleanups page, and more! Crawler also now takes screenshots and downloads the banner images locally. The CLI is now more powerful giving you ability to do mass modifications (e.g. archiving/adding to lists) until the mutli-select feature is developed.

Note: You probably want to read the Upgrading section for that release. No breaking changes, but some nice recommendations are mentioned.

Announcements

• The android app has been finally released on the play store (link)! Thanks a lot to all the beta testers during that time! Go check it out (and maybe give it a review while you're there 😉). The iOS app is still stuck in apple's review queue unfortunately, so you still need to use the beta version for now.
• The new chrome/firefox extensions allow you to quickly add tags and lists to your newly saved bookmarks!

New Features

• Nested lists are here! When you're creating a new list, you can choose a parent for that list and it'll be nested under it. The All lists page has been revamped to support showing the list tree.
• Merging tags is now a thing.
• A new Cleanups page that will suggest to you ways to cleanup your collection. As a start, it now detects potentially duplicate tags and suggests merging them. Currently, the algorithm is very simple and will detect duplicates if they have different cases (e.g. Chess and chess), and also tags that differ by at most one character (e.g. web-devopment and Web development). The inference job now is also fixed to avoid creating those duplicates in the first place.
  • In the future, tag duplication detection will be able to suggest merging semantically equivalent tags.
• The crawler now by default downloads the banner images. This is to protect against bookmarked links going away, but also to get around rate limits when you have multiple bookmarks from the same website (e.g. github). This will result into a larger storage space utilization. If you don't care about that, you can disable by setting CRAWLER_DOWNLOAD_BANNER_IMAGE=false.
• The crawler also will now take screenshots of the website it crawls and allows you to see them in the preview page. The screenshots will also be used as a fallback banner for links that don't have an image. Again, this will use some extra storage, so if you don't want screenshots, you can disable them with CRAWLER_STORE_SCREENSHOT=false.
• You can now edit lists to change its name or update its parent.
• You can now see (and manage) all the bookmarks a list is in from the new Manage Bookmarks drop down option.
• You can now delete tags from the All tags page. The All tags page will also show you the "unused" tags and gives you a quick way to mass delete them.
• The CLI got a lot of new commands. You can now update bookmarks from the CLI (e.g. to archive or favourite them), and you can add or remove bookmarks from lists. You can have a look on the updates docs page here. Also, you can now get the CLI directly from NPM using npm install -g @hoarderapp/cli. The CLI is very useful for now to do mass actions on your bookmark while we develop the multi-select feature #84.

Fixes

• When you preview a bookmark (using the maximize button), the address bar of the browser will now show a unique link for that bookmark.
• The crawler now can correctly find the images in amazon links.
• The search indexing worker was in some rare cases incorrectly reporting success while it actually failed. This is now fixed.

Screenshots

The new Cleanups page

Viewing and mass deleting unused tags

Nested lists

The new extension capabilities

Quick tag deletion

Viewing the screenshots

Upgrading

• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

After Ugprading:

• Now that we're taking screenshots for pages, you'll want to tell chrome to hide scrollbars by adding --hide-scrollbars to the command section under the chrome container. Check the update docker compose (here) for reference on how to do it.
• Apparently, meillisearch collects analytics by default unless you opt out. Add MEILI_NO_ANALYTICS=true in the environment section of the meillisearch container if you don't want that. Check the update docker compose (here) for reference on how to do it. Thanks for the report and fix @vivekmiyani!
• The release introduced the ability to download banner images and take screenshots from webpages. Those will be applied only on newly crawled pages. if you want to do the same for your old bookmarks, you can re-crawl all the links from the admin panel by pressing on the Recrawl All Links (Without Inference). Make sure you click the (Without inference) button otherwise you'll be re-running automatic tagging which can be expensive.

All commits

• docker: Attempt to improve build times by changing cache backend by @MohamedBassem in #5d805e2
• fix(inference): Attempt to reuse existing identical tags by @MohamedBassem in #b4c7de2
• feature: Allow users to delete all unused tags in one go by @MohamedBassem in #7d163f2
• fix: Fix error when merging empty tags by @MohamedBassem in #0b02f94
• fix: Change tag listing API to return unused tags by @MohamedBassem in #5aabbec
• fix: Re-index affected bookmarks when one of their tags gets updated by @MohamedBassem in #4efcac7
• feature(crawler): Allow increasing crawler concurrency and configure storing images and screenshots by @MohamedBassem in #a91aff2
• feature: A new cleanups page to suggest ways to tidy up your bookmarks by @MohamedBassem in #9dace18
• feature(web): Add ability to rename, merge and fast delete tags. Fixes #105 (#125) by @MohamedBassem in #d07f2c9
• docs: Update docs to include the links to the android app by @MohamedBassem in #da6df7c
• fix(web): Fix hydration error in the bookmark preview page by @MohamedBassem in #92cff68
• ui(web): Show bookmark URL when maximizing a bookmark by @MohamedBassem in #ec15d20
• docs: Update the CLI docs to reference the published NPM package by @MohamedBassem in #4804f98
• style(cli): Change all help messages to lowercase by @MohamedBassem in #ac2f78f
• build(cli): Prepare for publishing CLI to npm by @MohamedBassem in #3352a3e
• feature(cli): Add ability to manipulate lists, tags and update bookmarks by @MohamedBassem in #af0cf9c
• fix(crawler): Better extraction for amazon images by @MohamedBassem in #5dac180
• fix(docs): Update the social image from the generic docusaurus image by @MohamedBassem in #ab92969
• feature(web): Change the AddList modal to allow adding and removing bookmarks from lists by @MohamedBassem in #88b92d8
• fix(workers): Increase robustness of search worker and add extra logging. Fixes #118 by @MohamedBassem in #c1baada
• fix(workers): Set a modern user agent and update the default viewport size by @MohamedBassem in #cca81ca
• release(extension): Release version 1.2.0 by @MohamedBassem in #3d1eeef
• feature(extension): Allow adding tags and lists to newly hoarded bookmarks by @MohamedBassem in #77b1aba
• fix(web): Fix missing link card image in list view by @MohamedBassem in #0e26095
• ui(extension): Use shadcn and better dark mode support by @MohamedBassem in #7ddcb5f
• fix(mobile): Add a bigger button to collable the nested list by @MohamedBassem in #5f599f2
• fix: Update docker-compose.yml (#119) by @Rosin1 in #cbdaf56
• docker: Disable meilisearch analytics (#117) by @vivekmiyani in #9b9cdcf
• fix: Ensure that downloaded asset images are from the allowed content types by @MohamedBassem in #12c682b
• fix: Use next/image for serving bookmark images by @MohamedBassem in #e12fe02
• feature: Allow recrawling bookmarks without running inference jobs by @MohamedBassem in #4c589d4
• feature: Download images and screenshots by @MohamedBassem in #4402e6f
• mobile: Bump mobile version to 1.5 by @MohamedBassem in #b4a13ce
• fix(mobile): Fix the color of the collapsible triangle...

0.12.2

Another small hotfix for the 0.12.0 release.

Fixes:

• Preferred layout was getting lost when the browser is closed and reopened. This is fixed in this release.

Commits:

• fix: Fix user local settings cookie to have max age as by default it's a session cookie. Fix #104 by @MohamedBassem in bb44ebc

0.12.1

This is a small hotfix for the 0.12.0 release.

• fix: Fix slice call in the content truncation logic which was resulting in excessive usage of context tokens. Fixes #94 by @MohamedBassem in #4629dac

The bug fixed in this release resulted into mistakingly truncating the first 1500 words in large contents during tag inference (instead of taking only the first 1500 words). This had two side effects for bookmarks with a lot of content (> 1500 words):

1. Due to the truncation of the first 1500 words, you might have got some sub optimal tags for content with >1500 words.
2. For bookmarks with huge content size, you might have got inference failures due to hitting the context limits of the models you're using. And for the ones that were under the context size, they might have taken longer and used more credits than intended.

If this is a problem for you, you can trigger a re-index for the affected bookmarks.

0.12.0

Welcome to the 0.12.0 release of hoarder! This release is special because we have our first full feature contribution by @AhmadMuj. Welcome to the contributors list @AhmadMuj! If you're interested in helping us develop hoarder, you can join the new #development channel on hoarder's discord.

New Features

• Raspberry Pi support is now here!
• You can now upload pdfs and get them indexed and automatically tagged thanks to @AhmadMuj!
• You can now change the layout of the bookmark list. We're starting with three layouts:
  • Masonry which was the layout used before that release.
  • Grid where every card has a fixed height resulting in a strict grid.
  • List which shows all bookmarks under each other in a list (screenshot attached later in the release notes).
• Previously, you were only allowed to upload items from the homepage. Now you can upload directly in a list, tag or even the favourited/archived list.
• Previously, the admin panel only allowed you to recrawl all bookmarks. @AhmadMuj added support for only crawling the failed ones (and showed some stats about them there as well).
• The default OpenAI image model got upgraded to the new gpt-4-turbo which should be more robust and accurate while costing the same.
• You can now add titles to bookmarks (and edit existing ones). Uploaded assets (images and pdfs) now automatically get their filename as a title. A new bookmark action bar got added to the bookmark preview page.

Changes in Behavior

• Previously, archived items were hidden from almost everywhere except the archived list which made it not very different from just deleting the item. Starting from this release, archived items are only hidden from the homepage but will continue appearing in lists, search, etc. This is more aligned with other bookmark managers (e.g. pocket). With this change, you can now use archiving to achieve Inbox Zero on your homepage by archiving the items that no longer require your attention or articles that you've already read.

Fixes

• Crawling robustness:
  • Default crawling timeout increased from 30s to 60s and default navigation timeout increase from 10s to 30s. Both can now be controlled with the env variables CRAWLER_JOB_TIMEOUT_SEC and CRAWLER_NAVIGATE_TIMEOUT_SEC respectively. You might want to increase them further if you're running hoarder on a low-perf machine or have a slow internet connection.
  • Crawling jobs are now retried 5 times before giving up on them.
  • Crawler error messages are now more descriptive to help debug issues.
• You no longer need to refresh the All tags page when tags are changed.
• When a user is deleted, their on disk assets were not correctly deleted. This is now fixed.
• Fixed error when adding notes with a colon in the first word.

Screenshots of the new features

The new action bar in the preview page along with the new support for editing titles.

The new List view and its toggles.

PDF support

Upgrade

• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

All Commits

• feature: Add title to bookmarks and allow editing them. Fixes #27 by @MohamedBassem in 81e0b28
• fix(web): Fix some JS console errors in the preview page by @MohamedBassem in 5c9acb1
• feature(web): Allow changing the bookmark grid layout (#98) by @MohamedBassem in 4f17ea6
• fix: Differentiate between pending in db and in redis in admin job stats by @MohamedBassem in cf0df0e
• feature: Recrawl failed links from admin UI (#95) by @AhmadMuj in 95cf8f4
• fix: Increase default navigation timeout to 30s, make it configurable and add retries to crawling jobs by @MohamedBassem in 238c296
• feature: Add PDF support (#88) by @AhmadMuj in #be622e5
• feature(inference): Upgrade the default vision model to the new gpt-4-turbo by @MohamedBassem in 2806701
• fix(crawler): Skip validating URLs in metascrapper as it was already being validated. Fixes #22 by @MohamedBassem in a9242a5
• fix: Delete the API key on logout from phone or extension by @MohamedBassem in f696d33
• fix(web): Fix the toggle theme button allowing clicks from anywhere in the dropdown item by @MohamedBassem in 5ab6c33
• fix: Fix type in the useAddBookmarkToList hook by @MohamedBassem in cae543c
• feature(web): Allow uploading directly into lists/tags. Fixes #69 by @MohamedBassem in fe13408
• refactor(web): Move the tooltip provider to the root of the app by @MohamedBassem in 994691b
• feature: Extract hook logic into separate package and add a new action bar in bookmark preview by @MohamedBassem in 79d61be
• fix(workers): Increase default timeout to 60s, make it configurable and improve logging by @MohamedBassem in 4491831
• fix(docker): Remove BUILDPLATFORM from Dockerfile by @MohamedBassem in e0bb1fc
• fix: Refresh the all tags page automatically when a tag is modified by @MohamedBassem in 044659f
• ui(web): Add a tooltip explaining what archived means by @MohamedBassem in 4cf9908
• fix: Only consider text as URL in editor card if it has an http protocol. Fixes #86 by @MohamedBassem in bc6ee2e
• feature: Change archived meaning to only mean removed from homepage by @MohamedBassem in a81c394
• fix: Delete user's assets when the user is deleted by @MohamedBassem in 91b088d
• build: Increment android app versionCode by @MohamedBassem in 8af95fe

0.11.1

This is a small hotfix release for v0.11.0.

5ef2efe fix: Incorrect counts for tags in the AllTags page. Fixes #78

0.11.0

Welcome to the v0.11.0 release of Hoarder. This packs a bunch of new stuff and fixes.

New Features

• There's now a CLI tool to enable mass manipulations for your bookmarks. You can use it, for example, to mass import bookmarks from your previous tool. Check its docs here, and you can check the initial version of importing bookmarks docs here.
• You can now attach notes to your bookmarks.
• You can now delete tags in the tag view page.
• The All Tags page now differentiates between manually attached tags and AI attached tags.
• You can specify the language of the auto generated tags with INFERENCE_LANG=<lang> env variable. It currently defaults to english. Thanks @COOHU-Kr for the suggestion!
• The web UI now allows uploading multiple images at the same time. Webp support also got added.
• Clicking ⌘ + E or (ctrl + E) will now focus on the new bookmark editor. Thanks @chanunnaki for the idea!
• Current server version is now shown in the admin panel. It'll also check github API to know if there are new version. You can disable the new version check with DISABLE_NEW_RELEASE_CHECK=true.
• In case you missed it, the android app is almost there (under google's mercy). If you can't wait, you can download the APK from this issue

Fixes

• Search results are now sorted by relevance.
• In some rare instances, the crawler was getting stuck parsing some pages preventing any other links from getting crawled. There's now a 30s timeout to solve this problem.
• Fix the sharp is missing error in the web container logs.
• When logging out, people were getting redirected to localhost:3000 by mistake. Add NEXTAUTH_URL=<server_url> to your env file to fix that. The installation docs has been updated.

Upgrade

• [Optional but hightly recommended] The initial docker compose was using an old chrome version (v100). Bump the chrome container version to v123 by changing the chrome container image:
  • from: gcr.io/zenika-hub/alpine-chrome:100
  • to: gcr.io/zenika-hub/alpine-chrome:123
• If you're using HOARDER_VERSION=release, run docker compose pull && docker compose up -d.
• If you're pinning it to a specific version, upgrade the version and then run docker compose pull && docker compose up -d.

Some screenshots from the release

The new All Tags page

Attaching notes to bookmarks

Server version in admin panel

The new CLI

0.10.1

Mostly a lot of minor fixes:

• Fix: Attempt to increase the reliability of the ollama inference by working around some of the errors in the ollama package.
• Fix: There was a limit of 2k chars on notes. This is no longer there.
• Fix: When a bookmark is deleted in the search page, it wasn't getting removed from the grid. This is now fixed.
• feature: In the editor card, you can now save the bookmark by pressing Ctrl + Enter as well as the Cmd + Enter. This should be more intuitive to windows users.
• feature: You can now upload webp images.
• feature: Maximum asset size was 4MB. Now it's configurable with the MAX_ASSET_SIZE_MB env variable. This variable still defaults to 4MB.
• ui: Red destructive color looks slightly better now in dark mode.

0.10.0

Main features:

• Local LLM support via ollama. Check the configuration docs for how to set it up.

Other minor fixes:

• Drop CORS restrictions on the API because it is actually intended to be used cross origins).
• Fix the "Favourites" typo in the sidebar.
• Allow specifying a specific redis database index to use instead of the default 0 one.
• Document the DISABLE_SIGNUPS flag.