Misbehavior when using remote storage with rclone #116
Comments
|
Hi @QtFuta Thank you for reaching out. I am happy that you are trying to run the gallery with an S3 backend since it should run in theory but I never tried to do so. So you are helping to find it out. The error type sounds that exiftool fails to run on the extractor step. Depending on the exif extractir step the previews are generated. So short: No exif data, no preview, wired UI grid. Can you check your logs? Further would you mind to paste your docker compose setting for futher inspection? IMHO the theory with a remote docker volume via fuse is that docker requires a special security flag to work properly. Than the file access is transparent for the application like the gallery. The gallery needs read access to the media directories and read/write access to the config and storage directories. |
|
Hi @xemle ! Thanks for the quick response 😺 Oh, I didn't know there was a need for any special configuration in docker. Not sure why I don't need it since I can access my mounts inside my containers without issue 😅 Here are the errors I've found related to the exif process, so the extractor doesn't like to wait for rclone to fetch the file hehe. Probably I should be just waiting for rclone and the extractor to find the sweet spot where they work together to update the files hahaha. For the sake of completeness, here's my docker compose 😅 version: "3.8"
services:
# api:
# image: xemle/home-gallery-api-server
# environment:
# TensorflowJS backends
# - cpu: slowest and best support
# - wasm: good perfromance for arm64 and amd64 platforms
# - node: best performance on amd64 platform
#- BACKEND=cpu
# - BACKEND=wasm
#- BACKEND=node
gallery:
container_name: gallery
image: xemle/home-gallery:latest
restart: unless-stopped
stdin_open: true
tty: true
ports:
- "127.0.0.1:3000:3000"
environment:
- GALLERY_API_SERVER=http://gallery-api.local.dev
- GALLERY_API_SERVER_CONCURRENT=2 # On low powered devices
- GALLERY_API_SERVER_TIMEOUT=30 # On low powered devices
- GALLERY_OPEN_BROWSER=false
# Use polling for safety of possible network mounts. Try 0 to use inotify via fs.watch
- GALLERY_WATCH_POLL_INTERVAL=300
volumes:
- ./data:/data
- ./gallery.config.yml:/data/config/gallery.config.yml
- /mnt/static_qt/expanse/:/data/expanse/
entrypoint: ['node', '/app/gallery.js']
command: ['run', 'server']Here's the line in fstab to mount the remote And also here's my gallery.config.yml sources:
- dir: ~/expanse
excludeIfPresent: .galleryignore |
You are mounting the S3 on the host not inside the docker container. A fuse mount in the docker container requires special flags.
OK. Got it. This is a similar issue like #111. The timeout of 5sec seems to low for remote sources like your setup with S3. I plan to increase the timeout for exiftool as configuration value. Since I am working on a UI rewrite I will increase the timeout after that. In the meanwhile you can open a PR if you like. Maybe you can play around with the timeout value which would be good for your setting. I am interested also in video files how your setup will behave...
The indexer has no timeout implemented. Therefore the file indexer part works well. The problem is the extractor step with its timeouts. Maybe you can increase some cache strategies on rclone to get the initial setup right. After finalizing your initial setup, please run a full import via Further, the file indexer uses the file matcher strategy sources:
- dir: ~/expanse
excludeIfPresent: .galleryignore
matcher: size-ctime
Thank you. I was hoping to have a S3/rclone mount within the container. If you plan to do that, please provide an example of your settings, that would be great. |
Oh, got it. Then I don't think I want to go to that hazel 😅
This is the first time I know about a timeout when handling a file. That being said... There are some errors which are probably thrown because of I/O 😅
I added several videos, mostly short ones of about a minute, two large ones of 9minutes and 20 minutes and everything was fine 😸
Made the change. What should I watch for to know if something improved? 😅 |
I just moved my gallery to an s3 compatible storage and I mounted it using rclone, so I updated my docker compose to point to the new directory hosted in s3 instead of a local drive and when scanning new files the UI got all weird O.o.
The images before moving the data to s3 are fine, but new files look just as blank rectangles or as nothing at all.
I looked into the html, this is for the empty rectangles
And this is for the emptiness (:astonished:), a single row for just one img element
Reading other issues I found how to read the DB 😅
So, here are the entries for the html above (the first one is for the rectangle, and the second for the emptiness)
I tried the steps in here to try to fix the issue https://docs.home-gallery.org/faq/index.html#i-want-to-start-all-over-how-do-i-reset-the-gallery
But there was no difference.
There was no mention of deleting
database.dbso I didn't, looking at the file and seeing the records above maybe I should have deleted it.could I just manually remove the records from the file to process them again?
I don't know what is the initial issue tho, so maybe they'll end up broken again 🤔
All new files I've tried end up like the emptiness.
Any idea what could be the issue? 😓
The text was updated successfully, but these errors were encountered: