AutoUpload

A systemd unit to autoupload a directory’s contents to S3-compatible object storage with Rclone or to Immich with the Immich CLI. This is handy for automatically uploading files like photos or sound clips.

Overview

When a file appears in the designated directory, its presence is automatically detected by systemd. When an internet connection is next available, systemd uploads the files to either S3-compatible object storage with Rclone or to Immich with the Immich CLI. The files are then removed from local storage.

Storage

The files are uploaded somewhere, which requires configuration. Files can be uploaded to S3-compatible object storage or to Immich. For S3-compatible object storage, this documentation specifically refers to MinIO. Both my MinIO and Immich setups are documented in my Home Lab Helm repository. I use Tailscale to access these services. Using MinIO is more secure as an API token can be scoped with access that is limited to uploading files. For Immich, it is possible to use the local API token to delete uploaded files or to do even worse if the API token belongs to an admin user.

MinIO

The instructions here are for setting up S3-compatible object storage using a a self-hosted MinIO instance with a dedicated pi-camera bucket for the files. These instructions describe how to create the pi-camera bucket and generate an access key with the required write-access necessary to upload the files with Rclone. The instructions assume that podman is installed.

Create a configuration to access the MinIO server.

podman run \
  --interactive \
  --name minio-client \
  --rm \
  --tty \
  --user $(id -u):$(id -g) \
  --userns keep-id \
  --volume minio-client-config:/.mc:Z \
  quay.io/minio/mc:latest \
  alias set jwillikers https://minio.jwillikers.io
mc: Configuration written to `/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/.mc/share`.
mc: Initialized share uploads `/.mc/share/uploads.json` file.
mc: Initialized share downloads `/.mc/share/downloads.json` file.
Enter Access Key: abcde123
Enter Secret Key:
Added `jwillikers` successfully.

Create a pi-camera bucket in MinIO to store the files.

podman run \
  --interactive \
  --name minio-client \
  --rm \
  --tty \
  --user $(id -u):$(id -g) \
  --userns keep-id \
  --volume minio-client-config:/.mc:Z \
  quay.io/minio/mc:latest \
  mb jwillikers/pi-camera

Place a quota on the pi-camera bucket to prevent uploading too much data.

podman run \
  --interactive \
  --name minio-client \
  --rm \
  --tty \
  --user $(id -u):$(id -g) \
  --userns keep-id \
  --volume minio-client-config:/.mc:Z \
  quay.io/minio/mc:latest \
  mc quota set jwillikers/pi-camera --size 200gi

Generate an access token for the Minio server which uses the pi-camera-minio-policy.json policy. This policy allows only the minimal access necessary for Rclone to upload files to the bucket.

podman run \
  --interactive \
  --name minio-client \
  --rm \
  --tty \
  --user $(id -u):$(id -g) \
  --userns keep-id \
  --volume minio-client-config:/.mc:Z \
  --volume ./pi-camera-minio-policy.json:/pi-camera-minio-policy.json:Z \
  quay.io/minio/mc:latest \
  admin user svcacct add --description "Pi Camera" --name "Pi Camera" --policy "pi-camera-minio-policy.json" jwillikers core
Access Key: XXXXXXXXXXXXXXXXXXXX
Secret Key: ****************************************
Expiration: no-expiry

Immich

Uploading to Immich is pretty straightforward using the Immich CLI utility. Files will be uploaded to a user account. For security, create a dedicated user account to prevent any potential misuse of the API key. Unfortunately, since it isn’t possible to limit the scope of an API key, the key can still be abused to delete photos for this user’s account. With that in mind, there are important limitations with partner sharing at the moment which, is probably how you’ll want to access the pictures from a dedicated account. Right now, facial recognition and favorites won’t work across partner sharing, although these are requested features. It’s also not possible to import shared photos to your account either. The following instructions walkthrough how to create this account using the Immich API. The WebUI can be used to accomplish the same tasks. My Immich server resides at https://immich.jwillikers.io. Replace it with the URL of your Immich instance.

Login as an admin user and obtain an access token. Substitute your user’s username and password.

💡

Precede the following commands that use the sensitive data such as your password and access token with a space to omit them from your shell’s history.
curl --location 'https://immich.jwillikers.io/api/auth/login' \ --header 'Content-Type: application/json' \ --header 'Accept: application/json' \ --data-raw '{ "email": "jordan@jwillikers.com", "password": "password" }'
The successful response from this command includes the access token as the value for the accessToken key.

Create a dedicated user account for the Pi Camera with a quota. The following command creates the Pi Camera user with a quota of 5 GiB. Substitute the API key generated in the previous step for <access_token> and use a good password for the Pi Camera user, not password.

 curl --location 'https://immich.jwillikers.io/api/user' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{
    "email": "pi-camera@jwillikers.com",
    "name": "Pi Camera",
    "password": "password",
    "quotaSizeInBytes": 5368709120
  }'

The response will return the user’s id which will be used in the following command.

Disable the need to change the user’s password. Use the Pi Camera user’s id for id in this command.

 curl --location --request PUT 'https://immich.jwillikers.io/api/user' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data '{
    "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
    "shouldChangePassword": false
  }'

Log out from the admin user’s account.

 curl --location --request POST 'https://immich.jwillikers.io/api/auth/logout' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Accept: application/json'

Now log in as the Pi Camera user.

 curl --location 'https://immich.jwillikers.io/api/auth/login' \
  --header 'Content-Type: application/json' \
  --header 'Accept: application/json' \
  --data-raw '{
    "email": "pi-camera@jwillikers.com",
    "password": "password"
  }'

The successful response from this command includes the access token as the value for the accessToken key.

Find the id for any users you want to add as partners to the Pi Camera account. Obtain all of the users with this command.

 curl --location 'https://immich.jwillikers.io/api/user' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Accept: application/json'

For each desired user, run the following command to add them as a partner. Use the user’s id as the last path in the URL, replacing :id.

 curl --location --request POST 'https://immich.jwillikers.io/api/partner/:id' \
  --header 'Accept: application/json' \
  --header 'Authorization: Bearer <access_token>'

Generate an API key for the Pi Camera user. Getting an API key is documented in the Obtain the API Key section of the Immich CLI page.
curl --location 'https://immich.jwillikers.io/api/api-key' \ --header 'Authorization: Bearer <access_token>' \ --header 'Content-Type: application/json' \ --header 'Accept: application/json' \ --data '{ "name": "Pi Camera Upload" }'
The value for the secret key in the response is the API token that will be used later on the Pi Camera. Don’t lose it or you’ll have to generate another one.

Log out from the Pi Camera user’s account.

 curl --location --request POST 'https://immich.jwillikers.io/api/auth/logout' \
  --header 'Authorization: Bearer <access_token>' \
  --header 'Accept: application/json'

Install

These instructions install the autoupload systemd units and their dependencies. The instructions here are for both Debian and Fedora Atomic.

Install just by following the instructions in the installation section.
Follow the instructions to setup the Tailscale Online Target for the node where files will be uploaded. These instructions also document how to install Tailscale itself. After installing the necessary files, the following command enables the Tailscale Online Target for the meerkat node where my files will be uploaded. The Tailscale Online Target for this specific instance is added as a dependency for the autoupload systemd service.
System
sudo systemctl enable --now tailscale-dispatcher@meerkat.timer
User
systemctl --user enable --now tailscale-dispatcher@meerkat.timer

Clone this project’s repository.

git clone https://github.com/jwillikers/autoupload.git

Change to the project’s root directory.
```
cd autoupload
```
Install everything with just install.
```
just install
```

Immich CLI

Install Immich CLI by running just install-immich-cli.
System
sudo just install-immich-cli
User
just install-immich-cli
Set up the API key for authenticating with the Immich instance. This can be done with the just login command which will run the immich login-key command. The last argument should be the API key generated for the Pi Camera user in the Immich section. Run this command under the same user that will be running the systemd unit.
System
printf '%s' '<the API key>' | sudo podman secret create immich_api_key
User
printf '%s' '<the API key>' | podman secret create immich_api_key -
Enable the update-immich-container.timer systemd timer to automatically pull the latest Immich container image.
System
sudo systemctl enable update-immich-container.timer
User
systemctl --user enable update-immich-container.timer
Modify the autoupload-immich@.container file as needed. Make sure that the line for the appropriate installation method is uncommented and either comment or remove the line for the alternative method. Additionally, the Requires and After should be updated appropriately for the node where the Immich server resides.
System
/etc/containers/systemd/autoupload-immich@.container

link:immich-cli/autoupload-immich@.container[role=include]
User
~/.config/containers/systemd/autoupload-immich@.container

link:immich-cli/autoupload-immich@.container[role=include]
Enable and start the systemd units for the desired approach. There are two approaches to initiating uploads: systemd timers and systemd path units. One or both of these approaches can be used to trigger the systemd unit.
systemd.path
Enable and start the instantiable autoupload-immich@.path systemd unit passing the properly escaped path of the directory to monitor. Use the systemd-escape --path command to escape the directory’s path.
System

sudo systemctl enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).path

User

systemctl --user enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).path
systemd.timer
Enable and start the instantiable autoupload-immich@.timer systemd unit passing the properly escaped path of the directory to monitor. The timer unit files can be modified as desired. By default, they trigger once daily. Use the systemd-escape --path command to escape the directory’s path.
System

sudo systemctl enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).timer

User

systemctl --user enable --now autoupload-immich@$(systemd-escape --path ~/Pictures).timer
When running under a user, make sure to enable linger for that user account.
```
sudo loginctl enable-linger $USER
```

Rclone

Install Rclone with just install-rclone.
```
just install-rclone
```
Create the Rclone configuration directory which is /etc/rclone/ for the system and ~/.config/rclone for a user.
System
sudo mkdir --parents /etc/rclone/
User
mkdir --parents ~/.config/rclone/

Configure the Rclone credentials in the rclone.conf file in the corresponding configuration directory.

/etc/rclone/rclone.conf

[minio]
type = s3
provider = Minio
access_key_id = ********************
secret_access_key = ****************************************
region = us-east-1
endpoint = https://minio.jwillikers.io
acl = private

Ensure that only the owner can read and write the rclone.conf file.

System

sudo chmod 0600 /etc/rclone/rclone.conf

User

chmod 0600 ~/.config/rclone/rclone.conf

Modify the autoupload-rclone@.service file as needed. Additionally, the Requires and After should be updated appropriately for the node where the MinIO server resides.
System
/etc/systemd/system/autoupload-rclone@.service

link:systemd/system/autoupload-rclone@.service[role=include]
User
/etc/systemd/user/autoupload-rclone@.service

link:systemd/user/autoupload-rclone@.service[role=include]
Enable and start the instantiable autoupload-rclone@.path systemd unit passing the properly escaped path of the directory to monitor. Use the systemd-escape --path command to escape the directory’s path.
System
sudo systemctl enable --now autoupload-rclone@$(systemd-escape --path ~/pictures).path
User
systemctl --user enable --now autoupload-rclone@$(systemd-escape --path ~/Pictures).path

Code of Conduct

The project’s Code of Conduct is available in the Code of Conduct file.

License

This repository is licensed under the GPLv3. Please refer to the bundled license.

Authors

Jordan Williams

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
.vscode		.vscode
immich-cli		immich-cli
systemd		systemd
.gitignore		.gitignore
.justfile		.justfile
.pre-commit-config.yaml		.pre-commit-config.yaml
CODE_OF_CONDUCT.adoc		CODE_OF_CONDUCT.adoc
LICENSE.adoc		LICENSE.adoc
README.adoc		README.adoc
pi-camera-minio-policy.json		pi-camera-minio-policy.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.vscode

.vscode

immich-cli

immich-cli

systemd

systemd

.gitignore

.gitignore

.justfile

.justfile

.pre-commit-config.yaml

.pre-commit-config.yaml

CODE_OF_CONDUCT.adoc

CODE_OF_CONDUCT.adoc

LICENSE.adoc

LICENSE.adoc

README.adoc

README.adoc

pi-camera-minio-policy.json

pi-camera-minio-policy.json

Repository files navigation

AutoUpload

Overview

Storage

MinIO

Immich

Install

Immich CLI

Rclone

Code of Conduct

License

Authors

About

License

jwillikers/autoupload

Folders and files

Latest commit

History

Repository files navigation

AutoUpload

Overview

Storage

MinIO

Immich

Install

Immich CLI

Rclone

Code of Conduct

License

Authors

About

Topics

Resources

License

Stars

Watchers

Forks