Pretty much complete rewrite to use Docker

.dockerignore

@@ -0,0 +1,5 @@
+*
+!source/
+!package.json
+!saved_data/
+!secrets.env

.gitignore

@@ -1,8 +1,4 @@
-node_modules
+Furry-Database/
+node_modules/
 package-lock.json
-options.json
-json
-files
-test.js
-images
-logs
+saved_data/

Dockerfile

@@ -0,0 +1,10 @@
+FROM node:latest
+
+COPY ./secrets.env /source/secrets.env
+COPY ./package.json /source/package.json
+WORKDIR /source
+RUN npm install
+
+COPY ./source/ /source/
+
+ENTRYPOINT ["node", "./main.js"]

README.md

@@ -2,49 +2,29 @@
 
 I wanted an automated system for keeping track of posts on e621 for analysis of sources, tags, scores, and users. This program does that for a few different sites. It gets expanded to include more whenever I am in the mood.
 
-# Setup
-Prerequisites:
-* [Postgresql](https://www.postgresql.org/) (10 or higher)
-* [NodeJS](https://nodejs.org/en/) (14 or higher)
-
-After those are installed and setup you can download the repository with `git clone https://github.com/Sasquire/Furry-Database.git`.
+# Supported Sites
+* e621
+* ~~FurryNetwork~~
+* FurAffinity *coming soon* 
+* InkBunny *coming soon*
+* SoFurry *coming soon*
+* Weasyl *coming soon*
+* Twitter (from an artist list) *coming soon*
+* Reddit (from a user/subreddit list) *coming soon*
+* Patreon *coming soon*
+* Pixiv (Maybe?) *coming soon*
+* Instagram (Maybe?) *coming soon*
+* Tumblr *coming soon*
 
-Next copy `sample_options.json` to `options.json`. This is where all of the settings for the application are managed. You will want to configure these values making them appropriate to your setup.
+# Setup
+This was designed to be run 100% inside the docker container. I have no idea if/how it runs outside of the container. You will have to mess with things and get it to work on your own. On the bright side, installation should be as easy as cloning this repository and starting the docker container!
 
 # Documentation
-
-Updating posts and downloading images can be done through the `main.js` file. When you run the program you will need to specify a site (`-s` option) and a command (`-c` option). An example command is `node main.js -s e621 -c images`. Some commands will take extra parameters.
-
-A `--debug=<level>` option is available and will change the verbosity of outputs. The acceptable options are
-* `error` Will only output error messages
-* `info` The default logging level. Enough to tell you what is happening
-* `debug` More information than info, but extremely mundane details are left out
-* `all` Any and all log outputs
-
-Here is a list of all commands. Do not expect it to be comprehensive. If you want a comprehensive list of available commands, running the `main.js` file with a site selected will tell you the commands available and running without a site selected will tell you the sites available.
-
-* `-s maintenance`
-  * `-c every` Runs the init scripts leaving an empty database
-  * `-c check_all_files` Checks files and ensures their md5 and name match
-  * `-c create_folders` Creates the image folders structure
-  * `-c "don't do this it drops the db"` Literally drops the database, irreversibly destroying any data you had stored. (images, logs, and json remain saved to disk).
-* `-s e621`
-  * `-c minimal` Refreshes the database with recent updates
-  * `-c bulk` Tries to download every post
-  * `-c images` Downloads images of posts stored in database
-  * `-c import_json path/to/folder` Loads database from stored JSON files
-  * `-c import_md5_csv path/to/file.csv` Loads file statuses from a csv
-* `-s furry_network`
-  * `-c minimal bearer` Refreshes the database with recent updates
-  * `-c typed_bulk bearer type [starting_page]` Tries to download every post of \<type\> starting at \<starting_page\>
-  * `-c images` Download images of posts stored in database
-  * `-c import_json path/to/folder` Loads database from stored JSON files
-  * `-c import_md5_csv path/to/file.csv` Loads file statuses from a csv
-
-# Notes
-The e621 bulk download lacks a starting id parameter because when I tested it, I was able to download all posts in one go. The FurryNetwork ones does have a starting page parameter because every hour the bearer token updates and you will need to reacquire it.
-
-There are a few scripts that I use to maintain the database. I run these on an automated schedule with crontab, reducing the amount of effort required. They will need setup on your behalf before they can function how they are supposed to.
+TODO
 
 # License
-This program is licensed under the Unlicense. Do with it whatever you want.
+This program is licensed under the Unlicense. Do with it whatever you want.
+
+#### Notes to self
+* `docker-compose down && docker-compose build && docker-compose up -d`
+* `docker exec -it furry_database_postgres psql -U admin -W -d furry`

docker-compose.yml

@@ -0,0 +1,28 @@
+version: '3.8'
+name: "furry_database"
+services:
+    database:
+        image: postgres
+        restart: "unless-stopped"
+        env_file:
+          - secrets.env
+        container_name: furry_database_postgres
+        volumes:
+          - ./saved_data/postgres/:/var/lib/postgresql/data/
+        healthcheck:
+            test: ["CMD-SHELL", "pg_isready"]
+            interval: 2s
+            timeout: 10s
+            retries: 10
+        ports:
+          - 5432:5432
+    downloader:
+        build: .
+        container_name: furry_database_downloader
+        depends_on:
+            database:
+                condition: service_healthy
+        volumes:
+          - ./saved_data/images/:/images/
+          - ./saved_data/json/:/json/
+          - ./saved_data/csv/:/csvs/

package.json

@@ -1,57 +1,52 @@
 {
-	"name": "e621_database",
-	"version": "1.0.0",
-	"description": "",
-	"main": "api.js",
-	"scripts": {
-		"test": "echo \"Error: no test specified\" && exit 1"
-	},
-	"author": "",
-	"license": "ISC",
-	"dependencies": {
-		"axios": "^0.19.2",
-		"eslint": "^6.8.0",
-		"eslint-config-standard": "^14.1.0",
-		"eslint-plugin-import": "^2.20.1",
-		"eslint-plugin-node": "^11.0.0",
-		"eslint-plugin-promise": "^4.2.1",
-		"eslint-plugin-standard": "^4.0.1",
-		"get-pixels": "^3.3.2",
-		"minimist": "^1.2.0",
-		"neo-async": "^2.6.0",
-		"pg": "^8.2.0",
-		"request": "^2.88.0",
-		"stream-json": "^1.2.1"
-	},
-	"eslintConfig": {
-		"extends": "standard",
-		"parserOptions": {
-			"ecmaVersion": 11,
-			"sourceType": "module"
-		},
-		"env": {
-			"browser": true,
-			"node": true,
-			"es6": true,
-			"mocha": true
-		},
-		"rules": {
-			"semi": [
-				"error",
-				"always"
-			],
-			"no-console": "off",
-			"indent": [
-				"error",
-				"tab",
-				{
-					"SwitchCase": 1
-				}
-			],
-			"no-tabs": "off",
-			"camelcase": "off",
-			"generator-star-spacing": "off",
-			"yield-star-spacing": "off"
-		}
-	}
+  "dependencies": {
+    "async": "^3.2.4",
+    "axios": "^1.5.0",
+    "binary-search": "^1.3.6",
+    "csv-parse": "^5.5.0",
+    "dotenv": "^16.3.1",
+    "log4js": "^6.9.1",
+    "nanoid": "^4.0.2",
+    "pako": "^2.1.0",
+    "pg": "^8.11.3",
+    "readable-stream-clone": "^0.0.7"
+  },
+  "ecmaVersion": "2022",
+  "sourceType": "module",
+  "type": "module",
+  "eslintConfig": {
+    "extends": "standard",
+    "parserOptions": {
+      "ecmaVersion": 2022,
+      "sourceType": "module"
+    },
+    "env": {
+      "browser": true,
+      "node": true,
+      "es6": true,
+      "mocha": true
+    },
+    "rules": {
+      "semi": [
+        "error",
+        "always"
+      ],
+      "no-console": "off",
+      "indent": [
+        "error",
+        "tab",
+        {
+          "SwitchCase": 1
+        }
+      ],
+      "no-tabs": "off",
+      "camelcase": "off",
+      "generator-star-spacing": "off",
+      "yield-star-spacing": "off"
+    }
+  },
+  "devDependencies": {
+    "eslint": "^8.49.0",
+    "eslint-config-standard": "^17.1.0"
+  }
 }