couch db issue

[dazinator]

I have been running swarmpit for a while as a swarm stack.
the couch db data is on a docker volume
I needed to `docker stack rm swarmpit recently.
When I redeployed swarmpit - the service didn't come up and checking the logs

 23-07-14 14:53:59 2180a7cca2ce INFO [swarmpit.server:91] - Swarmpit is starting...
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | 23-07-14 14:53:59 2180a7cca2ce INFO [swarmpit.database:30] - Waiting for CouchDB ...
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | 23-07-14 14:54:00 2180a7cca2ce INFO [swarmpit.database:33] - ... CouchDB connected in 0 sec
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | 23-07-14 14:54:15 2180a7cca2ce ERROR [swarmpit.http:71] - Request execution failed! Scope: DB
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |> HEAD http://db:5984/swarmpit
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |> Headers: null
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |> Payload: null
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |< Message: 
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |< Data: { }
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | 23-07-14 14:54:15 2180a7cca2ce ERROR [swarmpit.http:71] - Request execution failed! Scope: DB
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |> PUT http://db:5984/swarmpit
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |> Headers: null
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |> Payload: null
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |< Message: clj-http: status 412
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | |< Data: {
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |   "headers" : {
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "Cache-Control" : "must-revalidate",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "Connection" : "close",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "Content-Length" : "95",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "Content-Type" : "application/json",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "Date" : "Fri, 14 Jul 2023 14:54:15 GMT",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "Server" : "CouchDB/2.3.0 (Erlang OTP/19)",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "X-Couch-Request-ID" : "b1dccb8149",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |     "X-CouchDB-Body-Time" : "0"
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |   },
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | 23-07-14 14:51:23 f79e2a83d5ba INFO [swarmpit.server:91] - Swarmpit is starting...
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |   "status" : 412,
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | 23-07-14 14:51:23 f79e2a83d5ba INFO [swarmpit.database:30] - Waiting for CouchDB ...
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |   "body" : "{\"error\":\"file_exists\",\"reason\":\"The database could not be created, the file already exists.\"}\n",
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | 23-07-14 14:51:24 f79e2a83d5ba INFO [swarmpit.database:33] - ... CouchDB connected in 0 sec
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |   "reason-phrase" : "Precondition Failed",
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    |   "type" : "clj-http.client/unexceptional-status"
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | 23-07-14 14:51:39 f79e2a83d5ba ERROR [swarmpit.http:71] - Request execution failed! Scope: DB
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | }
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | |> HEAD http://db:5984/swarmpit
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | Exception in thread "main" clojure.lang.ExceptionInfo: DB error: file_exists {:status 412, :type :http-client, :headers {"Cache-Control" "must-revalidate", "Connection" "close", "Content-Length" "95", "Content-Type" "application/json", "Date" "Fri, 14 Jul 2023 14:54:15 GMT", "Server" "CouchDB/2.3.0 (Erlang OTP/19)", "X-Couch-Request-ID" "b1dccb8149", "X-CouchDB-Body-Time" "0"}, :body {:error "file_exists"}}
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | |> Headers: null
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | 	at swarmpit.http$execute_in_scope.invokeStatic(http.clj:112)
swarmpit_app.1.ny99u09oic59@swarm-mgr-03    | |> Payload: null
swarmpit_app.1.gccpy22k65o6@swarm-mgr-03    | 	at swarmpit.http$execute_in_scope.invoke(http.clj:80)

I am not that familiar with couchdb but I read a similar issue where it seems swarmpit wants to run migrations again?
I ran some curl commands against the couch db instance for example it shows the following databases exist:

docker run -it --rm --network=swarmpit_net curlimages/curl:latest -X GET http://db:5984/_all_dbs
["_global_changes","_replicator","_users","swarmpit"]

From the related github issue it looked like I needed to instert a document like this:

{"_id":"1c516abfac159b53e5b9edd5d900492b", "_rev":"1-c8219b09423abbe3f9f08d5ca
d2ca9a0", "type":"migration", "name":"single-node-setup", "result":null}

but I tried

docker run -it --rm --network=swarmpit_net curlimages/curl:latest -X PUT http://db:5984/swarmpit -d '{"_id":"1c516abfac159b53e5b9edd5d900492b", "_rev":"1-c8219b09423abbe3f9f08d5ca
d2ca9a0", "type":"migration", "name":"single-node-setup", "result":null}'

and get an error that the database already exists - I think I need to put the document at some other path, other than the root - http://db:5984/swarmpit - is anyone able to kindly advise the correct curl command / path?