HTTP 500: 'internal error' when restoring, no logs available

[cmccab-icims]

My Environment

• ArangoDB Version: 3.9.10
• Deployment Mode: Cluster
• Deployment Strategy: Manual Start
• Configuration: Cluster
• Infrastructure: AWS
• Operating System: CentOS7
• Total RAM in your machine: 32Gb x 5
• Disks in use: SSD x 5
• Used Package: Debian

Component, Query & Data

Affected feature:
Backup restore

Size of your Dataset on disk:
373G

Replication Factor & Number of Shards (Cluster only):
rep factor: 3, No. of shards: 1

Problem:
Restoring a db dump which contains a particularly large collection 'resumes' approx 175mill doc(s) returns error HTTP 500: 'internal error' when restoring using arangorestore while testing our Disaster Recovery setup. There doesn't seem to be any more information given as to why this error is occurring and I can't find anything in the logs. See Error 1. To get around this issue I rerun the restore with --force enabled however these errors persist and the full collection fails to be restored. I also see another 403 error 'collection is read-only' which I don't understand as I have not set this collection to be read-only in either source/destination.

(NOTE: version 3.9.10, we are tied to 3.9 for now, I know it's reached EOL for some time so any help is greatly appreciated)
Error1:

2023-12-12T11:50:55Z [7219] INFO [75e65] {restore} # Current restore progress: restored 18 of 19 collection(s), read 110.6 GB from datafiles, sent 13196 data batch(es) of 110.5 GB total size, queued jobs: 0, workers: 4
2023-12-12T11:50:59Z [7219] WARNING [a595a] {restore} Error while restoring data into collection 'resumes': got invalid response from server: HTTP 500: 'internal error' while executing restoring data
Error2 (ran arangorestore with --force):
2023-12-14T14:13:42Z [31234] INFO [69a73] {restore} # Loading data into document collection 'resumes', 909.0 GB of 909.0 GB read (99 %)
2023-12-14T14:13:42Z [31234] WARNING [a595a] {restore} Error while restoring data into collection 'resumes': got invalid response from server: HTTP 403: 'collection is read-only' while executing restoring data

Further details

• Source Cluster
• No. of Nodes: 5
• Collections: 19
  - Problematic collection: "resumes"
• Size of problematic collection: 173million doc(s)
• Arangodb version: 3.9.10

backup command:

arangodump --server.endpoint "tcp://$host:8529" --server.username "$USER" --server.password "$PASS" --server.database "$database" --output-directory "$BKUP_DIR/$FILE_NAME" --compress-output false --progress --threads 4

Compression command: tar -I lz4 -cf "$BKUP_DIR/$FILE" --directory "$BKUP_DIR/$FILE_NAME" .
Arangodb cluster start command:

/usr/bin/arangodb \
    --starter.address=${PRIVATEIP} \
    --starter.data-dir=${DATADIR} \
    --starter.join=${STARTERENDPOINTS} \
    --auth.jwt-secret=/etc/arangodb3/jwtSecret \
    --server.storage-engine=rocksdb

Destination Cluster

• No. of Nodes: 5
• Collections: 19
• Arangodb version: 3.9.10
• DeCompress command: tar -I lz4 -xf /datadrive/$backupObj --directory "/datadrive/$database-$yyyymmdd/"

Restore command:

arangorestore --server.password "$PASS" --server.endpoint "tcp://$ip:8529" --server.username "root" --server.database "$database" --input-directory "/datadrive/$database-$yyyymmdd/"
Successfully restored: 144million doc(s)

[garyhampson]

Can we get an update from someone at ArangoDB on this issue?