Cannot submit archived expired file

[kam193]

Describe the bug
When I try to submit via UI a file that already exists in the malware archive, and it is old enough to have already been removed from the normal submissions, the UI fails with a timeout visible in the browser's networking tab. On the backend site, the ui container is freaking out with ConflictError exceptions and endless retries. In addition, it looks like interrupted submissions (e.g. because of restarting the container) aren't immediately removed from the quota - but I suspect they will be removed after some timeout.

To Reproduce
Steps to reproduce the behavior:

1. Have an archived submission that is after its TTL, and has already been removed from the normal submission index.
2. Go to the Submit page, open web developer tools to observe network traffic.
3. Try to upload the same file as in the submission from (1).
4. Wait until you see get an error in the UI or the traffic inspector.
5. Go to the logs of ui container.
6. Observe endless retries with the errors like:
   {"@timestamp": "2024-03-28 23:28:30,167", "event": { "module": "assemblyline", "dataset": "assemblyline.datastore" }, "host": { "ip": "172.16.0.6", "hostname": "7209ae405eaa" }, "log": { "level": "INFO", "logger": "assemblyline.datastore" }, "process": { "pid": "7" }, "message": "Retrying save or freshen due to version conflict: ConflictError(409, 'version_conflict_engine_exception', '[xxx]: version conflict, required seqNo [13307], primary term [35]. but no document was found')"}
7. Side note: due to the nature of gevent server workers, processing the request in ui container isn't interrupted even after being closed on the nginx side.

Expected behavior
I can upload a file even if it's already in the archive. The file is normally processed, and I eventually see additional information from the archive.

Screenshots

Environment (please complete the following information if pertinent):

• Assemblyline Version: 4.5.0.7
• Browser: Firefox

Additional context
I reproduced this with a few files, freshly expired (a day after TTL) as well as much older one (a few months after TTL). When I tried a file that is in the archive, but hasn't been expired yet, there was no problem.