Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ArchiveSupport ZIP extraction should handle ZIP archives with highly compressed large data more careful #3029

Open
3 tasks
winzj opened this issue Mar 18, 2024 · 0 comments

Comments

@winzj
Copy link
Member

winzj commented Mar 18, 2024

Situation

A test with a ZIP file with highly compressed large data, which when extracted exceed the PDS solutions workspace size, make a PDS job fail. The PDS will simply fail the job, because an exception happens with the message: No space left on device.
The PDS then cleans up the workspace for this job and starts the next job. This could cause problems for other jobs running on the same PDS as well, if someone tries this on purpose e.g. multiple times to create heavy load.

Important Insights

  • Files are uploaded and extracted
  • PDS fails the job if the workspace size is not sufficient for the extracted data size
  • ZIP archives are not extracted recursively, which means no recursive ZIP bombs are possible

Wanted

The ArchiveSupport must be able to check the uploaded sources ZIP file before or while extracting it and fail the job without risking to fail other jobs currently running on the PDS solution.

Discussed solutions

  • configurable time: Fail the job before extracting everything, if it takes longer than the configured time value to extract an entry.
  • configurable size: Fail the job before extracting everything, if the stream archive entry is to big
  • compression rate: Fail the job before extracting everything, if the compression rate is "unusually" high, like 5Kb would extract to like 10GB.

All these options should be configurable via a PDS parameter. This way administrators can adjust these values as needed. A reasonable default value must be set for each parameter.

@winzj winzj added this to the PDS 1.6.0 milestone Mar 18, 2024
@winzj winzj changed the title PDS ZIP extraction should handle ZIP archives with highly compressed large data more careful ArchiveSupport ZIP extraction should handle ZIP archives with highly compressed large data more careful Mar 19, 2024
@sven-dmlr sven-dmlr modified the milestones: PDS 1.6.0, PDS 1.7.0 Apr 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants