[Bug]: Local storage adaptor silently fails to upload when out of space

[YaphetKG]

What happened?

Lakefs opensource version was deployed in a Kubernetes cluster. The storage space was smaller than the size of files we wanted to upload. Using Aws cli tool to upload the files to the lakefs s3 interface , everything seems to have went fine. But storage fills up in the server and there is a mismatch on file sizes in the meta store vs the actual file size on disk. (Note this is using local storage for lakefs storage) .
Steps to Reproduce:

1. Configure lakefs to use local storage, with small space
2. upload files that would be larger than the storage in one go
3. try to download the files once the upload "finished" , and errors like "expect x bytes, but recieved y " happen (this was using the lakefs python sdk to download, error maybe different on a different client)

Expected behavior

The aws client or any other uploading tool should be notified and be aware of error if lakefs doesn't have enough storage.

lakeFS version

treeverse/lakefs:1.18.0

How lakeFS is installed

Kubernetes

Affected clients

Aws-cli s3 , python sdk

Relevant log output

probably this chat on lakefs slack help might be releavant than the logs... 
https://lakefs.slack.com/archives/C016726JLJW/p1713891191846399 

https://lakefs.slack.com/archives/C016726JLJW/p1714576349984149

Contact details

kebedey@renci.org