Check for available disk space

[poundifdef]

We want to make sure our software has enough disk space to work. If not, we want to stop processing requests.

Step 1: API Endpoint

The API consumes data and buffers on disk before uploading. If the disk is full, then we want to stop accepting new data and mark the server as unhealthy. This setting only applies to the pkg/datasink/filesystem package.

• Add a new config setting to filesystem.DataSink called FreeSpaceRequiredBytes which reads from the config yaml.
• If the value is zero or empty, then we want to interpret this as "no minimum space required"
• If there isn't at least x amount of free space in DataSink.DataDir, then WriteData() should return an error
• Similarly, if there isn't x amount of free space, then the /healthcheck endpoint should also return an error. This is a new endpoint that will have to be created.
• Emit a prometheus gauge with the current free space. (It would be fine to add it to /healthcheck.) api_free_space_bytes

Step 2: Worker

The software also runs workers that consume data and insert to the database. This should also respect the Workers.FreeSpaceRequiredBytes setting.

• As with the API, if the config has a zero value, then ignore it
• Check to see whether there is enough free space before consuming an item from the queue. If there is not enough free space, then do stop consuming from the queue.
• If disk space gets freed, then we should resume consuming from the queue
• Emit a prometheus gauge with the current free space. worker_free_space_bytes