Polyaxon-sidecar - ephemeral storage usage

[kornelro]

Hello!

I'm curious about why polyaxon-sidecar loads all the data saved to job output into local memory. Shouldn't the data be saved directly to the artifact store (NFS in my case)?

In more detail:
My job involves training a large ML model that exceeds the size limit of polyaxon-sidecar's ephemeral storage. Consequently, when I save the model to the job output, Kubernetes stops the job due to the sidecar local storage limit being exceeded. As a workaround, I now have to save the model outside of the job output, which poses challenges such as lack of support for automatically deleting models after deleting jobs, among other issues.

Could you kindly explain why the sidecar loads output data to local storage, and if there's any way to disable this behavior? Alternatively, could you recommend some best practices for model storage when using the Polyaxon platform?

Thank you for your assistance!

[polyaxon-team]

I think you can opt to not use the sidecar for that and automatically use the artifacts store to save the model.
You can disable the sidecar using:

plugins:
  mountArtifactsStore: true
  collectArtifacts: false

When using the tracking module it will save automatically to the NFS module.

More details: https://polyaxon.com/docs/core/specification/plugins/