Resources limit?

[mfin]

Is it possible to limit the amount of resoufces for particular replicationsource/replicationdestination jobs? Or even for all of them (globally)? I can't seem to find the required information.

[winkrs]

I'm also looking for the same information. From what I observed, it seems like the resource usage increases as the size of the data increases(tested this for rsync tls) and could not figure the right amount of limit to allocate. Would really appreciate if anyone can get this answered.

[JohnStrunk]

While not in a release yet (it's in main), there is the ability to set requests/limits for the mover pods:

• resourceReqs and podLabels for mover job/deploy #1072
• docs for mover resource requests/limits #1103
• Docs: https://volsync.readthedocs.io/en/latest/usage/resourcerequirements.html

While setting requests is fine, we aren't currently recommending limits. This is because (as @winkrs has noted) the amount of resources needed are data dependent. Unfortunately, that's just a property of the underlying movers (rsync, restic, and rclone).

If you want to set limits, my suggestion would be to measure the actual and set a limit at some multiple of that. You will then need to monitor it over time as your data changes.

[Shashankft9]

@JohnStrunk, would it be better to allow adding the limit fields in the CRD, then its on the admin to decide whether they want to set limits or not, and if they do set it, then they should do it considering the data size, wdyt?
Asking this because, as cluster admin, I'd much rather have the replication fail then the job taking unlimited resources eventually leading to the node (on which its scheduled) to go down, but this obviously entails below ideas:

• proper metrics or ways to identify if the replication jobs are failing
• a clear gotcha defined in the readme for pitfalls when limits are defined, but also why it could be important.

for example, in our testing, we observed one edge case where there were too many files with very less data, and the memory consumption was almost 30-40% of the pvc data, although the consumption was almost following a logarithmic curve, some data below:

[JohnStrunk]

Limits for the operator are set in the Helm chart, limits for the movers are set in the replication CRs. Since it's data-dependent, the mover requests/limits must be set in a CR otherwise, it'd have to be a one-size-fits-all choice for the cluster.
Users are responsible for setting requests/limits on applications that they deploy into their namespace. Setting those for ReplicationSource and ReplicationDestination are no different than any other application (it's just a CR instead of a Deployment).

I'm not surprised that you see wildly varying memory usage. The different movers track varying amounts of metadata about the files.

As an aside, going over a node's capacity shouldn't take down the node. If it does, there are other cluster configuration problems that need to be fixed.

[Shashankft9]

oh! apologies, I think I misread the PRs linked, if we are able to set limits through replication CRs, that should be enough, I thought we can only set requests through CRs, sorry again - we'll try it out first thing tomorrow.