Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make packs/virtualenv file copy more robust (use rsync w/ fallback to cp) #245

Open
1 of 2 tasks
cognifloyd opened this issue Sep 9, 2021 · 0 comments · May be fixed by #415
Open
1 of 2 tasks

Make packs/virtualenv file copy more robust (use rsync w/ fallback to cp) #245

cognifloyd opened this issue Sep 9, 2021 · 0 comments · May be fixed by #415
Labels
Docker enhancement New feature or request help wanted Extra attention is needed
Milestone

Comments

@cognifloyd
Copy link
Member

cognifloyd commented Sep 9, 2021

For system packs, and for any images in st2.packs.images, we copy packs and virtualenvs into /opt/stackstorm.

Before #199, the packs volumes were always emptyDir, so using /bin/cp -aR was just fine.
In #160 (comment), it was noted that the st2packs image does not have rsync so we have continued with /bin/cp even though rsync could be a bit better.

After #199, the underlying volume's filesystem might not support preserving ownership, timestamps or other attributes, as reported in #238. In #238, the NFS exports were setup with root_squash which prevented ownership preservation. Debugging that was quite annoying.

So, it would be better to handle different filesystems more gracefully. If we can preserve ownership and file permissions, great. But we should only fail if the file contents do not get copied. We might also need to fail if executable files are not executable after copy.

rsync is probably the best tool to do this -> We need to add it to the st2packs image.
But, we do not want to force everyone to rebuild their st2packs images, especially if they are not and do not intend to use the new st2.packs.volumes feature. So, if rsync is not available, we need to fall back to using cp.

See comments:

We should probably do this before prod/GA to have minimal impact on GA adopters.

@cognifloyd cognifloyd added enhancement New feature or request Docker labels Sep 9, 2021
@cognifloyd cognifloyd added this to the prod/GA milestone Sep 9, 2021
@cognifloyd cognifloyd added this to TODO in ST2 HA in K8s via automation Sep 9, 2021
@cognifloyd cognifloyd moved this from TODO to In Progress in ST2 HA in K8s Dec 19, 2021
@arm4b arm4b modified the milestones: prod/GA, post-GA Apr 21, 2022
@arm4b arm4b removed this from In Progress in ST2 HA in K8s Apr 28, 2022
@cognifloyd cognifloyd modified the milestones: post-GA, v1.0.0 Jul 30, 2022
@cognifloyd cognifloyd changed the title Make packs/virtualenv file copy more robust Make packs/virtualenv file copy more robust (use rsync w/ fallback to cp) Jan 28, 2023
@cognifloyd cognifloyd added the help wanted Extra attention is needed label Jan 28, 2023
@cognifloyd cognifloyd modified the milestones: v1.0.0, v1.1.0 Feb 16, 2024
@cognifloyd cognifloyd modified the milestones: v1.1.0, v1.2.0 Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docker enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants