Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor syncuser to reduce walks of the container fs #1212

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

anderbubble
Copy link
Collaborator

@anderbubble anderbubble commented May 2, 2024

Description of the Pull Request (PR):

This PR refactors syncuser to only walk the container fs twice: once for user changes and once for group changes. The previous implementation of syncuser walked the container file system for every user and group being sync'd. This caused syncuser to take a very long time when large images were combined with large user/group counts.

This PR also changes the (internal) semantics of syncuser to consistently use a "write" boolean throughout, rather than "write" in some places and "showOnly" in others.

To reduce some frequently-observed confusion, wwctl container import has been adjusted to only run syncuser if --syncuser is specified, rather than always running it in "showOnly" mode when --syncuser=false. This eliminates a spurious "error" report for users who do not intend to use syncuser.

This fixes or addresses the following GitHub issues:

Before submitting a PR, make sure you have done the following:

@anderbubble anderbubble added the bug Something isn't working label May 2, 2024
@anderbubble anderbubble added this to the v4.5.2 milestone May 2, 2024
@anderbubble anderbubble self-assigned this May 2, 2024
@anderbubble anderbubble changed the title Refactor syncuser to only walk the container fs once Refactor syncuser to reduce walks of the container fs May 2, 2024
@mslacken
Copy link
Member

mslacken commented May 6, 2024

I don't see any comparison like if id == info.ContainerID. So every exiting file is of a uid/gid is added to db[name] during the filesystem walk?

- Closes warewulf#1209

Signed-off-by: Jonathon Anderson <janderson@ciq.com>
@anderbubble
Copy link
Collaborator Author

From @mslacken in Slack:

In the syncuser code I saw following:

type syncInfo struct {
	HostID         int      `access:"r,w"`
	ContainerID    int      `access:"r,w"`
	ContainerFiles []string `access:"r,w"`
}

what does the access:"r,w" do? This looks like the #pragma oin C/C++ but I couldn't find documentation for this. Can you give me a hint?

@anderbubble
Copy link
Collaborator Author

@mslacken I looked into the access:"r,w" stuff. It looks like that originally came from 7f1636c:

7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  19) type completeUserInfo struct {
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  20)        Name        string
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  21)        UidHost     int      `access:"r,w"`
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  22)        GidHost     int      `access:"r,w"`
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  23)        UidCont     int      `access:"r,w"`
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  24)        GidCont     int      `access:"r,w"`
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  25)        FileListUid []string `access:"r,w"`
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  26)        FileListGid []string `access:"r,w"`
7f1636c9a (Christian Goll 2022-03-10 12:17:13 +0100  27) }

So I don't know what it does; but it looks like you wrote it in the first place. ^_^

I removed them locally and it didn't break tests or vet. I generally avoid making unrelated changes in a PR; but I'll do so here without objection if you'd like.

@anderbubble anderbubble modified the milestones: v4.5.2, v4.5.3 May 14, 2024
@mslacken
Copy link
Member

@anderbubble after fixing the changelog this ready to merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

syncuser walks the file system for each user/group
2 participants