Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

To reduce the lock pressure on the kernfs file system, we should reduce the frequency of cgroup reads. #3528

Open
xigang opened this issue May 9, 2024 · 0 comments

Comments

@xigang
Copy link

xigang commented May 9, 2024

Intro:
We call cadvisor subcontainers API to obtain the data of all cgroup containers. With the increase of QPS, the frequency of cgroup read will increase, which will aggravate the pressure of kernfs file system lock and even cause machine hang.

Question:
Do we need to read the cgroup less frequently when we call GetInfo to fetch ContainerInfo data by caching it?

https://github.com/google/cadvisor/blob/54dff2b8ccb147747d814b0ff3b4a3256dc569c0/manager/manager.go#L543C19-L543C47

func (m *manager) containerDataToContainerInfo(cont *containerData, query *info.ContainerInfoRequest) (*info.ContainerInfo, error) {
	// Get the info from the container.
	cinfo, err := cont.GetInfo(true)  
	if err != nil {
		return nil, err
	}

	stats, err := m.memoryCache.RecentStats(cinfo.Name, query.Start, query.End, query.NumStats)
	if err != nil {
		return nil, err
	}

	// Make a copy of the info for the user.
	ret := &info.ContainerInfo{
		ContainerReference: cinfo.ContainerReference,
		Subcontainers:      cinfo.Subcontainers,
		Spec:               m.getAdjustedSpec(cinfo),
		Stats:              stats,
	}
	return ret, nil
}

If the call GetInfo (shouldUpdateSubcontainers bool) function shouldUpdateSubcontainers parameter is set to true, The updateSpec() function is called on each request to read the cgroup data, which increases the pressure on kernsfs Lock.

func (cd *containerData) GetInfo(shouldUpdateSubcontainers bool) (*containerInfo, error) {
	// Get spec and subcontainers.
	if cd.clock.Since(cd.infoLastUpdatedTime) > 5*time.Second || shouldUpdateSubcontainers {
		err := cd.updateSpec()
		if err != nil {
			return nil, err
		}
		if shouldUpdateSubcontainers {
			err = cd.updateSubcontainers()
			if err != nil {
				return nil, err
			}
		}
		cd.infoLastUpdatedTime = cd.clock.Now()
	}
	cd.lock.Lock()
	defer cd.lock.Unlock()
	cInfo := containerInfo{
		Subcontainers: cd.info.Subcontainers,
		Spec:          cd.info.Spec,
	}
	cInfo.Id = cd.info.Id
	cInfo.Name = cd.info.Name
	cInfo.Aliases = cd.info.Aliases
	cInfo.Namespace = cd.info.Namespace
	return &cInfo, nil
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant