Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update memory statistics table #2304

Open
victorapm opened this issue Feb 20, 2023 · 6 comments · May be fixed by #3052
Open

Update memory statistics table #2304

victorapm opened this issue Feb 20, 2023 · 6 comments · May be fixed by #3052
Assignees
@arng40
Labels
type: feature New feature or request

Comments

@victorapm
Copy link
Contributor

What is the requested feature?
Add minimum and average (across ranks) memory utilization statistics from umpire and update format of the statistics table.

Is your request related to a specific problem?
No

Describe the solution you'd like
Here is an example of what we currently have:

Umpire          DEVICE sum across ranks: 15293.4 GB
Umpire          DEVICE         rank max:   31.3 GB
Umpire       DEVICE::0 sum across ranks: 15293.4 G
Umpire       DEVICE::0         rank max:   31.3 GB
Umpire            HOST sum across ranks: 7272.2 GB
Umpire            HOST         rank max:   14.7 GB
Umpire HYPRE_DEVICE_POOL sum across ranks: 7178.4 GB
Umpire HYPRE_DEVICE_POOL         rank max:   14.8 GB
Umpire          PINNED sum across ranks:  256.3 GB
Umpire          PINNED         rank max:  589.6 MB

This is what I propose:

+-------------------+---------------+---------------+---------------+--------------+
|       Umpire Pool |    Min (GB/%) |    Max (GB/%) |    Avg (GB/%) |   Sum (TB/%) |
|                   |    over ranks |    over ranks |      per rank |    all ranks |
+-------------------+---------------+---------------+---------------+--------------+
|            DEVICE |   31.3  (49%) |   31.3  (49%) |   31.3  (49%) |  15.2  (24%) |
|              HOST |   14.7 (2.9%) |   14.7 (2.9%) |   14.7 (2.9%) |   7.2 (2.9%) |
| HYPRE_DEVICE_POOL |   14.8  (23%) |   14.8  (23%) |   14.8  (23%) |   7.1  (13%) |
|            PINNED |  0.589 (0.1%) |  0.589 (0.1%) |  0.589 (0.1%) |   0.3 (0.1%) |
+-------------------+---------------+---------------+---------------+--------------+

Describe alternatives you've considered
N/A

Additional context
Having access to minimum, maximum, and average utilization leads to a better understanding of load balancing.
@victorapm victorapm added type: feature New feature or request type: new A new issue has been created and requires attention labels Feb 20, 2023
@victorapm victorapm changed the title Your Title Update memory statistics table Feb 20, 2023
@MelReyCG MelReyCG self-assigned this Feb 23, 2023
@TotoGaz TotoGaz removed the type: new A new issue has been created and requires attention label Feb 23, 2023
@jeannepellerin
Copy link
Contributor

Yes, this would definitely help.

We could also add the number of ranks in the output log to be able to trace that info more easily.

@TotoGaz
Copy link
Contributor

TotoGaz commented Mar 2, 2023

@MelReyCG you shouldn't put an issue in the ready status without filling the Impact / ... / component fields.
Also, ready means that you're going to work on this soon. Is it what you meant when changing the status?

@MelReyCG
Copy link
Contributor

MelReyCG commented Mar 2, 2023

I'm not sure what you're talking about, I don't think I did that myself, I just assigned myself to this issue because I will start working on it soon.

@TotoGaz
Copy link
Contributor

TotoGaz commented Mar 2, 2023

Didn't you do
image
?

@MelReyCG
Copy link
Contributor

MelReyCG commented Mar 3, 2023

Yes I did, we moved it. I just filled the fields.

@paveltomin
Copy link
Contributor

@MelReyCG any update?

@MelReyCG MelReyCG assigned arng40 and unassigned MelReyCG Mar 8, 2024
@arng40 arng40 linked a pull request Apr 3, 2024 that will close this issue
Umpire logs refactor - table output + adding percentages #3052
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arng40 arng40

Labels
type: feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Umpire logs refactor - table output + adding percentages GEOS-DEV/GEOS
6 participants
@paveltomin @jeannepellerin @TotoGaz @victorapm @MelReyCG @arng40