Performance Limit reasons #333
Replies: 22 comments 120 replies
-
|
For sure that is a great feature suggestion which I will implement. Clearing an event bit aka resetting its register may depend on the architecture. I'm remember freezing a Processor because writing the status log MSR was unsupported: despite the general case in specs. Definitely an enhancement to do. |
Beta Was this translation helpful? Give feedback.
-
MSR_CORE_PERF_LIMIT_REASONS (
|
Beta Was this translation helpful? Give feedback.
-
MSR_CORE_PERF_LIMIT_REASONS (
|
Beta Was this translation helpful? Give feedback.
-
MSR_CORE_PERF_LIMIT_REASONS (
|
Beta Was this translation helpful? Give feedback.
-
|
A quick bash script to check core performance limit reasons. I tried also resetting the log, it didn't hang my PC. #!/bin/bash
# MSR_CORE_PERF_LIMIT_REASONS (0x64f)
# https://www.intel.com/content/dam/develop/public/us/en/documents/335592-sdm-vol-4.pdf
# Table 2-39
printf "Cur: $( rdmsr -0 -f 0:0 0x64f ) Log: $( rdmsr -0 -f 16:16 0x64f ) \t PROCHOT \n"
printf "Cur: $( rdmsr -0 -f 1:1 0x64f ) Log: $( rdmsr -0 -f 17:17 0x64f ) \t Thermal Status \n"
printf "Cur: $( rdmsr -0 -f 2:2 0x64f ) Log: $( rdmsr -0 -f 18:18 0x64f ) \t [Reserved] \n"
printf "Cur: $( rdmsr -0 -f 3:3 0x64f ) Log: $( rdmsr -0 -f 19:19 0x64f ) \t [Reserved] \n"
printf "Cur: $( rdmsr -0 -f 4:4 0x64f ) Log: $( rdmsr -0 -f 20:20 0x64f ) \t Residency State Regulation Status \n"
printf "Cur: $( rdmsr -0 -f 5:5 0x64f ) Log: $( rdmsr -0 -f 21:21 0x64f ) \t Running Average thermal Limit Status \n"
printf "Cur: $( rdmsr -0 -f 6:6 0x64f ) Log: $( rdmsr -0 -f 22:22 0x64f ) \t VR Therm Alert Status \n"
printf "Cur: $( rdmsr -0 -f 7:7 0x64f ) Log: $( rdmsr -0 -f 23:23 0x64f ) \t VR Therm Design Current Status \n"
printf "Cur: $( rdmsr -0 -f 8:8 0x64f ) Log: $( rdmsr -0 -f 24:24 0x64f ) \t Electrical Design Current Limit / Other Status \n"
printf "Cur: $( rdmsr -0 -f 9:9 0x64f ) Log: $( rdmsr -0 -f 25:25 0x64f ) \t [Reserved] \n"
printf "Cur: $( rdmsr -0 -f 10:10 0x64f ) Log: $( rdmsr -0 -f 26:26 0x64f ) \t Package/Platform-Level Power Limiting PL1 Status \n"
printf "Cur: $( rdmsr -0 -f 11:11 0x64f ) Log: $( rdmsr -0 -f 27:27 0x64f ) \t Package/Platform-Level Power Limiting PL2 Status \n"
printf "Cur: $( rdmsr -0 -f 12:12 0x64f ) Log: $( rdmsr -0 -f 28:28 0x64f ) \t Max Turbo Limit Status \n"
printf "Cur: $( rdmsr -0 -f 13:13 0x64f ) Log: $( rdmsr -0 -f 29:29 0x64f ) \t Turbo Transition Attenuation Status \n"
printf "Cur: $( rdmsr -0 -f 14:14 0x64f ) Log: $( rdmsr -0 -f 30:30 0x64f ) \t [Reserved] \n"
printf "Cur: $( rdmsr -0 -f 15:15 0x64f ) Log: $( rdmsr -0 -f 31:31 0x64f ) \t [Reserved] \n"
printf "Cur: $( rdmsr -0 -f 63:32 0x64f ) \t [Reserved] \n"
# To reset Log:
# wrmsr 0x64f 0x00000000Output while running a stress test: |
Beta Was this translation helpful? Give feedback.
-
|
Nice! Up to now CoreFreq is employing MSR
We have more precised events for architectures from 6th up to 12th,
Status bits are RO and they may flash on UI, especially Turbo, whereas Log bits can be left to User for clearing. |
Beta Was this translation helpful? Give feedback.
-
|
For your testings commit 324bc29 is available in
|
Beta Was this translation helpful? Give feedback.
-
|
In the end I figured out that
Clearing the events by directly writing 0 in the MSR does not reset them all, and they do not change in real-time if the window is opened. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your test. Underscored Here, Tiger Lake-U is clearing events. But still partially on "Power Limitation" and "Turbo Max". Fyi any event register is not fully cleared by zero but only the concerned bit fields are. Thus Read-Modify-Write. I'm guessing some bits, although specified as shareable among Cores, are not so Package scope ... Can't really tell if clearing log bits are architectural or not among all Lake generations. UI window refreshed in RT is accepted but so far you'll need to reopen the window to verify clearing effects. |
Beta Was this translation helpful? Give feedback.
-
|
It wasn't obvious that the underscored Turbo Max will probably be re-enabled almost as soon as it is cleared, unless the CPU is running with a power saving CPU governor or on battery. Sometimes I can clear Power Limitation but it comes back on quickly. I cannot clear Thermal Threshold or Current Limitation. I only tested events from MSR 0x64f. These can be certainly cleared. I'm aware that ideally one would only clear the necessary log bits, but it seemed simpler to just fill the MSR with zeroes; read-only bits shouldn't be affected anyway. However I don't know if this is acceptable practice. By searching for "MSR_CORE_PERF_LIMIT" in https://www.intel.com/content/dam/develop/public/us/en/documents/335592-sdm-vol-4.pdf it appears that the clearing log bits always begin from bit 16, but there is no guarantee this will be the case forever. |
Beta Was this translation helpful? Give feedback.
-
This one is now available in commit fb30312 |
Beta Was this translation helpful? Give feedback.
-
Battery not involved. TGL is DC plugged in and the same happens
Specs defines the whole log bits as Package scope
Specs defines the whole Log bits as Package scope. Bit clear on any Core should apply to all. Investigations are:
|
Beta Was this translation helpful? Give feedback.
-
|
Very odd, without any change, except a fresh boot,
EDIT: No Errata found on subject. |
Beta Was this translation helpful? Give feedback.
-
|
In the past I put a condition to check presence of a thermal BIOS interrupt activation. Once
Commit b3ff656 is taking the risk to change that and now double checks the If Interrupts not capable then clearing bits in
Non regression tests are required on all Intel processors: Need everyone' reports! |
Beta Was this translation helpful? Give feedback.
-
|
Another test. After a few seconds of Prime95 AVX:
After
After
|
Beta Was this translation helpful? Give feedback.
-
|
Sounds great. Thank you for your tests. |
Beta Was this translation helpful? Give feedback.
-
|
Also be aware there are two STATUS MSRs, respectively per Core and for the whole Package |
Beta Was this translation helpful? Give feedback.
-
Writing 0 to this register cleared "Power Limitation" as well. "Max Turbo Limit" would be cleared by one of the other MSRs if frequency was not allowed to reach the maximum. Personally I would simply use ring-gt-limit-reasons.mp4 |
Beta Was this translation helpful? Give feedback.
-
|
Very well made. |
Beta Was this translation helpful? Give feedback.
-
|
A better mock-up of the real-time textual representation of CPU limit reasons that I had in mind and showed from another program in the opening post: 2022-03-31.16-58-41_limit-reasons-mockup.mp4What happens in the video:
Yellow = Logged limit reason |
Beta Was this translation helpful? Give feedback.
-
|
Looks really nice and I'm surely getting inspired for next enhancements: Backend
Frontend
Now just waiting for the bandwidth , Alder lake is a real concern. |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ : Preview c4421b3 {General, Core, GFX, Ring}
|
Beta Was this translation helpful? Give feedback.
-
|
On my system "Thermal Threshold" activated, but not "Thermal Sensor". I have however decreased the thermal limit to 92 °C via BIOS.
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for this test. It might explain why I have not been able to monitor Slowly other enhancements are committed. Line 937 in 064070d For ex. a heavy WebGL demo will flash |
Beta Was this translation helpful? Give feedback.
-
A dedicated Events window function is available for testings.
Four columns, General aka IA, Core, GFX, RING, don't fit the terminal width of 80 characters. A layout suggestion ? |
Beta Was this translation helpful? Give feedback.
-
|
I confirm it working also in real-time, although I have still issues clearing logged DTS events once they appear.
|
Beta Was this translation helpful? Give feedback.
-
|
Same here with |
Beta Was this translation helpful? Give feedback.
-
|
I can clear Max Turbo for a while longer if I use the CPU governor "powersave" instead of performance, but eventually it will trigger again within seconds. Possibly the CPU is using max turbo frequencies for periods briefer than what CoreFreq can monitor at 100 ms intervals, because I can't notice absolute core frequencies to reach the maximum when this happens. Logged DTS events however appear to stay lit for different reasons. |
Beta Was this translation helpful? Give feedback.
-
I'm sharing the same conclusions. Perhaps one of those technologies can make a difference ? Be also aware you can narrow the Monitoring interval down to Events would have been better monitored by processor if handled like the performance counters where we just have to read and compute the measurements delta. |
Beta Was this translation helpful? Give feedback.
-
|
After brief testing, I have already tried decreasing the monitoring interval to 100 ms as well, but although it can help detecting brief turbo events it will also increase CPU load. I normally use the |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ |
Beta Was this translation helpful? Give feedback.
-
|
Added a
|
Beta Was this translation helpful? Give feedback.
-
|
Appears to work. Like clearing single limit reasons, It's a bit slow to respond to user input; I guess it waits for the next polling event or something along these lines. 2022-04-09_clear-all.mp4 |
Beta Was this translation helpful? Give feedback.
-
Thanks for your test. Indeed CLI is slow to respond: drawing happens only when notified by backend or some keys pressed. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @BugReporterZ This is the on-going development: Yellow for Event log, Magenta for Event status
It has been simulated by code to raise all of them. Please let me know how does it work with your CPU ? |
Beta Was this translation helpful? Give feedback.
-
|
Without doing anything, the reported TjMax in CoreFreq seems correct (92 °C as manually set); offset is 0 as I believe it should be.
After doing Then, I interrupted the daemon, unloaded the kernel module with
However, the CPU still goes up to TjMax under load. |
Beta Was this translation helpful? Give feedback.
-
|
OH thanks, I'm reviewing this ADL part. stay tuned. |
Beta Was this translation helpful? Give feedback.
-
|
EDIT: forgot to say; it's the same here when a register is altered externally. |
Beta Was this translation helpful? Give feedback.
-
|
Yep, it looks like the |
Beta Was this translation helpful? Give feedback.
-
On the Roadmap, the ability to tweak those thresholds from the UI. |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ : In commit 7cadc81 temperature thresholds are now decoded based on TjMax. Thus you should read the Celsius in the right direction. But don't go beyond your TjMax. Here stressing two Cores to reach a temperature between the 2 indicators and to trigger only the first threshold alarm.
|
Beta Was this translation helpful? Give feedback.
-
|
Here's a test where they all trigger in sequence as temperatures increase—it seems to work. 2022-04-12._t-the-sequence.mp4 |
Beta Was this translation helpful? Give feedback.
-
|
Really neat demo which gives a large overview of indicators. I have been wondering if it would make sens to compute a tendency. For exemple alarm is increasing, decreasing or remain constant ? |
Beta Was this translation helpful? Give feedback.
-
|
On desktop systems, I'm not sure. There's always some sort of active limit, and due to the way they are reported on modern CPUs, temperatures tend to change quickly, for the most part depending on load rather than thermal inertia as with old CPUs. Sometimes depending on load limits occur in quick succession, e.g.: Max Turbo > PL2 > EDP Limit > Thermal Limit > PL1 > Steady-state operation This behavior would probably be better appreciated on a real-time graph rather than with a synthetic trend indicator. |
Beta Was this translation helpful? Give feedback.
-
|
Commit cb803e1 is adding an array parameter to the driver.
Examples
|
Beta Was this translation helpful? Give feedback.
-
It does indeed appear that temperature thresholds trigger depending on what is reached first, i.e. core or package. Here I just set package with the new module option: 2022-04-14_pkg-thrsh.mp4 |
Beta Was this translation helpful? Give feedback.
-
it's probably best to favor UI consistency even if it costs a bit in terms of responsiveness. |
Beta Was this translation helpful? Give feedback.
-
|
Adding separators since commit 0665d79 I also found the CTF (Critical Temperature Fault) in AMD/Zen/SMU which should be reflected DTS/Critical Temp. I can't manage to test it b/c I believe this bit is raised when Ryzen is thermally screw up! |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure how that could be tested on Intel CPUs either. Shutdown temperature should be at 130 °C, at least on Alder Lake, and the Critical temperature warning should trigger a bit below that. TJMax can be configured up to 115 °C; critical temperature should be above this.
|
Beta Was this translation helpful? Give feedback.
-
|
Let's hope we won't never see the red light on 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
It isn't documented in the datasheet, but Bit 14 (log bit 30) appears to be for Thermal Velocity Boost/TVB (a functionality that decreases or increases core speed by a fixed bin depending on temperature). In the test below I set it to 70 °C. 2022-04-18_TVB-bit.mp4In the MSR datasheet it just says "reserved".
I just found this since HWInfo64 (a popular CPU/system monitoring program for Windows) also lists TVB among CPU core limit reasons. |
Beta Was this translation helpful? Give feedback.
-
|
From die shot interpretations, E-cores should come in clusters of 4 cores.
I don't know about P-cores, but usually these processors have 2 "best" cores (on my i7-12700K, number 2 and 3), which require lower voltage on average. Mine also has 4 cores requiring average voltage (0, 1, 4, 5) and 2 which don't perform as well (6, 7). Since there are always 2 "best" cores and apparently "normal" / "worse" cores always in multiples of 2, it's possible that there P-cores might technically be arranged in clusters of 2 (i.e. 4 logical CPUs), although this is just my speculation. |
Beta Was this translation helpful? Give feedback.
-
|
Certain motherboards e.g. from Asus show core quality more directly. You can see from the i9-12900KS samples below that P-core VIDs are always arranged in multiples of 2:
|
Beta Was this translation helpful? Give feedback.
-
Probably an answer from the ACPI SLIT table. https://www.codeblueprint.co.uk/2019/07/12/what-are-slit-tables.html To try: |
Beta Was this translation helpful? Give feedback.
-
|
Output: |
Beta Was this translation helpful? Give feedback.
-
|
So one node with everything inside! I'm back to this reading: Remaining of document is about various masks provided by Windows API Coupling with these kernel patches: https://lore.kernel.org/lkml/cover.1638563225.git.tim.c.chen@linux.intel.com/t/#u I believe the hardware/firmware cluster awareness rely in vendor data; thus ACPI |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ : TVB event is ready for your testings in latest EDIT: plz wait, forgot the log label |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ Commit is complete. You can now test. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Here is a test with a partial load. TVB was temporarily set to 70°C and -400 MHz for 1-8 active cores (I don't normally use it). 2022-04-20_TVB-cfr.mp4 |
Beta Was this translation helpful? Give feedback.
-
Thank you for your answer. |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ Hello,
|
Beta Was this translation helpful? Give feedback.
-
|
Seems to work fine. 2022-05-04_compact-limit-reasons.mp4The only remaining issue is that DTS events cannot be cleared. I made a test and if it works if I write 0 to the corresponding MSR for both core and package, i.e.: |
Beta Was this translation helpful? Give feedback.
-
|
@BugReporterZ : Suggestions to deactivate the kernel agent:
/*
* When users space sets a trip temperature == 0, which is indication
* that, it is no longer interested in receiving notifications.
*/ |
Beta Was this translation helpful? Give feedback.
-
|
Tried compiling 5.18-rc5 disabling |
Beta Was this translation helpful? Give feedback.
-
|
Sorry same setup here but I can clear DTS log bits
In function line 8222 if (ClearBit)
{
if (!(ThermInterrupt.High_Temp_Int|ThermInterrupt.Low_Temp_Int))
{
if (PUBLIC(RO(Proc))->Features.Power.EAX.HWFB_Cap)
{
WRMSR(ThermStatus, MSR_IA32_THERM_STATUS);
RDMSR(ThermStatus, MSR_IA32_THERM_STATUS);
pr_info("CoreFreq: MSR_IA32_THERM_STATUS[0x%llx]\n",
ThermStatus.value);
} else {
pr_warn("CoreFreq: HWFB_Cap missing\n");
}
} else {
pr_warn("CoreFreq: ThermInterrupt access denied\n");
}
}line 8370 if (ClearBit)
{
if (!(ThermInterrupt.High_Temp_Int|ThermInterrupt.Low_Temp_Int))
{
if (PUBLIC(RO(Proc))->Features.Power.EAX.HWFB_Cap)
{
WRMSR(ThermStatus, MSR_IA32_PACKAGE_THERM_STATUS);
RDMSR(ThermStatus, MSR_IA32_PACKAGE_THERM_STATUS);
pr_info("CoreFreq: MSR_IA32_PACKAGE_THERM_STATUS[0x%llx>
ThermStatus.value);
} else {
pr_warn("CoreFreq: @Pkg HWFB_Cap missing\n");
}
} else {
pr_warn("CoreFreq: @Pkg ThermInterrupt access denied\n");
}
}
|
Beta Was this translation helpful? Give feedback.
-
|
I have a bug in the sequence ... |
Beta Was this translation helpful? Give feedback.
-
|
Corrected issue, added that in |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Here I switch the same workload on different cores and show the results with the Package Cycles view. When the load is on E-cores, uncore cycles seem lower, as they are supposed to be. At full load in my case Ring frequency (Ring is often often referred to as Uncore) should be up to 4.6 GHz when only P-cores are loaded, and 3.6 Ghz if E-cores are also loaded, so it doesn't look like just multiplying this uncore counter by 100000 is going to give good results. 2022-05-15_package-view-uncore.mp4 |
Beta Was this translation helpful? Give feedback.
-
Indeed! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Except this screenshot showing Could you please try to screenshot them ? Your PC will have to be mostly idle; C-States enabled in BIOS and Kernel. I'm expecting to read some Thank you |
Beta Was this translation helpful? Give feedback.
-
|
I'm seeing this:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
IINM your referring to a ratio/multiplier of which frequency depends on base/bus clock. |
Beta Was this translation helpful? Give feedback.
-
|
Thks, can you point source code lines ? |
Beta Was this translation helpful? Give feedback.
-
|
Source code? It's just a spot ratio reading taken from status MSR 0x621 and multiplied by base clock which usually sits around ~100MHz for a lot of systems but can be quite different on others. If your looking for a rolling average of counts then sorry idk |
Beta Was this translation helpful? Give feedback.
-
|
Not found in latest SDM, do you have the specifications of MSR |
Beta Was this translation helpful? Give feedback.
-
An important aspect of monitoring CPU performance is knowing what is currently limiting or throttling it. There is more than just thermal throttling.
On modern Intel CPUs counters exist to monitor performance limiting reasons for IA (Intel Architectures = Cores), Ring (Uncore), GT (Graphics Technology = Integrated graphics).
I'm aware that the following MSRs give this information:
MSR_GRAPHICS_PERF_LIMIT_REASONSMSR_CORE_PERF_LIMIT_REASONSMSR_RING_PERF_LIMIT_REASONS(source: https://www.intel.com/content/dam/develop/external/us/en/documents/335592-sdm-vol-4.pdf . I think some of the bits that were "reserved" in this document are now used for meaningful information like for example Voltage limit)
A new View could be added in CoreFreq for this, taking inspiration from the Windows application Throttlestop.
In ASCII it could be something like this (with all limits engaged, although unlikely in practice):
Real world situations may only show a few limits in one or two columns like this:
Throttlestop color codes with RED currently engaged limits and with YELLOW limits that have engaged since the last counter reset. Some sort of short term logging could be done in many ways, though.
Beta Was this translation helpful? Give feedback.
All reactions