Other Alder Lake-specific MSRs

[BugReporterZ]

https://www.intel.com/content/dam/develop/public/us/en/documents/335592-sdm-vol-4.pdf

It seems that for 12th gen processors with a hybrid architecture there are now "Primary" and "Secondary" turbo limits. Primary would be the P-cores, Secondary the E-cores. See table 2-46 in the document.

My 12700K has currently the following configuration:

P cores: 1c: 52x, 2c: 52x, 3c: 51x, 4c: 50x, 5-8c: 49x
E-cores: 1-2c: 38x, 3-4c: 36x.

The P-core ratios already work in CoreFreq. Let's test if the E-core ratios are correctly detected with msr utilities.

rdmsr -d --bitfield 7:0 1616
38

rdmsr -d --bitfield 15:8 1616
38

rdmsr -d --bitfield 31:24 1616
36

rdmsr -d --bitfield 31:24 1616
36

They appear to work as intended.

[cyring]

Wahoo; I completely passed beside the secondary Turbo ratio register.
Now that we aware of the P and E topology, methods switch should be a straightforward programming.

EDIT: except the UI which was designed before Hybrid.
Need to think about the use case and the refactoring of the unique ratios column.

[cyring]

...
It seems that for 12th gen processors with a hybrid architecture there are now "Primary" and "Secondary" turbo limits. Primary would be the P-cores, Secondary the E-cores. See table 2-46 in the document.
...

For your testings commit 7a9a994 should allow you to display and alter the Turbo ratio frequency of E-Cores and P-Cores, independently.

Here is the UI source code simulation:

Save and sync your files before trying!

If things are OK, please also reports with the full output of corefreq-cli -s -n -m

[cyring]

Reviewing your use case of 3C E-core , it messes here:

CoreFreq/corefreqk.c

Line 2605 in 7a9a994

case 3:

Because ratio, fixed or as an offset, is based on highest P-cores ratio.

I believe with the current commit, you may achieve to set an E-core when selecting a specific CPU number.
But not with the "For All" case.

I'm sorry you now have to rewrite Turbo ratio limits on both MSR (P and E) to restore sane frequencies. Or to reboot PC.

I was expecting code impacts when the Hybrid architecture has been introduced; but not as much.

[BugReporterZ]

After messing up too much with turbo ratios in CoreFreq I usually find it faster to reboot the system to bring things back to their initial state, given that some options do not seem to have a corresponding setting in BIOS, like HWP Min/Max or P-state TGT.

This could be a scenario where configuration presets could be useful, or perhaps a faster system where the user could set things like turbo ratios with direct numerical input and apply/confirm the settings all at once later on (as is done typically in BIOS).

[cyring]

After messing up too much with turbo ratios in CoreFreq I usually find it faster to reboot the system to bring things back to their initial state, given that some option do not seem to have a corresponding setting in BIOS, like HWP Min/Max or P-state TGT.

Is HWP not available as Intel Speed Shift Technology or Hardware Controlled P-State in BIOS ?

This could be a scenario where configuration presets could be useful, or perhaps a faster system where the user could set things like turbo ratios with direct numerical input and apply/confirm the settings all at once later on (as is done typically in BIOS).

Sure, many things would be greatly improved by directly entering digits, but that was not the direction taken in my UI library at first. I may but hardware features are comming first in CoreFreq. And they are a never end story! Manufucters are prolific.

About the Turbo Hybrid test: could you try again but this time selecting one of the E-core only. For exemple, the last one, like 3C of CPU 019

[BugReporterZ]

Is HWP not available as Intel Speed Shift Technology or Hardware Controlled P-State in BIOS ?

There's an Intel Speed Shift setting, but it only allows Auto/Disabled/Enabled. I cannot control target/min/max frequencies related to it. The 64x target ratio shown in CoreFreq is not featured anywhere in BIOS settings.

About the Turbo Hybrid test: could you try again but this time selecting one of the E-core only. For exemple, the last one, like 3C of CPU 019

I made a video.

2022-03-30.18-08-14.mp4

[cyring]

It appears to give the expected ratio when selecting an E-core.

Only the all for one selection is messing up with ratios (because P and E cores are not sharing same MSR and other max values).

If true then I just have to focus on UI improvements when Hybrid is in action.

[BugReporterZ]

I'm not sure what is going on, but the exact core that needs to be changed has to be selected in the screen that pops up, otherwise the ratio gets applied elsewhere, and apparently this goes both ways.

2022-03-30.18-59-20.mp4

[cyring]

Thanks for the use case, I will investigate further more.

It's a real pain and may claim a refactoring of Driver and Client to duplicate code to have two chaines of command submission, respectively P and E cores which don't have common bounds (MIN, MAX, Target and if available HWP ratios)

Without hardware under my hands, I fear it will take us long time to debug/achieve those changes.

[BugReporterZ]

Understandable that it might be difficult to debug this without actual access to the hardware.

There are also other issues that a refactor should probably care about. For example, turbo ratios for a higher number of active cores (within the same group) should be equal or higher than the turbo ratios for a lower number of active cores, but this isn't enforced in CoreFreq.

e.g. I can configure 1c: 52, 2c: 51, 3c: 50x (so far so good), but then 4c: 51, 5c: 52, etc.

[cyring]

within the same group

Just to share my specs understanding: I don't see a difference between a Package scope MSR (as found within Primary Cores or not a Hybrid processor) and a Group scope of E-cores.

Do you ?

[BugReporterZ]

I informally wrote "group" but I meant something different than what is used in the datasheet.

Here I only meant that the rule (= monotonically decreasing turbo ratio) is intended to be obeyed within the same bunch of homogeneous cores, i.e. E-cores or P-cores.

Turbo ratios can be in principle defined for a larger number of cores than one at a time, e.g. [1-2c], [3-4c], [5-8c]. To my understanding (as implied by the datasheet) these would be 3 "core groups". My motherboard can configure turbo ratios also in this way, but it is a bit buggy.

[cyring]

My way would be to write different use cases and observe those E-cores MSR.

Despite Group is part of specs, it sounds like the coherency requirements of old dual/quad Core/Core2 when multi-sockets had to be resolved by software.

I'm also expecting impacts from Alderlake Xeon multi-sockets.

[cyring]

@BugReporterZ Hello,

I would like to check for non-regression on Alder lake, could you give a try to the develop branch ?

Please provide the output of corefreq-cli -s -n -m -n -M -n -B -n -k -n -C 1 -c 1

[BugReporterZ]

How much should we get ?

It depends on what it is supposed to be. If it's the true DDR4 frequency, it should be half the reported DDR4 speed, i.e. 1800 MHz. If it's the DDR4 memory's reference clock, it should be 100 MHz in my case. The memory controller should run at either DDR4 frequency (Gear 1 / 1800 MHz), 1/2 (Gear 2 / 900 MHz) or 1/4 (Gear 4 / 450 MHz, but this would be used only for DDR5 memory).

After reading the rest of the post, if it's the "Ring Interconnect", i.e. Ring frequency, then 4600 Mhz seems fine under light load conditions, but I wouldn't have associated it with the memory controller. When E-cores are loaded, it automatically downclocks to 3600 MHz and the Memory Controller screen does show that.

[cyring]

Indeed, with Alder Lake I'm showing the Ring Interconnect frequency; and it makes confusions.
It has to be rethought: set bus rate with true DDR frequency; and find another location in UI to show the Ring frequency

[cyring]

How much should we get ?

It depends on what it is supposed to be. If it's the true DDR4 frequency, it should be half the reported DDR4 speed, i.e. 1800 MHz. If it's the DDR4 memory's reference clock, it should be 100 MHz in my case. The memory controller should run at either DDR4 frequency (Gear 1 / 1800 MHz), 1/2 (Gear 2 / 900 MHz) or 1/4 (Gear 4 / 450 MHz, but this would be used only for DDR5 memory).

After reading the rest of the post, if it's the "Ring Interconnect", i.e. Ring frequency, then 4600 Mhz seems fine under light load conditions, but I wouldn't have associated it with the memory controller. When E-cores are loaded, it automatically downclocks to 3600 MHz and the Memory Controller screen does show that.

If E-cores are loaded then bus rate is 3600 else 4600 : you mean E-cores enabled in BIOS ?

[BugReporterZ]

If any E-core is processing data, Ring frequency is automatically decreased to 3600 MHz due to stability issues at faster speeds. Disabling E-cores completely in BIOS can be a way for keeping Ring frequency fixed to 4600 or more when overclocking.

Ring frequency can be 4600 Mhz when E-cores are enabled, if none of them is processing any data (i.e. is not active).

This means that Ring frequency normally switches numerous times between 3600 and 4600 MHz depending on CPU load.

[cyring]

Thank you for your explanations: that's something not specified in Alder lake datasheet.

[cyring]

@BugReporterZ : Could try this attached version ?

CoreFreq_develop.tar.gz

I'm trying to fix the active channel in the corefreq-cli -M

Thank you

[cyring]

I now have 4 DIMMs populated. However, results from the latest CoreFreq from the development branch differ from expectations:

Hello,

Using your 4 DIMMs, can you print the output from this attached archive:

CoreFreq_develop.tar.gz

I'm expecting 2 controllers, 2 channels, 1 DIMM slot each with geometries distributed among channels.

[BugReporterZ]

It looks like I'm getting less data in the DIMM geometry section:

./corefreq-cli -M
                             Intel Z690  [7A84]                            
Controller #0                                                Dual Channel  
 Bus Rate  3600 MHz       Bus Speed 3609 MHz           DDR4 Speed 3609 MHz 
                                                                           
 Cha    CL  RCD   RP  RAS RRDs RRDl  FAW   WR RTPr WTPr  CWL CKE  CMD  GEAR
  #0    18   22   22   42    4    9   40   24   12   46   18   7   2T     1
  #1     0    0    0    0    0    0    0    0    0    0    0   0   1T     1
      sgRR dgRR drRR ddRR      sgRW dgRW drRW ddRW      sgWR dgWR drWR ddWR
  #0     7    4   12   12        14   15   15   16        36   29    7    7
  #1     0    0    0    0         0    0    0    0         0    0    0    0
      sgWW dgWW drWW ddWW                     REFI  RFC  XS   XP CPDED  ECC
  #0     7    4   12   12                    14055  630  648   11    4    0
  #1     0    0    0    0                        0    0    0    0    0    0
                                                                           
 DIMM Geometry for channel #0                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0                                                                  
 DIMM Geometry for channel #1                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0                                                                  
                                                                           
Controller #1                                                Dual Channel  
 Bus Rate  3600 MHz       Bus Speed 3609 MHz           DDR4 Speed 3609 MHz 
                                                                           
 Cha    CL  RCD   RP  RAS RRDs RRDl  FAW   WR RTPr WTPr  CWL CKE  CMD  GEAR
  #0    18   22   22   42    4    9   40   24   12   46   18   7   2T     1
  #1     0    0    0    0    0    0    0    0    0    0    0   0   1T     1
      sgRR dgRR drRR ddRR      sgRW dgRW drRW ddRW      sgWR dgWR drWR ddWR
  #0     7    4   12   12        14   15   15   16        36   29    7    7
  #1     0    0    0    0         0    0    0    0         0    0    0    0
      sgWW dgWW drWW ddWW                     REFI  RFC  XS   XP CPDED  ECC
  #0     7    4   12   12                    14055  630  648   11    4    0
  #1     0    0    0    0                        0    0    0    0    0    0
                                                                           
 DIMM Geometry for channel #0                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0                                                                  
 DIMM Geometry for channel #1                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0                                                                

[cyring]

It looks like I'm getting less data in the DIMM geometry section:

Thank you. Can you try with this version ?

CoreFreq_develop.tar.gz

[BugReporterZ]

DIMM Geometry data from one memory controller appears to be now working.

user@linux:~/Downloads/CoreFreq> ./corefreq-cli -M
                             Intel Z690  [7A84]                            
Controller #0                                                Dual Channel  
 Bus Rate  3600 MHz       Bus Speed 3609 MHz           DDR4 Speed 3609 MHz 
                                                                           
 Cha    CL  RCD   RP  RAS RRDs RRDl  FAW   WR RTPr WTPr  CWL CKE  CMD  GEAR
  #0    18   22   22   42    4    9   40   24   12   46   18   7   2T     1
  #1     0    0    0    0    0    0    0    0    0    0    0   0   1T     1
      sgRR dgRR drRR ddRR      sgRW dgRW drRW ddRW      sgWR dgWR drWR ddWR
  #0     7    4   12   12        14   15   15   16        36   29    7    7
  #1     0    0    0    0         0    0    0    0         0    0    0    0
      sgWW dgWW drWW ddWW                     REFI  RFC  XS   XP CPDED  ECC
  #0     7    4   12   12                    14055  630  648   11    4    0
  #1     0    0    0    0                        0    0    0    0    0    0
                                                                           
 DIMM Geometry for channel #0                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0    16    2     65536      1024          16384  CMK32GX4M2D3600C18
 DIMM Geometry for channel #1                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0    16    2     65536      1024          16384  CMK32GX4M2D3600C18
                                                                           
Controller #1                                                Dual Channel  
 Bus Rate  3600 MHz       Bus Speed 3609 MHz           DDR4 Speed 3609 MHz 
                                                                           
 Cha    CL  RCD   RP  RAS RRDs RRDl  FAW   WR RTPr WTPr  CWL CKE  CMD  GEAR
  #0    18   22   22   42    4    9   40   24   12   46   18   7   2T     1
  #1     0    0    0    0    0    0    0    0    0    0    0   0   1T     1
      sgRR dgRR drRR ddRR      sgRW dgRW drRW ddRW      sgWR dgWR drWR ddWR
  #0     7    4   12   12        14   15   15   16        36   29    7    7
  #1     0    0    0    0         0    0    0    0         0    0    0    0
      sgWW dgWW drWW ddWW                     REFI  RFC  XS   XP CPDED  ECC
  #0     7    4   12   12                    14055  630  648   11    4    0
  #1     0    0    0    0                        0    0    0    0    0    0
                                                                           
 DIMM Geometry for channel #0                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0                                                                  
 DIMM Geometry for channel #1                                              
      Slot Bank Rank     Rows   Columns    Memory Size (MB)                
       #0                                                                  

[cyring]

Unfortunately Controller #1 does not show the 2 remaining populated DIMMs; I won't commit these changes.

[cyring]

@BugReporterZ Hello

I just learned intel i3-n305 has no P-core, but 8 E-core only.
Does your BIOS allow to deactivate P-core on your i7-12700k ?
If feasible, can you run CoreFreq in such configuration ?

[BugReporterZ]

No, unfortunately I must have at least one active P-core on my configuration.

[cyring]

@BugReporterZ

ICL, ADL, RPL micro-arch have MSR_CORE_UARCH_CTL per P-Core, R/W access

Can you please read it ?

rdmsr -a 0x541

Next, toggle its value

wrmsr -a 0x541 [0x0 or 0x1]

Remark: access may fail on E-Core CPUs

[BugReporterZ]

Here's what happened.

sudo rdmsr -a 0x541
CPU 0: 0
CPU 1: 0
CPU 2: 0
CPU 3: 0
CPU 4: 0
CPU 5: 0
CPU 6: 0
CPU 7: 0
CPU 8: 0
CPU 9: 0
CPU 10: 0
CPU 11: 0
CPU 12: 0
CPU 13: 0
CPU 14: 0
CPU 15: 0
rdmsr: CPU 16 cannot read MSR 0x00000541


sudo wrmsr -a 0x541 0x1
wrmsr: CPU 16 cannot set MSR 0x00000541 to 0x0000000000000001


sudo rdmsr -a 0x541
CPU 0: 1
CPU 1: 1
CPU 2: 1
CPU 3: 1
CPU 4: 1
CPU 5: 1
CPU 6: 1
CPU 7: 1
CPU 8: 1
CPU 9: 1
CPU 10: 1
CPU 11: 1
CPU 12: 1
CPU 13: 1
CPU 14: 1
CPU 15: 1


sudo wrmsr -a 0x541 00 (typo, but it worked)
wrmsr: CPU 16 cannot set MSR 0x00000541 to 0x0000000000000000


sudo rdmsr -a 0x541
CPU 0: 0
CPU 1: 0
CPU 2: 0
CPU 3: 0
CPU 4: 0
CPU 5: 0
CPU 6: 0
CPU 7: 0
CPU 8: 0
CPU 9: 0
CPU 10: 0
CPU 11: 0
CPU 12: 0
CPU 13: 0
CPU 14: 0
CPU 15: 0
rdmsr: CPU 16 cannot read MSR 0x00000541

[cyring]

Sounds great for a new setting to add into CoreFreq
I will make sure to read/write bit on P-Core CPUs only.

Have you observe other impacts when toggling bit ?
... performance; temperature, voltage, consumed energy ?

[cyring]

I'm not finding explanation about this bit in the SDM but I guess it is part of the Intel RAS.
In BIOS, it could be called DCU Scrubbing. Same name employed within Lenovo servers.

[BugReporterZ]

Difficult to tell if there are differences with that; that would need extensive testing.

[cyring]

@BugReporterZ Hello,

Can you please try the develop branch whose commit 9f71f4a allows to toggle the L1 Scrubbing on capable architectures.

You should find the new item L1 Scrubbing in the Technologies window as shown below

Toggling that feature bit has to to the same on all P-core CPUs
... and confirmed by rdmsr -a 0x541

[BugReporterZ]

It appears to work.

[cyring]

Thanks for your fast confirmation answer.

BTW, do you have voluntary disable EIST ? From BIOS ?

[BugReporterZ]

Yes, I turned that off and just enabled SpeedShift, which is supposed to be an improvement over it anyway.

[cyring]

Thank you for sharing that knowledge.

[cyring]

@BugReporterZ Hello,

Can you help with #466 ?