Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Loop runs only shows last result #735

Open
kotarou3 opened this issue Jul 9, 2023 · 2 comments · May be fixed by #736
Open

Loop runs only shows last result #735

kotarou3 opened this issue Jul 9, 2023 · 2 comments · May be fixed by #736

Comments

@kotarou3
Copy link

kotarou3 commented Jul 9, 2023

Expected behaviour

If I loop a test or suite with the environment variables TOTAL_LOOP_COUNT or TOTAL_LOOP_TIME, every successful test run should contribute a sample to the result graph, so I can get more confidence in the numbers (both average and stddev/spread)

Actual behaviour

Only the results from the last loop of each test are shown in the graph.

For example, look at the results in this run.
The bar graph claims N=3 for each result, but if you export it as JSON you'll see that there's >100 total test runs.

See below for the TOTAL_LOOP_COUNT=3 run that produced the above result:

TOTAL_LOOP_COUNT=3 command output 
PS D:\phoronix-test-suite> $env:TOTAL_LOOP_COUNT=3
PS D:\phoronix-test-suite> .\phoronix-test-suite.bat run x264
Starting the Phoronix Test Suite




x264 2022-02-22:
    pts/x264-2.7.0
    Processor Test Configuration
        1: Bosphorus 1080p
        2: Bosphorus 4K
        3: Test All Options
        ** Multiple items can be selected, delimit by a comma. **
        Video Input: 1


Phoronix Test Suite v10.8.4
System Information


  PROCESSOR:              AMD Ryzen Threadripper 3960X 24-Core @ 3.80GHz
    Core Count:           21
    Microcode:            5510300800000000
    Core Family:          Zen 2

  GRAPHICS:               AMD Radeon RX 5700 XT
    OpenCL:               OpenCL 2.1 AMD-APP (3380.6)
    Display Driver:       10.0.17763.1
    Monitor:              QEMU Motor;DELL U41M;H E73q;H E73q
    Screen:               2560x1440

  MOTHERBOARD:            (1.16.2-debian-1.16.2-1 BIOS)
    BIOS Version:         1.16.2-debian-1.16.2-1
    Audio:                AMD Streaming Audio Device + VB-Audio Hi-Fi Cable
    Network:              Intel PRO/1000 MT Desktop + Red Hat VirtIO

  MEMORY:                 2 x 16384 MB 0MHz QEMU

  DISK:                   150GB QEMU QEMU HDD Disk + 40GB QEMU HDD
    File-System:          NTFS

  OPERATING SYSTEM:       Microsoft Windows 10 Enterprise Evaluation Build 17763
    Kernel:               10.0.17763.1935 (x86_64)
    Security:             __user pointer sanitization: Disabled
                          + Retpoline: Full
                          + IBPB: Always
                          + STIBP: Enabled
                          + VBS: Disabled

    Would you like to save these test results (Y/n):
    Recently Saved Test Results:
       loop-test          [Today]
       windows            [Today]
       numa-1             [3 days old]
       superpositionlog   [3 days old]

    Enter a name for the result file:
    Enter a name for the result file: loop-test

Current Test Identifiers:
- TOTAL_LOOP_TIME=300

    Enter a unique name to describe this test run / configuration: TOTAL_LOOP_COUNT=3

If desired, enter a new description below to better describe this result set / system configuration under test.
Press ENTER to proceed without changes.

Current Description: AMD Ryzen Threadripper 3960X 24-Core testing with a (1.16.2-debian-1.16.2-1 BIOS) and AMD Radeon RX 5700 XT on Microsoft Windows 10 Enterprise Evaluation Build 17763 via the Phoronix Test Suite.

New Description:

x264 2022-02-22:
    pts/x264-2.7.0 [Video Input: Bosphorus 1080p]
    Test 1 of 3
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 1 Minute [01:26 AEST]
        Started Run 1 @ 01:25:57
        Started Run 2 @ 01:26:06
        Started Run 3 @ 01:26:13

    Video Input: Bosphorus 1080p:
        185.53
        185.19
        187.73

    Average: 186.15 Frames Per Second
    Deviation: 0.74%

    Frames Per Second > Higher Is Better
    TOTAL_LOOP_TIME=300 . 179.79 |===========================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 180.22 |============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 180.51 |============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.00 |==============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.13 |===============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.66 |===============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.79 |================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 183.80 |=================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.35 |==================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.50 |==================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.75 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.89 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 185.14 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 185.50 |====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_COUNT=3 .. 186.15 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 186.55 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 186.63 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 188.43 |========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 189.23 |=========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 190.32 |===========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 190.46 |===========================================================================================================================================================================================================================================================================================

x264 2022-02-22:
    pts/x264-2.7.0 [Video Input: Bosphorus 1080p]
    Test 2 of 3
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 1 Minute [01:26 AEST]
        Started Run 1 @ 01:26:21
        Started Run 2 @ 01:26:28
        Started Run 3 @ 01:26:36

    Video Input: Bosphorus 1080p:
        187.97
        191.2
        190.84

    Average: 190.00 Frames Per Second
    Deviation: 0.93%

    Frames Per Second > Higher Is Better
    TOTAL_LOOP_TIME=300 . 179.79 |===========================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 180.22 |============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 180.51 |============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.00 |==============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.13 |===============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.66 |===============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.79 |================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 183.80 |=================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.35 |==================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.50 |==================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.75 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.89 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 185.14 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 185.50 |====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_COUNT=3 .. 186.15 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 186.55 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 186.63 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 188.43 |========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 189.23 |=========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_COUNT=3 .. 190.00 |==========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 190.32 |===========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 190.46 |===========================================================================================================================================================================================================================================================================================

x264 2022-02-22:
    pts/x264-2.7.0 [Video Input: Bosphorus 1080p]
    Test 3 of 3
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 1 Minute [01:26 AEST]
        Started Run 1 @ 01:26:44
        Started Run 2 @ 01:26:51
        Started Run 3 @ 01:26:58

    Video Input: Bosphorus 1080p:
        188.26
        189.69
        189.87

    Average: 189.27 Frames Per Second
    Deviation: 0.47%

    Frames Per Second > Higher Is Better
    TOTAL_LOOP_TIME=300 . 179.79 |===========================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 180.22 |============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 180.51 |============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.00 |==============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.13 |===============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.66 |===============================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 182.79 |================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 183.80 |=================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.35 |==================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.50 |==================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.75 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 184.89 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 185.14 |===================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 185.50 |====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_COUNT=3 .. 186.15 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 186.55 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 186.63 |=====================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 188.43 |========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 189.23 |=========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_COUNT=3 .. 189.27 |=========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_COUNT=3 .. 190.00 |==========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 190.32 |===========================================================================================================================================================================================================================================================================================
    TOTAL_LOOP_TIME=300 . 190.46 |===========================================================================================================================================================================================================================================================================================

    Do you want to view the results in your web browser (Y/n):
    Would you like to upload the results to OpenBenchmarking.org (y/n): y
    Would you like to attach the system logs (lspci, dmesg, lsusb, etc) to the test result (y/n): n

    Results Uploaded To: https://openbenchmarking.org/result/2307090-NE-LOOPTEST574
    Do you want to launch OpenBenchmarking.org (Y/n): n
@kotarou3
Copy link
Author

kotarou3 commented Jul 9, 2023 •
edited

I just checked the raw results file and it seems like all runs are actually saved.
But it seems like it fails to merge the results together, and the results viewer doesn't show the unmerged results either

@kotarou3 kotarou3 changed the title Loop runs only save last result Loop runs only shows last result Jul 9, 2023
@kotarou3
Copy link
Author

kotarou3 commented Jul 10, 2023 •
edited

Correction: Only runs that have a unique (identifier, value) pair are saved, likely caused by this code.
That means for extremely consistent or low-precision tests where each run produces the exact same average result, later loops are not saved at all. This happens to me with pts/unigine-super-1.0.7 with Quality: Ultra

kotarou3 added a commit to kotarou3/phoronix-test-suite that referenced this issue Jul 10, 2023
@kotarou3
Don't clear results buffer between loops
d81a4c1 
When tests or suites are looped with TOTAL_LOOP_COUNT or TOTAL_LOOP_TIME, each loop would produce an independent result entry with identical identifiers (descriptions).
However, the results viewer and exporters silently ignore any results with duplicate identifiers.
This change combines the results from all loops into a single result entry to avoid the issue.

Note: This change may cause data loss in existing result files if they have multiple results with the same identifiers.
But since those multiple results were never "visible" to users in the first place, it's not a practical issue.

Fixes phoronix-test-suite#735
@kotarou3 kotarou3 linked a pull request Jul 10, 2023 that will close this issue
Open
kotarou3 added a commit to kotarou3/phoronix-test-suite that referenced this issue Jul 11, 2023
@kotarou3
Don't clear results buffer between loops
d60a577 
When tests or suites are looped with TOTAL_LOOP_COUNT or TOTAL_LOOP_TIME, each loop would produce an independent result entry with identical identifiers (descriptions).
However, the results viewer and exporters silently ignore any results with duplicate identifiers.
This change combines the results from all loops into a single result entry to avoid the issue.

Note: This change may cause data loss in existing result files if they have multiple results with the same identifiers.
But since those multiple results were never "visible" to users in the first place, it's not a practical issue.

Fixes phoronix-test-suite#735
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Don't clear results buffer between loops kotarou3/phoronix-test-suite
1 participant
@kotarou3