CDN trace expected behavior

[siyuanchai1999]

Hi, I ran cdn/202303/ trace according instruction here https://cachelib.org/docs/Cache_Library_User_Guides/Cachebench_FB_HW_eval/.
The problem is that it seems never to finish even though I changed the config to "numOps": 50000. Is this expected behavior? I

The command I used is

opt/cachelib/bin/cachebench --json_test_config cdn_trace_reag0c01_20230315_20230322.json \
					--progress_stats_file cdn-trace-progress.log \
					--report_ac_memory_usage_stats human_readable \
					--report_api_latenc

Here's my config.

{
    "cache_config": {
        "allocator": "LRU2Q",
        "cacheSizeMB": 2857,
        "htBucketPower": 24,
        "htLockPower": 10,
        "allocSizes": [
            128,
            216,
            232,
            248,
            272,
            296,
            320,
            496,
            536,
            768,
            824,
            1024,
            1096,
            1176,
            1264,
            1352,
            1448,
            1552,
            1664,
            1784,
            1912,
            2048,
            2192,
            2352,
            2520,
            2696,
            2888,
            3096,
            3312,
            3544,
            3792,
            4064,
            4352,
            4656,
            4984,
            5336,
            5712,
            6112,
            6544,
            7008,
            7504,
            8032,
            8600,
            9208,
            9856,
            10552,
            11296,
            12088,
            12936,
            13848,
            14824,
            15864,
            16976,
            18168,
            19440,
            20800,
            22256,
            23816,
            25488,
            27272,
            29184,
            31232,
            38272,
            40952,
            43824,
            46896,
            50184,
            53696,
            57456,
            61480,
            65848,
            70392,
            75320,
            80592,
            86240,
            92280,
            98744,
            105656,
            113056,
            120976,
            129448,
            138512,
            148208,
            158584,
            169688,
            181568,
            194280,
            207880,
            222432,
            238008,
            254672,
            272504,
            291584,
            312000,
            333840,
            357208,
            382216,
            408976,
            437608,
            468240,
            501016,
            655360,
            786432,
            917504,
            1048576,
            1572864,
            2097152,
            2621440,
            3145728,
            3670016,
            4194304
        ],
        "numPools": 1,
        "poolSizes": [
            1.0
        ],
        "__nvmCacheSizeMB": 129619,
        "__nvmCachePaths": [],
        "navyBlockSize": 4096,
        "navySegmentedFifoSegmentRatio": [
            1,
            1,
            1
        ],
        "navyReqOrderShardsPower": 0,
        "navyBigHashSizePct": 0,
        "navyHitsReinsertionThreshold": 1,
        "navyProbabilityReinsertionThreshold": 0,
        "navyReaderThreads": 128,
        "navyWriterThreads": 128,
        "navyCleanRegions": 6,
        "navyNumInmemBuffers": 6,
        "navyParcelMemoryMB": 65536,
        "navyDataChecksum": false,
        "lru2qHotPct": 20,
        "lru2qColdPct": 20,
        "truncateItemToOriginalAllocSizeInNvm": true,
        "memoryOnlyTTL": 7200,
        "navyRegionSizeMB": 256,
        "printNvmCounters": true,
        "useTraceTimeStamp": true,
        "tickerSynchingSeconds": 600.0
    },
    "test_config": {
        "enableLookaside": false,
        "generator": "piecewise-replay",
        "___numOps": 1000000000,
        "numOps": 50000,
        "numThreads": 24,
        "populateItem": true,
        "prepopulateCache": false,
        "traceFileName": "reag0c01_20230315_20230322_0.2000.csv",
        "replayGeneratorConfig": {
            "numAggregationFields": 3,
            "numExtraFields": 0,
            "statsPerAggField": {}
        },
        "cachePieceSize": 65536
    }
}

[siyuanchai1999]

output is like this

Welcome to OSS version of cachebench
I0328 01:34:25.561690 40718 ReplayGeneratorBase.h:218] [0] Opened trace file reag0c01_20230315_20230322_0.2000.csv
Total 1.20M ops to be run
E0328 01:34:25.624919 40722 TimeStampTicker.cpp:57] A thread is trying to update too many buckets at one time: 0 to 2798105
E0328 01:34:25.624946 40724 TimeStampTicker.cpp:57] A thread is trying to update too many buckets at one time: 0 to 2798105
01:34:25       0.00M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)

== PieceWiseReplayGenerator Stats in Recent Time Window (1678863569 - 1678863601) ==
Total Processed Samples: 0.00 million
getBytes    :   0.48 GB, getBytesPerSec    :   0.00 GB/s, success :   4.01%, full success:   4.00%
getBodyBytes:   0.48 GB, getBodyBytesPerSec:   0.00 GB/s, success :   3.97%, full success:   3.97%
egressBytes :   0.48 GB, ingressBytes:   0.43 GB, egressBytesPerSec :   0.00 GB/s, ingressBytesPerSec:   0.00 GB/s, ingressEgressratio:  11.80%
objectGet   :        706, objectGetPerSec   :        1 /s, success :  37.11%, full success:  33.14%

== PieceWiseReplayGenerator Stats in Recent Time Window (1678863600 - 1678864202) ==
Total Processed Samples: 0.04 million
getBytes    :  19.70 GB, getBytesPerSec    :   0.03 GB/s, success :  22.67%, full success:  22.24%
getBodyBytes:  19.66 GB, getBodyBytesPerSec:   0.03 GB/s, success :  22.63%, full success:  22.22%
egressBytes :  19.68 GB, ingressBytes:  15.28 GB, egressBytesPerSec :   0.03 GB/s, ingressBytesPerSec:   0.03 GB/s, ingressEgressratio:  22.37%
objectGet   :     36,940, objectGetPerSec   :       61 /s, success :  51.39%, full success:  41.07%
I0328 01:34:26.649105 40718 PieceWiseReplayGenerator.cpp:271] Thread 15 finish, skip
01:35:25       0.79M ops completed. Hit Ratio  82.75% (RAM  82.75%, NVM   0.00%)
01:36:25       0.79M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)
01:37:25       0.79M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)
01:38:25       0.79M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)
01:39:25       0.79M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)
01:40:25       0.79M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)
01:41:25       0.79M ops completed. Hit Ratio   0.00% (RAM   0.00%, NVM   0.00%)

[haowu14]

Hi, you can bypass this by changing "tickerSynchingSeconds" to 0 (or remove it from your cachebench config).

There is a known issue in ticker syncing that sometimes would cause the trace consumption to hang.