Create Sysbench tool and testcases using it for CPU/Memory/FileIO performance #3075

smit-gardhariya · 2023-11-28T12:20:34Z

This PR has 2 commits.

Create Sysbench Tool in LISA
Create testcases for CPU/Memory/FileIO performance using Sysbench tool

lisa/tools/sysbench.py

microsoft/testsuites/sysbench/sysbenchperf.py

lisa/tools/sysbench.py

microsoft/testsuites/sysbench/sysbenchperf.py

lisa/tools/sysbench.py

microsoft/testsuites/sysbench/sysbenchperf.py

lisa/tools/sysbench.py

squirrelsc · 2023-12-06T17:51:16Z

lisa/messages.py

@@ -125,6 +125,22 @@ class PerfMessage(MessageBase):
    role: str = ""
    test_result_id: str = ""

+    # Sysbench common params


The PerfMessage is the base class for all perf messages. Please either move them into existing perf message, or new perf messages. The fields shouldn't be tool oriented, they should be metrics oriented.

These are the common one for sysbench. I put them here as i dont want to have redundant params for all child PerfMessage classes.

I created new perf message dataclass now for sysbench for all those common params and inherited it for Disk/CPU/Memory perf dataclasses

The perf messages should be metrics oriented, instead of tool oriented. It's easy to create dashboard and explain by metrics concepts. It's the same to sysbench common fields. It looks many common fields are not help a lot as a "common" field, the tested metrics are the key information of a perf message. We need to abstract the values to perf metrics.

For example, it's easy to understand x_latency_ms in network tests, but hard to understand in CPU tests, and I'm not sure what does it mean in the mem tests. The "events" number is highly depends on the test scenarios and settings.

If you look at the sysbench output it will always give latency metrics like min/max/avg. That was the reason to put it under common sysbench metrics

lisa/tools/sysbench.py

squirrelsc · 2023-12-06T18:13:08Z

lisa/tools/sysbench.py

+            test_name,
+            other_fields,
+        )
+        print(subtest_msg)


remove the debug lines. You can add console notifier to print all messages in debug level.

It's still here, not be removed.

Done now. I added it later again and still pushed it.

squirrelsc · 2023-12-06T18:17:44Z

lisa/messages.py

@@ -168,6 +190,38 @@ class DiskPerformanceMessage(PerfMessage):
    randwrite_iops: Decimal = Decimal(0)
    randwrite_lat_usec: Decimal = Decimal(0)

+    # Sysbench FileIO params
+    read_io_per_sec: Decimal = Decimal(0)


Please try best to reuse above fields, like read_io_per_sec should be read_iops above. So the data from different tools can be compared together.

Please check all other fields too.

Make sure it's a common concept, instead of sysbench concept.

Create it in the Disk message, if it doesn't exist. But not group them like "sysbench params".

Will check on this

squirrelsc · 2023-12-06T18:19:54Z

lisa/messages.py

+
+@dataclass
+class MemoryPerformanceMessage(PerfMessage):
+    total_operations: Decimal = Decimal(0)


operations_per_sec is easier to understand, if it includes the time unit. And similar to other fields.

There are 2 field for memory perf message. One is total_operation and other is operations_per_second.

If so, the total_operation doesn't need here. It's hard to compare cross different settings. We don't need to save all details in sysbench output.

Removed it now

squirrelsc · 2023-12-06T19:15:00Z

lisa/messages.py

@@ -149,6 +165,12 @@ class PerfMessage(MessageBase):
 )


+@dataclass
+class CPUPerformanceMessage(PerfMessage):
+    cpu_max_prime: Decimal = Decimal(0)


Is it a test parameter to calculate max prime? If so, use a field like "benchmark", and value is "prime". So it's easy to extend to other perf methodologies.

It is max number till which sysbench will try to get the prime numbers to generate CPU stress. Default is 10000 i.e. sysbench cpu run would try to get prime number upto 10000.

It's a test approach. If it's set as a column name, it's hard to extend other tests in the same CPU table. So, the columns should be like below.

benchmark: prime_10000
max_cpu_speed: ???

lisa/tools/sysbench.py

squirrelsc · 2023-12-07T01:55:11Z

lisa/tools/sysbench.py

+        threads: int = 1,
+        events: int = 0,
+        time_limit: int = 10,
+        memory_block_size_in_kb: int = 1,


Those arguments are not set in test case level too, they should be removed, unless they are used. Similar for fild and cpu.

That is correct that we are not using them as of now as we are running with default values. There is plan to run the CPU/Memory perf test using sysbench in near future for our validation stack which would required to run sysbech to be run with different parameter.

Those args are necessary to expose all command-line parameter of sysbench.

I would suggest if we can keep this so we dont need or need minimal changes at tool level. Please let me know your thoughts on this

Let's add them back in "near future", when it's used. That would be clearer than add "all" now.

squirrelsc · 2023-12-11T20:09:02Z

microsoft/testsuites/sysbench/sysbenchperf.py

+        memory_access_mode: List[str] = ["seq", "rnd"]
+        sysbench = node.tools[Sysbench]
+
+        for op in memory_operation:


similar to fileio, please move the loop into the tool.

squirrelsc · 2023-12-11T20:18:08Z

lisa/tools/sysbench.py

+            raise LisaException(
+                f"Invalid memory_scope. Valid memory_scope: {valid_mem_scope}"
+            )
+        elif memory_oper not in valid_mem_operation:


Similar with fileio, use 3 if, instead of elif on unrelated checks.

squirrelsc · 2023-12-11T20:21:51Z

lisa/messages.py

+@dataclass
+class MemoryPerformanceMessage(PerfMessage, SysbenchPerfMessage):
+    total_operations: Decimal = Decimal(0)
+    total_mib_transferred: int = 0


Should it be total_mib_transferred_per_sec? It's easy to cross check on different test settings.

We have 2 fields for this. total_mib_transferred and mib_per_second

Here is the reference output. We are fetching both the figure under above metrics.

64190.27 MiB transferred (6417.03 MiB/sec)

It looks similar to total operations, it's not a metrics to measure perf. We don't need to collect all information. The comparable metrics is important. Other data can be found in log for troubleshooting.

This commit has changes to add sysbench as tool to lisa This is needed for CPU/Memory/Fileio perf test using sysbench tool. Signed-off-by: Smit Gardhariya <sgardhariya@microsoft.com>

This commit has changes to add testcases for CPU/Memory/FileIO using sysbench tool of LISA. Signed-off-by: Smit Gardhariya <sgardhariya@microsoft.com>

smit-gardhariya requested review from squirrelsc and LiliDeng as code owners November 28, 2023 12:20

smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 3 times, most recently from 4f150a9 to 225ce70 Compare November 28, 2023 12:49

LiliDeng reviewed Nov 28, 2023

View reviewed changes

lisa/tools/sysbench.py Show resolved Hide resolved

smit-gardhariya force-pushed the sgardhariya/sysbench_tool branch 4 times, most recently from 7caff91 to 7bf8623 Compare November 28, 2023 14:07

squirrelsc reviewed Nov 28, 2023

View reviewed changes