[BUG] general degradation of the work of Iroha2

[timofeevmd]

OS and Environment

linux, k8s

GIT commit hash

2ffcd00

Minimum working example / Steps to reproduce

The test was carried out with different load intensity from 1 to 15 virtual users, the test result was similar - a systematic decrease in tps

testing was carried out in several approaches
I tried to apply loads from 1500 to 125 tps in order to find a TPS indicator with which we can work stably. But the bad news is that the overall degradation pattern of Iroha2 was observed in every test. The presence or absence of a profiler does not affect the degradation of the system

load model

1. assemble Iroha2 with the parca profiler, at the request of @BAStos525 @RamioMustafin
   the service is available via this link
2. start 4 virtual users, about 360tps and 3rps (if you turn off the profiler the indicator will be higher)

preconditions:
genesis.json contains
5 domains
1200 users
each user has 10,000 assets

test script
jenkins job to start load testing

params:

testScenario=load
simulationClass=jp.co.soramitsu.load.simulation.transaction.StabilitySimulation
logLevel=INFO
configuration=standard
intensity=4
rampDuration=4
stageDuration=3600

jenkins job to clean up load generator

1. sending an asset from one user to another without waiting for the transaction status

analytics tools
profiler

profiler before and after degradation
iroha2 current state
iroha2 TPS/queue status
iroha2 cluster monitoring
OpenSearch

Actual result

The expected TPS level is at least 350 throughout the scenario
There are no CPU or RAM leaks
Actual state of the world for each peer

Expected result

TPS reaches the desired value and then begins to decrease to minimum values
Based on the profiler data, I assume that Iroha spends a large amount of time updating the state of the world

Logs in JSON format

Log contents

Replace this text with a JSON log,
so it doesn't grow too large and has highlighting.

Who can help to reproduce?

No response

Notes

No response

[mversic]

verify with rc21