Cluster degradation in 8.8.3 (on tombstone eviction)

[macrergate]

Just tried to use 8.8.3 Gridgain Community release and came across following issue.
Under activity cluster became slower and slower
I should note that I don't use persistence - only in-memory caches
in all working threads I see this:
"pub-#158" #459 prio=5 os_prio=0 tid=0x00007fc4f401b000 nid=0x425d waiting on condition [0x00007fb8030ee000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00007fc8bcfe8398> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:837) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:872) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1202) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.lock0(StripedCompositeReadWriteLock.java:193) at org.apache.ignite.internal.util.StripedCompositeReadWriteLock$WriteLock.lock(StripedCompositeReadWriteLock.java:162) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.localPartition0(GridDhtPartitionTopologyImpl.java:983) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.localPartition(GridDhtPartitionTopologyImpl.java:866) at org.apache.ignite.internal.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.localPartition(GridCachePartitionedConcurrentMap.java:68) at org.apache.ignite.internal.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.putEntryIfObsoleteOrAbsent(GridCachePartitionedConcurrentMap.java:89) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:1000) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:455) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:991) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager.expire(PartitionsEvictManager.java:434) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expireTombstones(IgniteCacheOffheapManagerImpl.java:1420) at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:241) at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:187) at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:884) at org.apache.ignite.internal.processors.cache.GridCacheGateway.leaveNoLock(GridCacheGateway.java:243) at org.apache.ignite.internal.processors.cache.GridCacheGateway.leave(GridCacheGateway.java:228) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1618) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)

and also plenty of exceptions are generated and as I found thet are later catched and just ignored (see PartitionsEvictManager.expire ) but they looks suspicious, because no any rebalance was on the cluster)

ttl-cleanup-worker-#59" #137 prio=5 os_prio=0 tid=0x00007fc43404c800 nid=0xe1e runnable [0x00007fc4ec204000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.fillInStackTrace(Throwable.java:784) - locked <0x00007fc6ad3f7dd0> (a org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtInvalidPartitionException) at java.lang.Throwable.<init>(Throwable.java:266) at java.lang.Exception.<init>(Exception.java:66) at java.lang.RuntimeException.<init>(RuntimeException.java:62) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtInvalidPartitionException.<init>(GridDhtInvalidPartitionException.java:34) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.localPartition0(GridDhtPartitionTopologyImpl.java:1018) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtPartitionTopologyImpl.localPartition(GridDhtPartitionTopologyImpl.java:866) at org.apache.ignite.internal.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.localPartition(GridCachePartitionedConcurrentMap.java:68) at org.apache.ignite.internal.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.putEntryIfObsoleteOrAbsent(GridCachePartitionedConcurrentMap.java:89) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:1000) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:455) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:991) at org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager.expire(PartitionsEvictManager.java:434) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expireTombstones(IgniteCacheOffheapManagerImpl.java:1420) at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:241) at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.lambda$body$0(GridCacheSharedTtlCleanupManager.java:344) at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker$$Lambda$749/1430920913.apply(Unknown Source) at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1769) - locked <0x00007fcbb329c900> (a java.util.concurrent.ConcurrentHashMap$Node) at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:343) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) at java.lang.Thread.run(Thread.java:748)
I switched on debug logging and after I stopped activity on the cluster I still see inifinite loop of following messages:

`
DEBUG 02-04-2021 18:58:11,237 [sys-#312][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=10000, tombstone=true, size=10000]
DEBUG 02-04-2021 18:58:11,405 [sys-#313][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=0, tombstone=false, size=0]
DEBUG 02-04-2021 18:58:11,750 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=9000]
DEBUG 02-04-2021 18:58:11,769 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=8000]
DEBUG 02-04-2021 18:58:11,786 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=7000]
DEBUG 02-04-2021 18:58:11,804 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=6000]
DEBUG 02-04-2021 18:58:11,822 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=5000]
DEBUG 02-04-2021 18:58:11,840 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=4000]
DEBUG 02-04-2021 18:58:11,861 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=3000]
DEBUG 02-04-2021 18:58:11,880 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=2000]
DEBUG 02-04-2021 18:58:11,900 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=1000]
DEBUG 02-04-2021 18:58:11,911 [sys-#314][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=0, tombstone=false, size=0]
DEBUG 02-04-2021 18:58:11,919 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=0]
DEBUG 02-04-2021 18:58:11,926 [sys-#308][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=10000, tombstone=true, size=10000]
DEBUG 02-04-2021 18:58:12,416 [sys-#302][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=0, tombstone=false, size=0]
DEBUG 02-04-2021 18:58:12,453 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=9000]
DEBUG 02-04-2021 18:58:12,476 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=8000]
DEBUG 02-04-2021 18:58:12,498 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=7000]
DEBUG 02-04-2021 18:58:12,520 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=6000]
DEBUG 02-04-2021 18:58:12,542 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=5000]
DEBUG 02-04-2021 18:58:12,569 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=4000]
DEBUG 02-04-2021 18:58:12,600 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=3000]
DEBUG 02-04-2021 18:58:12,629 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=2000]
DEBUG 02-04-2021 18:58:12,669 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=1000]
DEBUG 02-04-2021 18:58:12,698 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=0]
DEBUG 02-04-2021 18:58:12,709 [sys-#311][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=10000, tombstone=true, size=10000]
DEBUG 02-04-2021 18:58:12,921 [sys-#306][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After filling the evict queue [res=0, tombstone=false, size=0]
DEBUG 02-04-2021 18:58:13,227 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=9000]
DEBUG 02-04-2021 18:58:13,250 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=8000]
DEBUG 02-04-2021 18:58:13,272 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=7000]
DEBUG 02-04-2021 18:58:13,294 [ttl-cleanup-worker-#75][o.a.i.i.p.c.d.d.t.PartitionsEvictManager] {} - After the expiration [cleared=1000, tombstone=true, remaining=6000]

`
It's also strange, I expected that expiration should have been finished at some moment.