CPU goes crazy in ignite server

[billleyn023]

Greetings,

Our system works well before but recently CPU goes high and never get down.
The architecture is like that, we have three clients, five servers whose CPU have 16 cores per each.
The CPU of clients look good but the five servers are in very high cpu%.
After checking the thread dump, we can see all busy threads are from Query Thread Pool with prefix "query-#".

e.g.

"query-#115" #174 prio=5 os_prio=0 tid=0x000055c49d171000 nid=0x14b runnable [0x00007f30fb564000]
java.lang.Thread.State: RUNNABLE
at java.lang.StringCoding.decode(StringCoding.java:229)
at java.lang.String.(String.java:463)
at org.apache.ignite.internal.binary.BinaryObjectImpl.fieldByOrder(BinaryObjectImpl.java:428)
at org.apache.ignite.internal.binary.BinaryFieldImpl.value(BinaryFieldImpl.java:112)
at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.fieldValue(QueryBinaryProperty.java:223)
at org.apache.ignite.internal.processors.query.property.QueryBinaryProperty.value(QueryBinaryProperty.java:120)
at org.apache.ignite.internal.processors.query.h2.opt.GridH2RowDescriptor.columnValue(GridH2RowDescriptor.java:235)
at org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.getValue0(H2CacheRow.java:107)
at org.apache.ignite.internal.processors.query.h2.opt.H2CacheRow.getValue(H2CacheRow.java:91)
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:479)
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:72)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5377)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.findLowerBound(BPlusTree.java:5494)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5767)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.nextPage(BPlusTree.java:5607)
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.next(BPlusTree.java:5840)
at org.apache.ignite.internal.processors.query.h2.H2Cursor.next(H2Cursor.java:66)
at org.h2.index.IndexCursor.next(IndexCursor.java:316)
at org.h2.table.TableFilter.next(TableFilter.java:502)
at org.h2.table.TableFilter.next(TableFilter.java:541)
at org.h2.table.TableFilter.next(TableFilter.java:541)
at org.h2.command.dml.Select.queryGroup(Select.java:323)
at org.h2.command.dml.Select.queryWithoutCache(Select.java:628)
at org.h2.command.dml.Query.queryWithoutCacheLazyCheck(Query.java:114)
at org.h2.command.dml.Query.query(Query.java:352)
at org.h2.command.dml.Query.query(Query.java:333)
at org.h2.command.CommandContainer.query(CommandContainer.java:114)
at org.h2.command.Command.executeQuery(Command.java:202)
- locked <0x00007f3545d078b0> (a org.h2.engine.Session)
at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:114)
- locked <0x00007f3545d078b0> (a org.h2.engine.Session)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:821)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:914)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest0(GridMapQueryExecutor.java:407)
at org.apache.ignite.internal.processors.query.h2.twostep.GridMapQueryExecutor.onQueryRequest(GridMapQueryExecutor.java:241)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.onMessage(IgniteH2Indexing.java:2171)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.lambda$start$17(IgniteH2Indexing.java:2124)
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$$Lambda$379/2097614581.onMessage(Unknown Source)
at org.apache.ignite.internal.managers.communication.GridIoManager$ArrayListener.onMessage(GridIoManager.java:3477)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1907)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1528)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$5300(GridIoManager.java:241)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1421)
at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

our code is simplified like this: 100 requests per seconds

public void save(Order Order) {
    order.setLastModified(new Timestamp(System.currentTimeMillis()));
    this.getCache().put(workingOrder.getOrderId(), workingOrder);
}

My questions are:

1. where can I check SQL histories of the server? I checked the view SQL_QUERIES_HISTORY, but it only contains the SQLs executed from the connected client.
2. how we can troubleshoot such problem? what kind of metrics we can use? we run pure ignite server referring to https://ignite.apache.org/docs/2.9.1/installation/installing-using-docker

Any idea appreciated.