[Bug report] Doris table operation integration test failed occasionally

[FANNG1]

Version

PR: #3307

Describe what's wrong

doris table operation failed in ci occasionally, seems the state of doris is not healthy.

Failed ci: https://github.com/datastrato/gravitino/actions/runs/9025764657/job/24801995236.

Error message and/or stacktrace

DorisTableOperationsIT > testAlterTable() FAILED
    com.datastrato.gravitino.exceptions.GravitinoRuntimeException: errCode = 2, detailMessage = Table[doris_alter_test_table_9de02715]'s state is not NORMAL. Do not allow doing ALTER ops
        at app//com.datastrato.gravitino.catalog.doris.converter.DorisExceptionConverter.toGravitinoException(DorisExceptionConverter.java:66)
        at app//com.datastrato.gravitino.catalog.jdbc.operation.JdbcTableOperations.alterTable(JdbcTableOperations.java:247)
        at app//com.datastrato.gravitino.catalog.doris.integration.test.DorisTableOperationsIT.testAlterTable(DorisTableOperationsIT.java:184)

        Caused by:
        java.sql.SQLException: errCode = 2, detailMessage = Table[doris_alter_test_table_9de02715]'s state is not NORMAL. Do not allow doing ALTER ops
            at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
            at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
            at com.mysql.cj.jdbc.StatementImpl.executeUpdateInternal(StatementImpl.java:1333)
            at com.mysql.cj.jdbc.StatementImpl.executeLargeUpdate(StatementImpl.java:2106)
            at com.mysql.cj.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1243)
            at org.apache.commons.dbcp2.DelegatingStatement.executeUpdate(DelegatingStatement.java:330)
            at org.apache.commons.dbcp2.DelegatingStatement.executeUpdate(DelegatingStatement.java:330)
            at com.datastrato.gravitino.catalog.jdbc.utils.JdbcConnectorUtils.executeUpdate(JdbcConnectorUtils.java:28)
            at com.datastrato.gravitino.catalog.jdbc.operation.JdbcTableOperations.alterTable(JdbcTableOperations.java:244)

How to reproduce

CI

Additional context

No response

[FANNG1]

seems couldn't reproduce on main branch, but easy to reproduce on PR: #3307

[yuqi1129]

This bug is a known bug that occasionally occurs, but not frequently. The core reason is that alter operation in Doris is asynchronous and increasing the wait time will alleviate the problem, but it can't completely eliminate it.
The following are related similar cases:
https://github.com/datastrato/gravitino/actions/runs/9013450467/job/24764301966?pr=3100
https://github.com/datastrato/gravitino/actions/runs/9013889875/job/24765516186

[FANNG1]

seems not the same problem, the problem you mentioned is alter operation does not takes affect. but in this case, alter operation is not allowed for state is not NORMAL

[zhoukangcn]

Maybe it's a bug of Doris 1.2.7, in some cases, the Doris tablet is damaged.

I will track this issue.