Repeatedly scanning files

[lurnagao-dahua]

Hi,
I have some doubts about TableRuntimeRefreshExecutor and DefaultOptimizingService.
TableRuntimeRefreshExecutor scans all files to confirm the partition of the table to be optimized.
DefaultOptimizingService will scans files filter by TableRuntimeRefreshExecutor.
So will it repeatedly scan the files that need optimization?

[zhoujinsong]

Yes, in the current optimizing process, both the evaluation and plan stages involve scanning files. During the evaluation stage, the system assesses whether tables need optimizing. This evaluation occurs periodically. When the evaluation determines that a table requires optimizing, the table's status changes from idle to pending, triggering the formal planning process.

In the future, we can attempt to optimize the evaluation process by having it determine whether tables need to be optimized based on certain states on the tables, eliminating the need to scan all files.

[lurnagao-dahua]

Thank you very much!