自动创建Tags的Bug

[Hi-luca-Gao]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

版本0.8

Compute Engine

flink

Minimal reproduce step

test表结构：
tEnv.executeSql("CREATE TABLE dim_log_2 (\n" +
" url STRING,\n" +
" ts BIGINT,\n" +
" color STRING,\n" +
" PRIMARY KEY (url) NOT ENFORCED" +
" ) WITH (\n" +
" 'merge-engine' = 'partial-update',\n" +
" 'changelog-producer' = 'input',\n" +
" 'tag.automatic-creation' = 'watermark',\n" +
" 'tag.creation-period' = 'daily',\n" +
" 'tag.num-retained-max' = '90'" +
");");
使用 WatermarkStrategy
.<>forBoundedOutOfOrderness(Duration.ofSeconds()).withTimestampAssigner（SerializableTimestampAssigner）

第一次数据处理，直接会触发paimon tryToCreateTags（）方法，此时的watermark为-9223372036854775808，因为此策略watermark默认200ms下发一次，这样会导致 this.periodHandler.normalizeToPreviousTag(time);这个方法构建的返回值为+1705471-09-26，因此tagName命名成为tag-1705471-09-26，后续会因此不再有正常的例如2024-05-17这种正常事件的数据触发自动创建Tags，因为1705471-09-26这个日期过于大。

此为问题代码：
//实际是因为Timestamp.fromEpochMillis上限导致这里计算错误，但是根本问题在于watermark的处理没有考虑到第一次由event去触发创建目录的时候，watermark还没有来得及更新。
public LocalDateTime normalizeToPreviousTag(LocalDateTime time) {
long mills = Timestamp.fromLocalDateTime(time).getMillisecond();
long periodMills = this.onePeriod().toMillis();
//此处导致错误
LocalDateTime normalized = Timestamp.fromEpochMillis(mills / periodMills * periodMills).toLocalDateTime();
return normalized.minus(this.onePeriod());
}

What doesn't meet your expectations?

希望paimon越来越好，paimon是数据集成的未来，
加油！！！！
希望有机会可以提供一些向你们学习的机会。

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

Thanks @Hi-luca-Gao for reporting. Can you use English?

I think we should ignore Long.MIN there.

[Hi-luca-Gao]

You're welcome. @JingsongLi
Yes

[Hi-luca-Gao]

@JingsongLi

I think this kind of repair will have unexpected dangers, so it should be watermark == Long.MIN.
A timestamp less than 00:00:00 on January 1, 1970 Greenwich Mean Time is a negative number.
This is a normal phenomenon。
for example：
-315585870 ===>1960-01-01T10:15:30+01:00