Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Combine mode sink only supports FIXED bucket mode, but table is DYNAMIC #3321

Open
2 tasks done
ljx240 opened this issue May 13, 2024 · 0 comments
Open
2 tasks done
Labels
bug Something isn't working

Comments

@ljx240
Copy link

ljx240 commented May 13, 2024

Search before asking

  • I searched in the issues and found nothing similar.

Paimon version

0.8.0

Compute Engine

flink

Minimal reproduce step

sync database from kafka topic, when the table bucket is -1,the error:java.lang.UnsupportedoperationException: Combine mode Sink only supports FIxED bucket mode, but org t main is DYNAMIC;

my command is:
flink run-application -t yarn-application
-Dyarn.application.name="paimon-load-realtime-orc-test"
-Dpipeline.name='paimon-canal-databasesync-orc-test'
-Djobmanager.memory.process.size=4096m
-Dtaskmanager.memory.process.size=4096m
-Dexecution.checkpointing.interval=60000
-Dexecution.checkpointing.timeout=60000
-Dexecution.checkpointing.tolerable-failed-checkpoints=3
-Dstate.backend.type='rocksdb'
-Dexecution.checkpointing.checkpoints-after-tasks-finish.enabled='true'
-Dpipeline.operator-chaining='false'
-Dstate.savepoints.dir='hdfs://mycluster/flink/savepoints'
-Dstate.checkpoints.dir='hdfs://mycluster/flink/checkpoints'
-Dstate.checkpoints.num-retained=100
-Dcluster.evenly-spread-out-slots='true'
/opt/bigdata/flink/lib/paimon-flink-action-0.8.0.jar
kafka-sync-database
--warehouse hdfs://mycluster/user/paimon/warehouse
--database paimon_odata
--table-prefix ""
--table-suffix ""
--kafka-conf properties.bootstrap.servers='node1:9092,node2:9092,node3:9092'
--kafka-conf topic="new_tables_canal"
--kafka-conf properties.security.protocol=SASL_PLAINTEXT
--kafka-conf properties.sasl.mechanism=PLAIN
--kafka-conf properties.sasl.jaas.config='org.apache.kafka.common.security.plain.PlainLoginModule required username="client" password="";'
--kafka-conf properties.group.id=paimon-canal-orc-test
--kafka-conf scan.startup.mode=earliest-offset
--kafka-conf value.format=canal-json
--catalog-conf metastore=hive
--catalog-conf uri=thrift://node2:9083
--table-conf bucket=-1
--table-conf changelog-producer=lookup
--table-conf sink.parallelism=32
--including_tables org_t_main

What doesn't meet your expectations?

cant sync table or database,the error:
image

Anything else?

No response

Are you willing to submit a PR?

  • I'm willing to submit a PR!
@ljx240 ljx240 added the bug Something isn't working label May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant