Replies: 8 comments 1 reply
-
|
After binlog is purge, it means that maxwell must complete the collection of the current binlog data, so the above problem will not occur (we have always used maxwell to collect rds data, but there has never been a problem). |
Beta Was this translation helpful? Give feedback.
-
It may be that the backup policies set by each company's RDS are inconsistent. Currently, our service error log shows that the binlog file no longer exists. Maxwell previously saved the binlog file list to the database, and by the time of polling and parsing, the file has been deleted。 |
Beta Was this translation helpful? Give feedback.
-
|
先要确定的问题是：maxwell消费binlog是一个增量、实时的行为，你可以理解为主从复制，maxwell消费的速率好比是主从延迟，比如你的binlog purge时间为4个小时，如果maxwell找不到被purge的binlog，那就证明主从maxwell消费延迟已经大于4个小时了，这种场景基本是不存在的，除非你手动set binlog position 到一个被purge的binlog中，这样会存在你说的问题，解决的方式是重置clientid或者清空positions表，让maxwell从最尾端的点位开始消费。（当binlog purge时，maxwell一定是消费完这个binlog了，如果找不到，可能存在大事务，这个事务已经超过了binlog purge的时间，但是这种情况极少） |
Beta Was this translation helpful? Give feedback.
-
|
这个应该不是maxwell的问题，可能是ds存储设置binlog文件有效期的问题，1个设置是binlog的最大文件个数，超过它的设置会自动删除，再有总得存储磁盘空间比如我们是1.2T，到了报警点1.15T的时候rds也会清掉老的文件，这个时候可能maxwell还没解析到这个文件，但是把这个meta信息存到了待处理的文件列表。 |
Beta Was this translation helpful? Give feedback.
-
|
除非你的binlog增长速度超过了maxwell消费的速度，但是这种情况极少见 |
Beta Was this translation helpful? Give feedback.
-
|
看1298后面评论 这个错误应该有多人遇到 看看怎么优化一下解决一下吧 。 |
Beta Was this translation helpful? Give feedback.
-
rds清除binlog的规则 |
Beta Was this translation helpful? Give feedback.
-
|
golang-mysql process binlog evevt also exist the problem， copy the issue Because the cloud platform database will clean the binlog to the cloud storage after a certain period of time, the local binlog will disappear, and the bigger the binlog, the faster it will disappear. In order to ensure the continuity of consumption, PR #741 is added in case that the abnormal consumption of sites leads to the interruption of consumption, and if it takes a long time to repair, it will be impossible to continue to consume sites after recovery. When the binlog file disappears, the binlog file is downloaded and analyzed locally through the file download method, and then pushed to EventHandler for processing. Complete the automatic switch between offline binlog and online binlog to ensure the continuity of canal consumption. Each cloud platform can implement the file download method according to its own API |
Beta Was this translation helpful? Give feedback.
-
Sorry, I'm not a Java developer. I didn't look at the source code for Maxwell
I have a question whether Maxwell has implemented the function of downloading binlog files to locally disk then parsing them, and deleting files after processing。
Most Chinese users use Alibaba Cloud's RDS MySQL products。
It will set an expiration time for the binlog file. If the RDS storage space reaches a certain threshold or the number of files exceeds the limit, it will be automatically cleaned up, and then maxwell will trigger the following error
Error:
23:13:15,284 INFO BinlogConnectorReplicator - Binlog connected.
23:13:15,285 WARN BinlogConnectorReplicator - communications failure in binlog:
com.github.shyiko.mysql.binlog.network.ServerException: Could not find first log file name in binary log index file
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:926) [mysql-binlog-connector-java-0.23.2.jar:0.23.2]
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:595) [mysql-binlog-connector-java-0.23.2.jar:0.23.2]
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:839) [mysql-binlog-connector-java-0.23.2.jar:0.23.2]
at java.lang.Thread.run(Thread.java:834) [?:?]
23:13:15,304 INFO BinlogConnectorReplicator - Binlog disconnected.
Beta Was this translation helpful? Give feedback.
All reactions