V3.5.16 Release Notes

New Features

1. Added support for bi-directional synchronization from MySQL to MySQL.
2. Added support for importing files of MongoDB relmig 1.3.0+ version.
3. Added support for oplog synchronization when MongoDB is used as the source.
4. Added support for fast filtering of source time fields.
5. Added display of task milestones and fast filtering in the task list page.

Feature Enhancements

1. Optimized the unwind node function and added support for setting the expansion method.
2. The full sync details page supports quick filtering of tables by table name keywords.

Issue Fixes

1. Fixed an issue where adjusting alarm settings sometimes affected the normal operation of tasks.
2. Fixed an issue where adding mining tables would directly display mining task exceptions for tasks.
3. Fixed an issue where after a cached task was deleted, the task startup still retrieved data from the deleted cached task.

Release v3.5.15

New Features:

1. Added support for MySQL master-slave architecture, enabling synchronization tasks to sync data properly during MySQL master-slave switch.
2. Added table-level breakpoint resume capability for data replication tasks, allowing tasks to resume synchronization directly from the unfinished tables upon restart.
3. Added support for quick setting of task and connection labels through drag-and-drop.

Feature Enhancements:

1. Optimized the unwind node function, adding support for setting expansion methods.
2. Added digital certificate signing for the Windows version of Agent, enhancing security during Agent installation.
3. Optimized the user center page.

Issue Fixes:

1. Fixed an issue where some events were not supported by Alibaba Cloud PolarDB MySQL, causing task errors.
2. Fixed an issue where full-sync tasks were completed, but the progress indicator in the statistics displayed less than 100%.

v3.5.11

What's Changed

• fix(join-node): create join node with nodes and edges, check pre node… by @11000100111010101100111 in #1436
• fix(iengine): 修复pdk state map会定时清理数据的问题 by @ply0011 in #1444
• fix(MongoDB): #140123 Delete HeartbeatEvent after event batch, becaus… by @HarsenLin in #1483
• Fix limit task release v3.5.6 by @ningmeng777 in #1487
• fix(MongoDB): #140123 Delete HeartbeatEvent after event batch, becaus… by @HarsenLin in #1491
• fix(iengine): Master-slave merge will occur java.util.NoSuchElementEx… by @ply0011 in #1495
• Merge release-v3.5.6 into main by @ply0011 in #1511
• fix: update workflow by @dreamcoin1998 in #1569
• feat(engine): Differentiation of Value Conversion in Isomorphic and Heterogeneous Tasks by @11000100111010101100111 in #1571
• fix(python-node): init python not load yaml package by @11000100111010101100111 in #1572
• fix: update workflow by @dreamcoin1998 in #1576
• fix: update workflow by @dreamcoin1998 in #1579
• fix: update workflow by @dreamcoin1998 in #1580
• fix(npe): #150548 no non empty judgment after heartbeat connection failure - develop by @11000100111010101100111 in #1582
• fix(engine): do codec when task is not isomorphism by @11000100111010101100111 in #1577
• feat:Support circuit breaker recovery notification to Feishu by @weiliang110100 in #1575
• Update mrci by @dreamcoin1998 in #1583
• fix: add workflow dispatch by @dreamcoin1998 in #1584
• fix(iengine): 修复主从合并出现DateTime类型处理错误 by @ply0011 in #1589
• fix(iengine): 外存配置toString输出时，如果是mongodb，则自动对密码进行脱敏 by @ply0011 in #1585
• fix #150538:After the instance is unsubscribed, the engine is still r… by @weiliang110100 in #1574
• Sam/test develop by @ply0011 in #1591
• fix(iengine-app): 修复union节点启动报错的问题 by @ply0011 in #1595
• fix: cloud ldp MasterDataModel delete failed by @weiliang110100 in #1590
• Fix load schema dev gavin by @11000100111010101100111 in #1592
• feature: preload schema by @mnianqi in #1578
• 主从合并节点，增加对共用关联键的分析，并加工到MergeInfo中，供后续目标节点操作合并时使用，以防止误删除关联键字段 by @ply0011 in #1598
• Feat mongodb support timeseries by @weiliang110100 in #1596
• Fix develop cron task limit by @ljvv7 in #1599
• Feat limit scheduled task by @weiliang110100 in #1601
• fix(qps): sample one event size from event batch list for qps'size by @11000100111010101100111 in #1603
• feature: support capped collection by @mnianqi in #1604
• fix:MigrateTypeFilterProcessorNode cannot be cast DatabaseNode by @weiliang110100 in #1609
• Fix develop 150648 harsen by @HarsenLin in #1611
• fix(tm): Shared mining will not be able to run if there is connection… by @ply0011 in #1613
• fix: by @jarad0628 in #1619
• Fix dev email by @mnianqi in #1620
• Feat develop 150616 harsen by @HarsenLin in #1625
• fix: show table schema for subaccount by @mnianqi in #1627
• fix: fix on chart to show counts for all viewable tasks when login by… by @mnianqi in #1622
• Update mrci by @dreamcoin1998 in #1630
• fix: circuit breaker recovery bug by @weiliang110100 in #1624
• fix(unit-test): mock function with ture params by @11000100111010101100111 in #1633
• fix: update mr-ci.yaml by @dreamcoin1998 in #1634
• fix(iengine):#150614 js节点迁移任务添加tapField没有添加pos by @ljvv7 in #1635
• Update mrci by @dreamcoin1998 in #1636
• fix: CircuitBreakerRecoveryTest test case by @weiliang110100 in #1640
• fix: CircuitBreakerRecoveryTest test case by @weiliang110100 in #1641
• fix(cocec): do codec anytime by @11000100111010101100111 in #1644
• fix(cocec): do codec anytime by @11000100111010101100111 in #1643
• fix(tm): execute processFieldToDB anyway by @11000100111010101100111 in #1647
• fix(tm): execute processFieldToDB anyway by @11000100111010101100111 in #1645
• fix(python): print all message info when execute ProcessBuilder shell to avoid process blocking by @11000100111010101100111 in #1638
• fix: update mrci-workflow by @dreamcoin1998 in #1639
• fix: update mrci-workflow by @dreamcoin1998 in #1660
• Fix develop email subject by @ljvv7 in #1659
• fix: #150671 Wrong number of engine tasks by @weiliang110100 in #1656
• fix: fix for retry start time by @mnianqi in #1651
• fix: #150714 by @jarad0628 in #1650
• Fix develop jarad by @jarad0628 in #1648
• Fix develop mongo sharelog by @ljvv7 in #1661
• Feat develop rm project by @ljvv7 in #1662
• Fix diff inspect develop by @ningmeng777 in #1652
• Update mr-ci.yaml by @dreamcoin1998 in #1665
• Fix develop circular dependency by @ljvv7 in #1663
• fix: fix for mongo edit test connection by @mnianqi in #1664
• Develop merge into release-v3.5.10 by @ply0011 in #1666
• Task cannot be started problem by @ply0011 in #1673
• fix agent alarm email missing title by @openlg in #1674
• Fix agent alarm missing title by @openlg in #1675
• fix: update mrci-workflow by @dreamcoin1998 in #1670
• Fix task scheduler memory bug by @ply0011 in #1677
• Master-slave merge nodes support changes in join key value by @ply0011 in #1669
• Master-slave merge nodes support changes in join key value by @ply0011 in #1668
• fix: #150708 Optimize node index query interface by @weiliang110100 in #1672
• fix:#150542 engine loss count by @weiliang110100 in #1682
• fix:#150542 engine loss count by @weiliang110100 in #1681
• Feat develop add dagcheck by @ljvv7 in #1676
• Sam/fix release v3.5.10 merge bug #150766 by @ply0011 in #1683
• Merge release v3.5.10 into main by @ply0011 in #1690
• fix(iengine): Fix pdk state map resource leak problem by @ply0011 in #1693
• fix(tm): Fix when save dummy-dummy task, tm will npe by @ply0011 in #1692
• add agent expire notify by @openlg in #1691
• Fix develop update commons text version by @ljvv7 in #1687
• fix(iengine): Master Slave Merge Node Writes duplicate data into embe… by @ply0011 in #1684
• Feat share dig develop by @ningmeng777 in #1678
• fix: sonarQ question by @weiliang110100 in #1686
• Fix cloud setting develop by @weiliang110100 in #1694
• Feat develop add dagcheck by @ljvv7 in #1695
• Feat develop add dagcheck by @ljvv7 in #1698
• Pre Release Merge Develop Into Main by @ply0011 in #1699
• fix external error by @ningmeng777 in #1700
• fix:merge node npe by @weiliang110100 in #1701

Full Changelog: v3.5.8...v3.5.11

Release v3.5.9

What's Changed

• fix: update workflow by @dreamcoin1998 in #1569
• feat(engine): Differentiation of Value Conversion in Isomorphic and Heterogeneous Tasks by @11000100111010101100111 in #1571
• fix(python-node): init python not load yaml package by @11000100111010101100111 in #1572
• fix: update workflow by @dreamcoin1998 in #1576
• fix: update workflow by @dreamcoin1998 in #1579
• fix: update workflow by @dreamcoin1998 in #1580
• fix(npe): #150548 no non empty judgment after heartbeat connection failure - develop by @11000100111010101100111 in #1582
• fix(engine): do codec when task is not isomorphism by @11000100111010101100111 in #1577
• feat:Support circuit breaker recovery notification to Feishu by @weiliang110100 in #1575
• Update mrci by @dreamcoin1998 in #1583
• fix: add workflow dispatch by @dreamcoin1998 in #1584
• fix(iengine): 修复主从合并出现DateTime类型处理错误 by @ply0011 in #1589
• fix(iengine): 外存配置toString输出时，如果是mongodb，则自动对密码进行脱敏 by @ply0011 in #1585
• fix #150538:After the instance is unsubscribed, the engine is still r… by @weiliang110100 in #1574
• Sam/test develop by @ply0011 in #1591
• fix(iengine-app): 修复union节点启动报错的问题 by @ply0011 in #1595
• fix: cloud ldp MasterDataModel delete failed by @weiliang110100 in #1590
• Fix load schema dev gavin by @11000100111010101100111 in #1592
• feature: preload schema by @mnianqi in #1578
• 主从合并节点，增加对共用关联键的分析，并加工到MergeInfo中，供后续目标节点操作合并时使用，以防止误删除关联键字段 by @ply0011 in #1598
• Feat mongodb support timeseries by @weiliang110100 in #1596
• Fix develop cron task limit by @ljvv7 in #1599
• Feat limit scheduled task by @weiliang110100 in #1601
• fix(qps): sample one event size from event batch list for qps'size by @11000100111010101100111 in #1603
• feature: support capped collection by @mnianqi in #1604
• fix: #150648 Fixed the abnormal stop of a task due to the short time … by @HarsenLin in #1610
• fix:MigrateTypeFilterProcessorNode cannot be cast DatabaseNode by @weiliang110100 in #1608
• fix: #150648 Add testcase by @HarsenLin in #1615
• fix(tm): Shared mining will not be able to run if there is connection… by @ply0011 in #1614
• fix: by @jarad0628 in #1618
• Fix v3.5.9 email by @mnianqi in #1621

Full Changelog: v3.5.8...v3.5.9

Release v3.5.8

What's Changed

• fix(python-node): skip file which name not contains setup.py by @11000100111010101100111 in #1526
• fix(python-node): skip file which name not contains setup.py by @11000100111010101100111 in #1524
• fix: by @jarad0628 in #1527
• Fix function retry retry start time dev gavin by @11000100111010101100111 in #1532
• fix(task-retry): when task retry and sign timestamp and not sign befor by @11000100111010101100111 in #1535
• Sam/feat develop sonar hazelcast base node by @ply0011 in #1537
• fix custom aggregate count bug by @ningmeng777 in #1538
• Fix v3.5.7 jarad by @jarad0628 in #1542
• fix(iengine): [sonar] 修复HazelcastBaseNode代码问题; by @ply0011 in #1543
• feat: by @jarad0628 in #1545
• fix: disable sonar check when schedule by @dreamcoin1998 in #1548
• fix: #150249 Fixed processor node measurement that still reports an i… by @HarsenLin in #1547
• feat: #148903 Optimize snapshot count, exec by async. by @HarsenLin in #1544
• feat(iengine): 删除不会再使用的类 by @ply0011 in #1546
• fix: Fixed HazelcastSourcePdkBaseNodeTest.DoAsyncTableCountMethodTest… by @HarsenLin in #1557
• Feat develop license by @mnianqi in #1552
• fix(PythonUtils): add full unit test case to covert sonarqube as 100% (dev) by @11000100111010101100111 in #1555
• Fix updateagent bug develop by @ningmeng777 in #1553
• Fix diff inspect develop by @ningmeng777 in #1562
• fix(python-util): uncreate mock file by @11000100111010101100111 in #1568
• Sam/fix develop 150260 by @ply0011 in #1567

Full Changelog: v3.5.7...v3.5.8

Release v3.5.7

What's Changed

• fix: #148971 Fixed an issue where the restart log disappeared within … by @HarsenLin in #1430
• fix(iengine): 修复共享挖掘，当内嵌文档有ObjectId时，写入会报错: 无法识别的类型 by @ply0011 in #1427
• fix(join-node): create join node with nodes and edges, check pre node if disabled or not. by @11000100111010101100111 in #1433
• fix(task-retry): when a task is retried, modify the task status while saving the task retry start time by @11000100111010101100111 in #1428
• fix(iengine): 修复pdk state map会定时清理数据的问题 by @ply0011 in #1443
• Fix develop harsen by @HarsenLin in #1447
• Fix develop harsen by @HarsenLin in #1452
• Feat(disabled-node): #149108 The legacy issue with node disabling is that if a JOIN node is disabled, it can only disable the entire link. Need to detect and disable a complete link. by @11000100111010101100111 in #1448
• fix: #149605 engine task thread has been cleared, but the task status… by @weiliang110100 in #1465
• fix(TM): #149040 After the user specified the engine to start, the current maximum number of startup tasks was not set, resulting in task startup failure by @11000100111010101100111 in #1460
• perf(tapdata): Modify github codeowners file by @ply0011 in #1466
• Feat event data filter dev by @11000100111010101100111 in #1458
• fix: #6196 【V3.1】JS试运行，数据源连接测试正常，但是试运行JS报错：Can not get data from sour… by @mnianqi in #1474
• Feat share miner character by @ljvv7 in #1469
• fix(DAGUtil): npe in sonarqube. by @11000100111010101100111 in #1475
• Feat aggr release v3.5.5 by @ningmeng777 in #1472
• Feat add table log jarad by @jarad0628 in #1471
• fix: #149213 spring circular dependency by @weiliang110100 in #1479
• fix: type cast error by @ljvv7 in #1468
• fix mongodb include filed cast exception by @ningmeng777 in #1480
• Fix python util sonarqube develop by @11000100111010101100111 in #1477
• fix:JS试运行 标准js和增强js节点日志过滤 by @mnianqi in #1476
• fix: update cicd by @dreamcoin1998 in #1478
• fix(MongoDB): #140123 Delete HeartbeatEvent after event batch, becaus… by @HarsenLin in #1482
• Fix ci by @dreamcoin1998 in #1488
• Revert "Fix ci" by @dreamcoin1998 in #1493
• Feat connector download progress develop by @weiliang110100 in #1489
• Fix thread group util sonarqube develop by @11000100111010101100111 in #1485
• Feat add jacoco for unit test coverage by @jackin-code in #1501
• fix: show unittest coverage by @dreamcoin1998 in #1503
• Sam/feat develop sonar hazelcast base node by @ply0011 in #1509
• Sam/fix develop merge node by @ply0011 in #1496
• Fix optimize ci by @dreamcoin1998 in #1510
• fix: stop testsigma test by @dreamcoin1998 in #1515
• Feat cloud send mail limit develop by @weiliang110100 in #1507
• fix: prevent skip unittest by @dreamcoin1998 in #1516
• Fix develop inspect service by @mnianqi in #1517
• Fix node dispaly bug develop by @ningmeng777 in #1514
• fix:Clean files after testing by @weiliang110100 in #1508
• fix #149215 inspect progress bug by @ningmeng777 in #1505
• fix(MongoDB): #149167 Fixed CDC restart loss data. Fixed CDC batch ev… by @HarsenLin in #1500
• feat(qps): qps with memory to show by @11000100111010101100111 in #1519
• feat: add clean TTL Rules by @weiliang110100 in #1518
• fix(QPS): write record with outputSizeSpeed by @11000100111010101100111 in #1520

Full Changelog: v3.5.5...v3.5.7

Release v3.5.5

What's Changed

• Develop agent circuit breaker by @weiliang110100 in #1370
• Feat log info v3.5.4 by @11000100111010101100111 in #1380
• Feat log info develop by @11000100111010101100111 in #1383
• Feat node disable v3.5.4 by @11000100111010101100111 in #1382
• feature(checkhttp):Engine and task reporting heartbeats are switched … by @weiliang110100 in #1371
• Develop shared cache by @weiliang110100 in #1384
• Feat table of pk nopk v3.5.4 by @11000100111010101100111 in #1386
• Feat filter with unique index develop by @11000100111010101100111 in #1387
• Fix filter table unwind develop by @11000100111010101100111 in #1390
• fix(tm): filtering table with unique indexs by @11000100111010101100111 in #1389
• [HA] 数据源名字过长, 导致共享挖掘任务报错] by @ply0011 in #1392
• fix: 任务使用动态调整内存时报错java.lang.IllegalArgumentException - TapdataEvent{… by @mnianqi in #1391
• Fix v3.5.4 cicd by @dreamcoin1998 in #1394
• Fix v3.5.4 cicd by @ply0011 in #1395
• fix: update branch by @dreamcoin1998 in #1398
• Fix v3.5.4 cicd by @dreamcoin1998 in #1404
• fix(MongoDB): #140674 Fixed MongoDB target concurrent write record th… by @HarsenLin in #1407
• fix(MongoDB): #140674 Fixed MongoDB target concurrent write record th… by @HarsenLin in #1408
• Feat unwind node by @11000100111010101100111 in #1403
• fix: Failed to send engine offline notification by @weiliang110100 in #1412
• fix: update workflow by @dreamcoin1998 in #1405
• fix: #149083 spring circular dependency by @weiliang110100 in #1414
• fix(node): skip transform schema when node has disabled. by @11000100111010101100111 in #1416
• fix: print testsigma result link always by @dreamcoin1998 in #1415
• Develop cicd by @ply0011 in #1419
• LDP创建任务时TM端对源是否支持增量的判断逻辑有问题 by @cn-xufei in #1421
• fix(ignore-node): inignore and not remove node from DAG when node has disabled, and disabled node support each node type by @11000100111010101100111 in #1420
• 当共享挖掘任务删除重建后，会导致增量无法进行 by @ply0011 in #1423
• Release v3.5.5 merge to main by @ply0011 in #1424

Full Changelog: v3.5.4...v3.5.5

Release v3.5.4

What's Changed

• fix(py-node): not api document in python node by @11000100111010101100111 in #1351
• fix task run bug by @ningmeng777 in #1352

Full Changelog: v3.5.2...v3.5.4

Release v3.5.2

What's Changed

• Fix merge task cannot stop issue by @ply0011 in #1201
• fix: #148101 by @jarad0628 in #1204
• fix: #148085 by @jarad0628 in #1207
• fix(iengine): java内存占用过高 by @mnianqi in #1211
• Fix pg insertupdate v3.3 by @11000100111010101100111 in #1220
• fix: #148133 by @jarad0628 in #1215
• Fix v3.3 jarad by @jarad0628 in #1223
• fix #146986 redis database by @ningmeng777 in #1228
• add log by @ningmeng777 in #1224
• feature(iengine): 2.x kafka 支持 自定义消息体格式 by @mnianqi in #1231
• Fix v3.3 jarad by @jarad0628 in #1233
• fix: #148200 by @jarad0628 in #1237
• fix: tableComment with null by @jarad0628 in #1238
• fix: #148217 by @jarad0628 in #1239
• fix: by @jarad0628 in #1242
• Fix v3.3 jarad by @jarad0628 in #1243
• Fix v3.3 jarad by @jarad0628 in #1245
• fix: add dml policy by @jarad0628 in #1246
• add log by @ningmeng777 in #1247
• fix: mysql datetime 0000-00 to null by @jarad0628 in #1248
• fix: #147899 任务状态一直启动中但是有心跳 by @weiliang110100 in #1251
• Fix v3.3 jarad by @jarad0628 in #1257
• fix: no index will NPE by @jarad0628 in #1258
• Fix v3.3 sharecdc by @ply0011 in #1272
• fix: update tm/build/build.sh by @dreamcoin1998 in #1302
• fix mongodb pks filter bug by @ningmeng777 in #1303
• Feat python engine v3.5.2 by @11000100111010101100111 in #1333

Full Changelog: v3.5.1...v3.5.2

Release v3.5.1

What's Changed

• Fix v3.4 jarad by @jarad0628 in #1261
• Fix v3.4 jarad by @jarad0628 in #1263

Full Changelog: v3.4.1...v3.5.1