[DISCUSS] Release Plan of ShardingSphere 5.2.0

[strongduanmu]

Release 5.2.0

Hi community,

After the release of 5.1.2 version, we have received a lot of feedback from users and developers. Now we plan to release the ShardingSphere 5.2.0 version these days, which has made a lot of optimization and improvement for these feedback.

Please refer to this release notes to confirm whether it contains the features you expect. If I missed anything, please remind me ASAP. If there are any suggestions, please feel free to tell us.

Mail: https://lists.apache.org/thread/rr474o9hk3nng6ty9cwtcxsr1vlvgrmc

Release Notes

New Feature

1. Support SQL audit for sharding feature
2. Support MySQL show processlist and kill process list id feature
3. Scaling: Add dedicated DistSQL for data migration
4. Scaling: Basic support migrate data to heterogeneous database
5. DistSQL: New syntax CREATE MIGRATION PROCESS CONFIGURATION
6. DistSQL: New syntax ALTER MIGRATION PROCESS CONFIGURATION
7. DistSQL: New syntax SHOW MIGRATION PROCESS CONFIGURATION
8. DistSQL: New syntax ADD MIGRATION SOURCE RESOURCE
9. DistSQL: New syntax SHOW SQL_TRANSLATOR RULE
10. DistSQL: New syntax CREATE SHARDING AUDITOR
11. DistSQL: New syntax ALTER SHARDING AUDITOR
12. DistSQL: New syntax SHOW SHARDING AUDIT ALGORITHMS

Enhancement

1. Support column visible feature for MySQL, Oracle, SQLServer and H2.
2. Support cartesian product configuration for read write splitting
3. Support spring namespace and spring boot usage for sql translator
4. Support JSR-310 Year and Month in IntervalShardingAlgorithm
5. broadcast table support update/delete limit statement
6. Support create index on table(column) statement rewrite when config encrypt
7. Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
8. Support encrypt column rewrite when execute column is null in predicate
9. Support encrypt show create table return logic columns
10. Support create table with index statement rewrite when config encrypt
11. Support PostgreSQL create operator statement parse
12. Support PostgreSQL create materialized view statement parse
13. Support PostgreSQL nested comments parse
14. Support PostgreSQL alter subscription statement parse
15. Support PostgreSQL create group statement parse
16. Support PostgreSQL alter statictics statement parse
17. Support PostgreSQL create foreign table statement parse
18. Support PostgreSQL alter server statement parse
19. Support PostgreSQL create foreign data wrapper statement parse
20. Support PostgreSQL create event trigger statement parse
21. Support PostgreSQL security label statement parse
22. Support PostgreSQL reindex statement parse
23. Support PostgreSQL reassign owned statement and refresh materialized view statement parse
24. Support PostgreSQL prepare transaction statement parse
25. Support PostgreSQL create collation statement parse
26. Support PostgreSQL lock statement parse
27. Support PostgreSQL alter rule statement parse
28. Support PostgreSQL notify statement parse
29. Support PostgreSQL unlisten statement parse
30. Support Oracle alter function and alter hierarchy statement parse
31. Support Oracle alter pluggable database statement parse
32. Support Oracle alter materialized view log statement parse
33. Support Oracle alter diskgroup statement parse
34. Support Oracle alter operator statement parse
35. Support oracle alter cluster statement parse
36. Support oracle alter audit policy statement parse
37. Support Oracle alter index type statement parse
38. Support Oracle lock table statement parse
39. Support Oracle alter java statement parse
40. Support Oracle inline constraint statement parse
41. Support openGauss geometric operator statement parse
42. Optimize MySQL visible/invisible parse of create/alter table statements
43. Support scope of variable prefixed with @@ in MySQL SET statement parse
44. Support MySQL create procedure with create view parse
45. Support column segments parse in create index on table statement
46. Support openGauss cursor, fetch, move, close statement for sharding, readwrite-splitting
47. Support encrypt column rewrite when execute column is null in predicate
48. Support encrypt show create table return logic columns
49. Support create table with index statement rewrite when config encrypt
50. Support parsing ALTER LOCKDOWN PROFILE in Oracle
51. Support parsing ALTER MATERIALIZED VIEW in Oracle
52. Support parsing ALTER MATERIALIZED ZONEMAP in Oracle
53. Support parsing ALTER LIBRARY in Oracle
54. Support parsing ALTER INMEMORY JOIN GROUP in Oracle
55. Support parsing DROP OPERATOR in Oracle
56. Support parsing DROP RESTORE POINT in Oracle
57. Support parsing CREATE RESTORE POINT in Oracle
58. Support parsing DROP INMEMORY JOIN GROUP in Oracle
59. Support parsing create_bit_xor_table in MySQL
60. Support parsing MySQL DO statement.
61. Support parsing DropServer in openGauss
62. Support parsing CREATE AGGREGATE In openGauss
63. Support parsing ALTER ROUTINE in PostgreSQL
64. Add PostgreSQL Create Cast Statement
65. Add PostgreSQL Create Aggregate Statement
66. Support fetch/move/close cursor statement in PostgreSQL
67. Support Parsing ALTER PUBLICATION in PostgreSQL
68. Add PostgreSQL Create Access Method Statement
69. Support Parsing ALTER POLICY in PostgreSQL
70. Support parsing ALTER OPERATOR in PostgreSQL
71. Add PostgreSQL Copy Statement
72. Add PostgreSQL Comment Statement
73. Support listen statement in postgreSQL
74. Support DECLARE cursor statement
75. add default serverConfig in helm charts
76. Assemble openGauss JDBC Driver into Proxy distribution
77. ShardingSphere-Proxy listen on specified IP addresses
78. Support COM_STMT_SEND_LONG_DATA in MySQL Proxy
79. SELECT VERSION() support alias in MySQL Proxy
80. Fix openGauss Proxy could not be connected if no resource defined
81. Support using JRE defined in JAVA_HOME in ShardingSphere-Proxy's startup script
82. Avoid client blocked when OOM occurred in ShardingSphere-Proxy
83. Support using c3p0 in ShardingSphere-JDBC
84. Support SET NAMES with value quoted by double-quote
85. Connection.prepareStatement with columns arguments is available in ShardingSphere-JDBC
86. Scaling: Improve MySQL connect and reconnect
87. Scaling: Fix MySQL json column may cause leak at incremental task
88. Scaling: Add permission check for PostgreSQL data sources
89. Scaling: Incremental migration support for MySQL MGR mode
90. Scaling: Improve job progress persistence
91. Scaling: Start job DistSQL execute and return synchronously
92. Scaling: Inventory migration support table has primary key and unique key
93. Scaling: Close unerlying ElasticJob when stopping job
94. Scaling: Improve logical replication slot name generation for PostgreSQL and openGauss
95. Scaling: Make query DistSQL could be executed when no database selected
96. DistSQL: Add worker_id to the result set of SHOW INSTANCE LIST & SHOW INSTANCE INFO
97. DistSQL: Improve the result of EXPORT DATABASE CONFIG
98. DistSQL: Support more databases for FORMAT SQL
99. DistSQL: Optimize the execution logic of CREATE TRAFFIC RULE
100. DistSQL: Add paramter writeDataSourceQueryEnabled for RDL READWRITE_SPLITTING RULE.
101. DistSQL: Support assistEncryptor for Encrypt RDL
102. DistSQL: Add sharding algorithm type check when CREATE SHARDING TABLE RULE
103. Support database discovery to configure multiple groups of high availability under the same logic database
104. Support ShardingSphere-Proxy to start up under empty logic library
105. Support for isolating EventBus events by instance
106. Support the database to detect changes in the master node and restart the detection heartbeat task
107. Support ShardingSphere-Proxy to generate new worker-id when re-registering in cluster mode
108. Thrown exception when inserting expression value in shadow column on executing insert
109. Support distributed transactions across multiple logical databases
110. Support executing truncate in XA & PostgreSQL
111. Support alter local transaction rule with DistSQL
112. Support global transaction manager
113. Delete support for branch transaction on proxy

Bug Fix

1. Fix single table metadata refresh error caused by filtering DataSourceContainedRule
2. Fix parsing exception caused by the null value of MySQL blob type
3. Fix PostgreSQL/openGauss reset statement parse error
4. Fix wrong parameter rewrite when use sharding and encrypt
5. Fix the failed conversion of Month related classes on IntervalShardingAlgorithm
6. Fix NullPointerException when execute select union statement contains subquery
7. Fix wrong encrypt rewrite result due to incorrect order of metadata
8. fix MySQL trim function parse error
9. fix MySQL insert values with _binary parse error
10. Fix MySQL syntax error cannot be thrown to client
11. Avoid EventLoop blocked because of closing JDBC resources
12. Correct server status flags returned by MySQL Proxy
13. Fix a possible connection leak issue if Proxy client disconnected in transaction
14. Fixed a possible consistency issue with the statement being executed when the Proxy client is disconnected
15. Avoid pooled connection polluted by executing SET statements
16. Make SHOW TABLES FROM work in ShardingSphere-Proxy
17. Fix PostgreSQL DDL could not be executed by Extended Query
18. Fix SHOW VARIABLES could not be executed in PostgreSQL Proxy without resource
19. Fix FileNotFoundException when use ShardingSphere Driver with SpringBoot fatjar
20. Scaling: Fix the problem that the table contains both primary key and unique index at inventory migration
21. Scaling: Improve incremental migration, support the latest position in the middle of batch insert event
22. Scaling: Fix the error caused by null field value in openGauss incremental migration
23. DistSQL: Fix incorrect strategy name in result of SHOW SHARDING TABLE RULES
24. DistSQL: Fix current rule config is modified in advance when ALTER SHARDING TABLE RULE
25. DistSQL: Fix connection leak when ALTER RESOURCE
26. DistSQL: Fix CREATE TRAFFIC RULE failed when load balance algorithm is null
27. Fix that the monitoring heartbeat task was not stopped when the database was discovered and the logical library was deleted
28. Fix cluster mode ShardingSphere-JDBC load all logic database
29. Fix worker-id generated by SnowflakeKeyGenerateAlgorithm in cluster mode may exceed the maximum value
30. Fix DistSQL adding shadow algorithm exception without shadow data source
31. Fix cross-database data source confusion caused by same data source name in multiple logical databases
32. Fix RUL DistSQL execution failure in transaction
33. Fix begin for PostgreSQL & openGauss
34. Agent: Fixed the error of null value in contextManager when collecting metric data

API Changes

1. Remove SQL passthrough to data source feature
2. Add new assistedQueryEncryptorName and remove QueryAssistedEncryptAlgorithm interface
3. Refactor readwrite-splitting api to improve user experience
4. Remove check-duplicate-table-enabled configuration
5. Remove useless config item show-process-list-enabled configuration
6. Scaling: Change keyword for part of data migration DistSQL
7. Scaling: Redesign part of data migration DistSQL
8. DistSQL: Unify parameter type specification
9. DistSQL: Split SHOW INSTANCE MODE to SHOW MODE INFO and SHOW INSTANCE INFO
10. DistSQL: Change DROP SCALING jobId to CLEAN MIGRATION jobId
11. DistSQL: Remove COUNT INSTANCE RULES
12. Add database found that high availability supports all the slave libraries to go offline, and the main library undertakes the read traffic configuration
13. SnowflakeKeyGenerateAlgorithm supports configuring worker-id in standalone mode
14. Replace sourceDataSourceName with productionDataSourceName in Shadow API Configuration
15. Authority: Remove deprecated native authority provider

Refactor

1. ShardingSphere metadata refactoring for splitting actual metadata and logical metadata
2. Use ConnectionContext, QueryContext to remove ThreadLocal in FetchOrderByValueQueuesHolder, TrafficContextHolder, SQLStatementDatabaseHolder and TransactionHolder
3. Modify the default value of the ShardingSphere-Proxy version in the helm chart
4. Docker container will exit if ShardingSphere-Proxy failed to startup
5. Helm Charts in ShardingSphere repository are transferred to sub-project shardingsphere-on-cloud
6. Scaling: Plenty of refactor for better code reuse
7. DistSQL: Add a new category named RUL
8. Refactor the schedule module and split it into cluster schedule and standalone schedule
9. Remove memory mode, keep standalone mode and cluster mode
10. Refactoring metadata table loading logic and persistence logic
11. Refactoring distributed locks to retain the most concise interface design
12. Refactor : Unify The Creation for Proxy Containers in IT from ENV Modules
13. Refactor : Unify The Configuration for container created by testcontainer