1.2.0

Release InLong 1.2.0-incubating - Released (as of 2022-06-08)

Agent

ISSUE	Summary
INLONG-4483	[Bug][Agent] Many ConnectException logs in unit test of Kafka source
INLONG-4193	[Improve][Agent] Add Java docs
INLONG-4112	[Feature][Agent] Support collect data from a specified position for MySQL binlog
INLONG-2563	[Feature][Agent] Move public domain from agent to agent-common
INLONG-4397	[Feature][Agent] Supports collect of full data for file type
INLONG-4359	[Improve][Agent] Simplify agent process commands
INLONG-4292	[Improve][Agent][TubeMQ][Sort] Upgrade the property file for all modules from log4j to log4j2
INLONG-4283	[Bug][Agent] The kafka-clients with 3.0.0 does not package to the jar
INLONG-4235	[Bug][Agent] Config the log info for unit tests of the agent plugin

Audit

ISSUE	Summary
INLONG-4594	[Improve][Audit] Make Elasticsearch authentication configurable
INLONG-4520	[Improve][Audit] Audit-proxy supports Pulsar authenticate
INLONG-4477	[Improve][Audit] Audit-store supports Pulsar authenticate
INLONG-3895	[Bug][Audit] Proxy store startup script log path error
INLONG-3853	[Bug][Audit] audit can not start for script bug

Dashboard

ISSUE	Summary
INLONG-4612	[Improve][Dashboard][Manager] Iceberg sink configuration protocol key update
INLONG-4610	[Improve][Dashboard] Save sink fields failed
INLONG-4588	[Improve][Dashboard] Approval management splits different routes
INLONG-4580	[Bug][Dashboard] File source IP and Hive conf dir are mandatory
INLONG-4577	[Feature][Dashboard] Support TubeMQ cluster management
INLONG-4550	[Feature][Dashboard] Support DataProxy Cluster and Node management
INLONG-4544	[Improve][Dashboard] Pulsar Cluster support tenant param
INLONG-4523	[Feature][Dashboard] Support Clusters Module
INLONG-4500	[Improve][Manager][Dashboard] Remove non-null restriction on hiveConfDir parameter
INLONG-4436	[Bug][Dashboard] Hive Sink confDir parameter tooltip error
INLONG-4488	[Improve][Dashboard] Remove useless modules
INLONG-4423	[Improve][Dashboard] Hive Sink adds hiveConfDir parameter
INLONG-4319	[Improve][Dashboard] Approve page support cluster info
INLONG-4284	[Improve][Dashboard] Modify ClickHouse sink parameters
INLONG-4274	[Improve][Dashboard] Update the parameters to adapt the manager module
INLONG-4218	[Feature][Dashboard] Unified group page
INLONG-4119	[Improve][Dashboard] Change the keyWord to keyword in query request params
INLONG-4102	[Improve][Dashboard] Add tooltip for dataPath of the Hive sink
INLONG-4089	[Feature][Dashboard] Create ClickHouse sink need to fill more params
INLONG-4031	[Bug][Dashboard] Kafka sink storage topic partitionNum key error
INLONG-4028	[Bug][Dashboard] ClickHouse types do not match what ClickHouse using
INLONG-3938	[Bug][Dashboard] CI workflow build failed incidentally
INLONG-3851	[Improve][Dashboard] Unify the naming of data streams

DataProxy

ISSUE	Summary
INLONG-4521	[Improve][DataProxy][Manager] Change the naming of third-party cluster related classes
INLONG-4056	[Feature][DataProxy] Return the response to the SDK/Agent after saving the event to the cache cluster

Manager

ISSUE	Summary
INLONG-4648	[Improve][Manager] Read the path of Sort connectors from configuration file
INLONG-4626	[Improve][Manager] Remove redundant connector sub directory
INLONG-4622	[Bug][Manager] Append db name in JDBC URL for load node
INLONG-4615	[Improve][Manager] PluginClassLoader adapts to the Windows system
INLONG-4612	[Improve][Dashboard][Manager] Iceberg sink configuration protocol key update
INLONG-4607	[Bug][Manager] Add ClickHouse field types for FormatInfo ...

1.1.0

Release InLong 1.1.0-incubating - Released (as of 2022-4-15)

Agent

ISSUE	Summary
INLONG-3699	[Improve][Agent] Exclude mysql-connector-java
INLONG-3692	[Bug][Agent] There are many agent processes after recovering the directory
INLONG-3652	[Improve][Agent] Improve TestBinlogOffsetManager unit test
INLONG-3650	[Bug] Agent fix timeoffset npe
INLONG-3638	[Bug] Agent and DataProxy can not listen to the 8080 port for Prometheus
INLONG-3629	[Improve][Agent] Improve TestFileAgent unit test
INLONG-3620	[Improve] Update the file agent guide document
INLONG-3587	[Bug][Agent]Resource leak
INLONG-3476	[Bug][Agent] debezium 1.8.1 has npe
INLONG-3466	[Feature][Agent] Remove protobuf dependency
INLONG-3463	[Bug][Agent] Fix unit test of TestTaskWrapper
INLONG-3448	[Improve][Manager] Limit the number of Agent pull tasks
INLONG-3437	[Agent] Sort out the LICENSEs of the third-party components of inlong-agent
INLONG-3381	[Feature] Agent wait one minute for dataproxy to prepare topic config
INLONG-3349	[Feature] Agent add limitation for job number
INLONG-3335	[Bug] fix agent snapshot mode won't work and optimize jvm parameters
INLONG-3326	[Improve][Agent] The unit test for TestTaskWrapper was running too long
INLONG-3317	[Improve][Agent] Change agent heartbeat/report interval to 10s
INLONG-3308	[Bug][Agent] NPE occurred in parsing deliveryTime
INLONG-3306	[Feature][Agent] Use rocksdb as default db in agent
INLONG-3304	[Bug][Agent] Reader cost too much CPU
INLONG-3299	[Bug][Agent] Report job result rather than task result
INLONG-3298	[Feature][Agent] Remove dbd implementation
INLONG-3297	[Feature] Add version control in Agent CommandEntity
INLONG-3274	[Bug][Agent] When Kafka topic is deleted
INLONG-3271	[Bug][Agent] Cannot get localip in docker.sh
INLONG-3168	[Bug][Agent] Change the deserialization type from String to byte array
INLONG-3148	[Bug][Agent] fix avro serialization
INLONG-3104	[Bug][Agent] Add default value for kafka consumer group
INLONG-3100	[Bug][Agent] Upgrade Kafka to newest version 3.1.0
INLONG-3099	[Bug][Agent] Duplicate send message when agent receive data
INLONG-3083	[Bug][Agent] Upgrade Scala version in Kafka client
INLONG-3077	[Bug][Agent] FileNotFoundException occurred in unit tests
INLONG-3076	[Bug][Agent] MalformedObjectNameException occurred in unit tests
INLONG-3050	[Bug][Agent] Update guava version
INLONG-3045	[Feature][Agent] Add rocksDb implementation
INLONG-3027	[Feature][Agent] Upgrade snappy version
INLONG-3022	[Bug] agent pod start failed
INLONG-2985	[Bug][Manager] Fix task type and UTF question for agent
INLONG-2974	[Improve][Manager] Support agent to pull tasks without ip and uuid
INLONG-2933	[Bug][Agent][Manager] Change the type of the deliveryTime field from Date to String
INLONG-2908	[Bug][Agent] Delete uuid around space
INLONG-2894	[Improve][Agent] Adapt the interface and field modification of the Inlong-Manager
INLONG-2883	[Bug][Agent] ManagerFetcher throws exception when invoke the Gson.fromJson method
INLONG-2877	[Bug][Agent] Task position manager throws NPE when send dataproxy ack success
INLONG-2870	[Bug][Agent] Use base64 to encode snapshot instead of using iso-8859-1
INLONG-2860	[Feature][Agent] Create file folder when history file set by user does not exist
INLONG-2859	[Improve][Agent] Optimize stopping Kafka tasks
INLONG-2857	[Feature][Agent] Support to destroy task
INLONG-2851	[Feature] Agent change task id string to integer
INLONG-2826	[Bug] Agent mysql connection should set allowPublicKeyRetrieval to true to support mysql 8.0
INLONG-2818	[Bug] Agent kafka job and binlog job has jar conflict
INLONG-2790	[Bug][Agent] Log4j cannot be output due to jar conflict
INLONG-2788	[Feature] Agent support sync send data to dataproxy when needed (binlog etc.)
INLONG-2786	[Feature] Agent jetty server support different job type
INLONG-2779	[Feature] Agent support delete job using jetty server
[INLONG-2756](https://github.com/apache/incubator-inlong/is...

1.0.0

FEATURES:

ISSUE	Summary
INLONG-2347	[Feature]Support hive sink in sort-single-tenant
INLONG-2334	[Feature][inlong-dataproxy]create pulsar client need support config ioThreads
INLONG-2333	[Feature]Support clickhouse sink in sort-single-tenant
INLONG-2266	[Feature]Support reporting metrics by audit-sdk in sort
INLONG-2250	[Feature][InLong-Sort] Support Kafka sink in InLong-Sort
INLONG-2247	Read the consume group offset and store to the specified topic
INLONG-2236	[Feature]Support iceberg sink in sort-single-tenant
INLONG-2232	Add start and end timestamp of segment
INLONG-2218	[Feature][InLong-DataProxy] Inlong-DataProxy support authentication access pulsar
INLONG-2217	[Feature][InLong-DataProxy] Add TCP protocol client demo and config doc feature
INLONG-2216	[Feature][InLong-DataProxy] Add UDP protocol client demo and config doc
INLONG-2215	[Feature][InLong-DataProxy] Add http protocol client demo and config doc
INLONG-2326	[Feature] Inlong-Sort-Standalone support to sort the events to ElasticSearch cluster.
INLONG-2322	[Feature][InLong-Sort] Support json format for kafka sink
INLONG-2301	[Feature] Support Standalone deployment for InLong
INLONG-2207	[Feature][InLong-Website] Add component about charts
INLONG-2187	[Feature] Website support audit view
INLONG-2183	[Feature][InLong-Sort] Bump flink version to 1.13.5
INLONG-2176	Add histogram metric and client-side metric output
INLONG-2170	[Feature] add Inlong-Sort-standalone document.
INLONG-2169	[Feature] [Agent] should provide docs for agent db sql collect
INLONG-2167	[Feature] [Agent] support db SQL collect
INLONG-2164	[Feature] Sort-standalone expose metric data using prometheus HttpServer.
INLONG-2161	[Feature][InLong-Manager] Manager support getClusterConfig
INLONG-2138	[Feature] Agent should provide docs for programmers to customize their own source or sink
INLONG-2106	[Feature] DataProxy expose metric data using prometheus HttpServer.
INLONG-2096	[Feature] DataProxy add InlongGroupId+InlongStreamId metric dimensions in TDSDKSource and TubeSink.
INLONG-2077	[Feature]sort-sdk change pulsar consume mode from listener to fetch
INLONG-2076	[Feature] Tube sink of DataProxy support new Message format.
INLONG-2075	[Feature] SDK Source of DataProxy support new Message format.
INLONG-2058	[Feature] The metric of Sort-standalone append a dimension(minute level) of event time, supporting audit reconciliation of minute level.
INLONG-2056	[Feature]The metric of DataProxy append a dimension(minute level) of event time, supporting audit reconciliation of minute level.
INLONG-2055	[Feature] [InLong audit] Audit SDK Support real-time report
INLONG-2054	[Feature] [InLong audit] Audit SDK Support disaster tolerance
INLONG-2053	[Feature] [InLong audit] Audit Web Page Display
INLONG-2051	[Feature] [InLong audit] Add Audit API for Manager
INLONG-2050	[Feature] [InLong audit] Audit Strore for Elasticsearch
INLONG-2045	[Feature]sort-sdk support Prometheus monitor
INLONG-2028	[Feature][CI] Add support for docker build on GitHub Actions
INLONG-1992	[Feature]sort-flink support configurable loader of getting configuration.
INLONG-1950	[Feature] DataProxy add supporting to udp protocol for reporting data
INLONG-1949	[Feature] DataProxy sdk add demo
INLONG-1931	[Feature]Inlong-Sort-Standalone-readapi support to consume events from inlong cache clusters(tube)
INLONG-1895	[Feature]Inlong-Sort-Standalone support to sort the events to Hive cluster.
INLONG-1894	[Feature]Inlong-Sort-Standalone support JMX metrics listener for pushing.
INLONG-1892	[Feature]Inlong-Sort-Standalone support to consume events from Pulsar cache clusters.
INLONG-1738	[Feature] InLong audit

IMPROVEMENTS:

ISSUE	Summary
INLONG-2373	[Improve] Refactor of CreateBusinessWorkflow
INLONG-2358	[InLong audit] modify audit proxy name of introduction
INLONG-2352	[InLong audit] add audit introduction
INLONG-2349	[inlong-dataproxy] change log file name from flum.log to dataproxy.log
INLONG-2331	[Improve] Extract connector related code to sort-connector module
INLONG-2329	[Improve][inlong-dataproxy-sdk] asyncSendMessage in sender.java can be optimized to reduce the number of invalid objects
INLONG-2297	[Improve][agent] support audit for source and sink
INLONG-2296	Added lag consumption log
INLONG-2294	Rename the variable BROKER_VERSION to SERVER_VERSION
INLONG-2279	[Improve] Supplement TubeMQ's Javadoc information
INLONG-2274	[Improve][Manager] Supports configuring whether to create a Hive database or table
INLONG-2271	[Improve] rename the TDMsg to InLongMsg
INLONG-2258	[Improve][dashboard] Audit page support auto select datastream
INLONG-2254	Add historical offset query API
INLONG-2245	[Improve] Supports database-level isolation of audit queries
INLONG-2229	[Improve] Manager support pulsar authentification
INLONG-2225	[Improve][InLong-Dashboard] Audit module support i18n
INLONG-2220	[Improve] move dataproxy-sdk to inlong-sdk
INLONG-2210	[Improve] package inlong-manager-web as inlong-manager
INLONG-2200	[Feature] DataProxy add supporting to http protocol for reporting data
INLONG-2196	[Improve] move website to dashboard
INLONG-2193	[Improve] optimize inlong manager structure
INLONG-2160	[Improve] Time format conversion using DateTimeFormatter
[INLONG-2151](https://github.com/apache/inc...

0.12.0

FEATURES:

ISSUE	Summary
INLONG-1310	[Feature] [Feature] Support Pulsar
INLONG-1711	[feature] website support process pulsar dataflow
INLONG-1712	[Feature][agent] Add agent metric statistics
INLONG-1722	[Feature] Add IssueNavigationLink for IDEA
INLONG-1725	[Feature] [InLong-Manager] Modify bid and tid (or dsid) to inlongGroupId and inlongStreamId
INLONG-1726	[Feature] [InLong-Website] Adapt the Manager module and modify the field names of bid and dsid
INLONG-1732	[Feature] [InLong-Agent] Modify bid and tid to inlongGroupId and inlongStreamId
INLONG-1738	[Feature] InLong audit
INLONG-1764	[Feature]Use black for code block background style
INLONG-1768	[Feature] Adding consume type that allows partition assign from the client
INLONG-1785	[Feature] add 0.11.0 release article for blog
INLONG-1786	[Feature]Inlong-common provide monitoring indicator reporting mechanism with JMX, user can implement the code that read the metrics and report to user-defined monitor system.
INLONG-1791	[Feature][InLong-Manager] Some bid fields have not been modified
INLONG-1796	[Feature]DataProxy support monitor indicator with JMX.
INLONG-1809	[Feature] Adjust the font style of the official home page
INLONG-1814	[Feature] Show document file subdirectories
INLONG-1817	[Feature][InLong-Manager] Workflow supports data stream for Pulsar
INLONG-1821	[INLONG-810] Sort Module Support store data to ApacheDoris
INLONG-1826	[Feature] Use jmx metric defined in inlong-common
INLONG-1830	[Feature] Add a star reminder
INLONG-1833	[Feature] Add Team button to the navigation bar
INLONG-1840	[Feature] add a Welcome committer articles to official website blog
INLONG-1847	[Feature][InLong-Manager] Add consumption APIs for Pulsar MQ
INLONG-1849	[Feature][InLong-Manager] Push Sort config for Pulsar
INLONG-1851	[Feature]TubeMQ supports monitoring indicators with JMX.
INLONG-1853	[Feature] Agent should provide docs for jmx metrics
INLONG-1854	[Feature] Agent Rmi args should be added in agent-env.sh
INLONG-1856	[Feature] Add a news tab on the official website
INLONG-1867	[Feature] Add a user column display to the official website
INLONG-1873	[Feature] refactor the structure of the document for the official website
INLONG-1874	[Feature] Add contact information and common links at the bottom of the homepage of the official website
INLONG-1878	[Feature] Optimize user display page layout style
INLONG-1901	[Feature] Optimize the layout of the user display page
INLONG-1910	[Feature]Inlong-Sort-Standalone-sort-sdk support to consume events from inlong cache clusters(pulsar)
INLONG-1926	[Feature]Inlong-Sort-Standalone support JMX metrics listener for pulling.
INLONG-1938	[Feature] DataProxy send message to multi-pulsar cluster conf demo
INLONG-2002	[Feature]creating data access with pulsar, users should be able to change the ensemble param

IMPROVEMENTS:

ISSUE	Summary
INLONG-1708	[Improve] Add restrict of @author and Chinese in java file
INLONG-1729	[Improve] Avoid using constant value as version when referencing other modules
INLONG-1739	[Improve] Optimization of TubeMQ SDK usage demo
INLONG-1740	[Improve] change bid/tid to be more identifiable
INLONG-1746	[improve] the log4j properties for dataproxy contains some useless code and some class name are incorrect
INLONG-1756	[Improve] Use metadata to manage data sources and flow fields
INLONG-1772	[Improve]Adjust the ProcessResult class implementation
INLONG-1798	[Improve]RestTemplate does not read configuration from the configuration file
INLONG-1802	[Improve] Optimize document version management
INLONG-1808	[Improve] Optimize document of DataProxy about monitor metric.
INLONG-1810	[Improve] update the architecture for office-website
INLONG-1811	[Improve] Modify the architecture diagram of README.md
INLONG-1815	[Improve][translation] the blog of the 0.11.0 release should be translated into English
INLONG-1819	Optimize GC parameter configuration in TubeMQ's env.sh file
INLONG-1822	Optimize the table formatting in some MD documents
INLONG-1824	Refine the how-to-vote-a-committer-ppmc.md
INLONG-1857	[Improve] Adjust the content of the Disclaimer and Events column
INLONG-1859	[Improve][InLong-Manager] Remove duplicate SQL files
INLONG-1861	[Improve] Update document for docker-compose
INLONG-1863	[Improve][TubeMQ] repHelperHost for master should be exposed in configuration
INLONG-1864	[Improve] Agent Website doc contains a typo
INLONG-1865	[Improve] There are several errors in TubeMQ's guidance document
INLONG-1877	[Improve] improve the document's format for the office website
INLONG-1886	[Improve][InLong-Manager] Refactor and delete unused entities
INLONG-1916	[Improve][website] modify the Business to InLong Group
INLONG-1934	[Improve] update the image of the hive example after the bid changed
INLONG-1935	[Improve] package the SQL file for the manager
INLONG-1939	[Improve] add basic concepts for InLong
INLONG-1952	[Improve] Update the office website structure image
INLONG-1987	[Improve] Add function comment information in TubeMQ
INLONG-2017	[Improve] Add more guide documents for Pulsar

BUG FIXES:

ISSUE	Summary
INLONG-1706	[Bug] there are some incorrect expressions for issues tracking in the how-to-contribute file
INLONG-1716	[Bug][manager] can not login successfully
INLONG-1731	[Bug] release template has sth wrong with KEY URL
INLONG-1745	[Bug]TubeMQ HTTP API do...