Release v2.2.0

update version (#516)

Release v2.1.0

add latest datasource package (#484)

* add datasource pre-build package 2.1.0

* remove last latest datasource package

* add latest datasource package

Release v2.0.0

Merge pull request #431 from aliyun/release-200

Release 200

Release v1.9.0

add latest jars (#402)

Release v1.8.0

• New features
  • #338: Add datahub DataSourceV2
  • #366: Initial add TableStore streaming source
  • #372: Support stream sql on ots table
• Improvements
  • #335: Wait at most 5 minutes for server side exception
  • #360: Enhance retry for log service
  • #368: Do not retry if consumer group already exist
  • #377: Add retry error code blacklist
  • #363: Add user agent for log service client
  • #280: Update ons version to newest
  • #348: Migrate to latest log sdk
• Bug fix
  • #334: Fix get datahub schema
  • #337: Wrong type convert for ots record
  • #341: Failed to read logstore tag info
  • #344: Datahub source job recovered from a wrong starting offset
  • #349: Fix bug for ObjectMetadataDeserializer has the wrong date format
  • #369: Remove explicit calls to System.gc()
  • #350: Fix gson producing different date formats in diferent os/locale
  • #378: Duplicate data when persist InternalRow

Release v1.7.0

• New features
  • Support datahub datasource in Spark Structured Streaming
  • Add jdbc datasource support
  • Add hbase datasource support
  • Loghub relation support write
  • Add druid sink source
• Improvements
  • Better support loghub datasource schema type convert
  • Check if shard has finished after pull logs
  • Add retry for internal server error
  • Dont use SinkLog when there is no config
  • Loghub table should be defined with schema
  • Optimize to reduce extra count job when generate loghub RDD
  • Optimize logstore read step to avoid exceeding read quota limits
• Bug fix
  • Fix NPE when no data fetched from datahub
  • Could not generate RDD from a parent for unifying at time
  • Sometimes failed to get loghub schema
  • DatahubRDD 'count' should initial in each ShardPartition

Release v1.6.0

• New feature
  • Spark Streaming SQL Test tools.
  • support loghub datasource in spark streaming sql.
  • support datahub direct api implementation in Spark Streaming.
• Improvement
  • loghub python function support direct api.
• Bug fix
  • OnsUtilsHelper.createDefaultStreams lost ONS body message.
  • Failed to insert values into table store.

Release v1.5.0

• Improvement

  • Spark Structured Streaming support Loghub datasource.
  • Support parallel batch processing in loghub shard.
  • Cost too much time when update checkpoint in loghub direct api.
  • Add createRDD() in LoghubUtilsHelper.
  • Add java constructor for direct loghub dstream.

• Bugfix

  • Fix wrong loghub RDD partition index.
  • Direct loghub dstream data dose not contain tag information.
  • Direct loghub dstream dose not support to process data from specific position.
  • OdpsUtils.runSQL connection timeout.
  • ODPS STRING type convert failed.
  • Failed to re-run structured loghub job.
  • structured loghub job failed with org.I0Itec.zkclient.exception.ZkNoNodeException.
  • Wrong endTime in LoghubBatchRDD.

Release v1.4.4

• Spark Streaming support DataHub
• DirectLoghubInputDStream aupport multiple actions

v1.4.3

• Provide a kafka metrics reporter implementation to collect kafka client metrics to own kafka cluster.
• Retry when connection exception in DirectLoghubInputDStream.
• Support do a batch computing on LogService.
• Add tag information into LogService RDD.