sparkMeasure v0.24

Release 0.24 highlights:

• Added support for Prometheus Gateway in Flight Recorder mode.
• Added computed metric average number of active tasks to the Stagemetrics report
• The Python API checks if sparkMeasure jar is loaded
• Refreshed dependencies

sparkMeasure v0.23

Release 0.23 highlights:

• new: aggregate_stagemetrics and aggregate_taskmetrics added to the Python API
• added Scala and Python tests
• improvements to the documentation
• improvements to the GitHub workflows

sparkMeasure v0.22

Pushed new release to fix issues with 0.21 not being correctly published, see #43

sparkMeasure v0.21

Added print memory report to stage-level metrics collection.
Example:

(scala)> stageMetrics.printMemoryReport
(python)> stagemetrics.print_memory_report()

sparkMeasure v0.20

• The Apache Spark dependency has been bumped up to Spark 3.3.0
• This release is cross compiled for Scala 2.12 and 2.13

sparkMeasure v0.19

Refactoring of StageMetrics and TaskMetrics:

• report will not use Spark SQL but rather compute the metrics using Scala code.
• removed API for accumulables metrics capture and reporting
• added a short report "Stages and their duration" to StageMetrics report
• updated the Python package sparkmeasure match the Scala code refactoring

Improved documentation:

• notably added documentation for the KafkaSink and a reference guide of sparkMeasure API

Latest release to build with Spark 2.4.8 and Scala 2.11

• use this release if you run sparkMeasure with Spark 2.x and Scala 2.11

sparkMeasure v0.18

This adds the Apache Kafka sink.
The versions of several dependencies have been bumped up.

sparkMeasure v0.17

This release has minor improvements.
Updated and extended the list of shuffle metrics.
Improved output of taskMeasure.
Note: this release breaks the compatibility with Spark 2.1 and 2.2. Use for Spark 2.3 and higher only.

sparkMeasure v0.16

Minor improvements in this release.
Updated InfuxDB sink to map executorId, executorHost with the InfluxDB sink. This is used by the Spark dashboard as detailed in Spark dashboard annotations

sparkMeasure v0.15

This release extends sparkMeasure Flight recorder mode with an InfluxDB sink:

• It allows to write Spark queries, jobs and stages information to InfluxDB.
• Optionally it can write stage and task metrics to InfluxDB.
• These features have been added to improve Spark performance dashboard built with Grafana and InfluxDB.