Release 1.15.1

Bug fixes, Minor Improvements

• Build platform binaries as part of CI (#1909, @yurishkuro)
• Upgrade and fix dependencies (#1907, @yurishkuro)

Release 1.15.0

Backend Changes

Breaking Changes

• The default value for the Ingester's flag ingester.deadlockInterval has been changed to 0 (#1868, @jpkrohling)

  With the new default, the ingester won't panic if there are no messages for the last minute. To restore the previous behavior, set the flag's value to 1m.

• Mark --collector.grpc.tls.client.ca flag as deprecated for jaeger-collector. (#1840, @yurishkuro)

  The deprecated flag will still work, but will be removed removed in the future, it's recommended to use --collector.grpc.tls.client-ca instead.

New Features

• Support TLS for Kafka (#1414, @MichaHoffmann)
• Add ack and compression parameters for Kafka #1359 (#1712, @chandresh-pancholi)
• Propagate the bearer token to the gRPC plugin server (#1822, @radekg)
• Add Truncate and ReadOnly options for badger (#1842, @burmanm)

Bug fixes, Minor Improvements

• Use correct context on ES search methods (#1850, @rubenvp8510)
• Handling of expected error codes coming from grpc storage plugins #1741 (#1814, @chandresh-pancholi)
• Fix ordering of indexScanKeys after TraceID parsing (#1809, @burmanm)
• Small memory optimizations in badger write-path (#1771, @burmanm)
• Set an empty value when a default env var value is missing (#1777, @jpkrohling)
• Decouple storage dependencies and bump Go to 1.13.x (#1886, @yurishkuro)
• Update gopkg.in/yaml.v2 dependency to v2.2.4 (#1865, @objectiser)
• Upgrade jaeger-client 2.19 and jaeger-lib 2.2 and prom client 1.x (#1810, @yurishkuro)
• Unpin grpc version and use serviceConfig to set the load balancer (#1786, @guanw)

UI Changes

• The changelog is available here v1.5.0

Release 1.14.0

Backend Changes

Breaking Changes

• Create ES index templates instead of indices (#1627, @pavolloffay)

  This can break existing Elasticsearch deployments if security policies are applied.
  For instance Jaeger X-Pack configuration now requires permission to create index templates - manage_index_templates.

New Features

• Add Elasticsearch version configuration to rollover script (#1769, @pavolloffay)

• Add Elasticsearch version flag (#1753, @pavolloffay)

• Add Elasticsearch 7 support (#1690, @gregoryfranklin)

  The index mappings in Elasticsearch 7 are not backwards compatible with the older versions.
  Therefore using Elasticsearch 7 with data created with older version would not work.
  Elasticsearch 6.8 supports 7.x, 6.x, 5.x compatible mappings. The upgrade has to be done
  first to ES 6.8, then apply data migration or wait until old daily indices are removed (this requires
  to start Jaeger with --es.version=7 to force using ES 7.x mappings for newly created indices).

  Jaeger by default uses Elasticsearch ping endpoint (/) to derive the version which is used
  for index mappings selection. The version can be overridden by flag --es.version.

• Support for Zipkin Protobuf spans over HTTP (#1695, @jan25)

• Added support for hot reload of UI config (#1688, @jpkrohling)

• Added base Grafana dashboard and Alert rules (#1745, @jpkrohling)

• Add the jaeger-mixin for monitoring (#1668, @gouthamve)

• Added flags for driving cassandra connection compression through config (#1675, @sagaranand015)

• Support index cleaner for rollover indices and add integration tests (#1689, @pavolloffay)

• Add client TLS auth to gRPC reporter (#1591, @tcolgate)

• Collector kafka producer protocol version config (#1658, @marqc)

• Configurable kafka protocol version for msg consuming by jaeger ingester (#1640, @marqc)

• Use credentials when describing keyspaces in cassandra schema builder (#1655, @MiLk)

• Add connect-timeout for Cassandra (#1647, @sagaranand015)

Bug fixes, Minor Improvements

• Fix gRPC over cmux and add unit tests (#1758, @yurishkuro)
• Add CA certificates to agent image (#1764, @yurishkuro)
• Fix badger merge-join algorithm to correctly filter indexes (#1721, @burmanm)
• Change Zipkin CORS origins and headers to comma separated list (#1556, @JonasVerhofste)
• Added null guards to 'Process' when processing an incoming span (#1723, @jpkrohling)
• Export expvar metrics of badger to the metricsFactory (#1704, @burmanm)
• Pass TTL as int, not as float64 (#1710, @yurishkuro)
• Use find by regex for archive index in index cleaner (#1693, @pavolloffay)
• Allow token propagation if token type is not specified (#1685, @rubenvp8510)
• Fix duplicated spans when querying Elasticsearch (#1677, @pavolloffay)
• Fix the threshold precision issue (#1665, @guanw)
• Badger filter duplicate results from a single indexSeek (#1649, @burmanm)
• Badger make default dirs work in Windows (#1653, @burmanm)

UI Changes

• UI pinned to version 1.4.0. The changelog is available here v1.4.0

Release 1.13.1

Backend Changes

Breaking Changes

New Features

Bug fixes, Minor Improvements

Change default for bearer-token-propagation to false (#1642, @wsoula)

UI Changes

Release 1.13.0

Backend Changes

Breaking Changes

• The traces related metrics on collector now have a new tag sampler_type (#1576, @guanw)

  This might break some existing metrics dashboard (if so, users need to update query to aggregate over this new tag).

  The list of metrics affected: traces.received, traces.rejected, traces.saved-by-svc.

• Remove deprecated index prefix separator : from Elastic (#1620, @pavolloffay)

  In Jaeger 1.9.0 release the Elasticsearch index separator was changed from : to -. To keep backwards
  compatibility the query service kept querying indices with : separator, however the new indices
  were created only with -. This release of Jaeger removes the query capability for indices containing :,
  therefore it's recommended to keep using older version until indices containing old separator are
  not queried anymore.

New Features

• Passthrough OAuth bearer token supplied to Query service through to ES storage (#1599, @rubenvp8510)
• Kafka kerberos authentication support for collector/ingester (#1589, @rubenvp8510)
• Allow Cassandra schema builder to use credentials (#1635, @PS-EGHornbostel)
• Add docs generation command (#1572, @pavolloffay)

Bug fixes, Minor Improvements

• Fix data race between Agent.Run() and Agent.Stop() (#1625, @tigrannajaryan)
• Use json number when unmarshalling data from ES (#1618, @pavolloffay)
• Define logs as nested data type (#1622, @pavolloffay)
• Fix archive storage not querying old spans older than maxSpanAge (#1617, @pavolloffay)
• Query service: fix logging errors on SIGINT (#1601, @jan25)
• Direct grpc logs to Zap logger (#1606, @yurishkuro)
• Fix sending status to health check channel in Query service (#1598, @jan25)
• Add tmp-volume to all-in-one image to fix badger storage (#1571, @burmanm)
• Do not fail es-cleaner if there are no jaeger indices (#1569, @pavolloffay)
• Automatically set GOMAXPROCS (#1560, @rubenvp8510)
• Add CA certs to all-in-one image (#1554, @chandresh-pancholi)

UI Changes

• UI pinned to version 1.3.0. The changelog is available here v1.3.0

Release 1.12.0

Backend Changes

Breaking Changes

• The kafka flags were removed in favor of kafka.producer and kafka.consumer flags (#1424, @ledor473)

  The following flags have been removed in the Collector and the Ingester:

  --kafka.brokers
  --kafka.encoding
  --kafka.topic
  --ingester.brokers
  --ingester.encoding
  --ingester.topic
  --ingester.group-id
  

  In the Collector, they are replaced by:

  --kafka.producer.brokers
  --kafka.producer.encoding
  --kafka.producer.topic
  

  In the Ingester, they are replaced by:

  --kafka.consumer.brokers
  --kafka.consumer.encoding
  --kafka.consumer.topic
  --kafka.consumer.group-id
  

• Add Admin port and group all ports in one file (#1442, @yurishkuro)

  This change fixes issues #1428, #1332 and moves all metrics endpoints from API ports to admin ports. It requires re-configuring Prometheus scraping rules. Each Jaeger binary has its own admin port that can be found under --admin-http-port command line flag by running the ${binary} help command.

New Features

• Add gRPC resolver using external discovery service (#1498, @guanw)
• gRPC storage plugin framework (#1461, @chvck)
• Supports customized kafka client id (#1507, @newly12)
• Support gRPC for query service (#1307, @annanay25)
• Expose tls.InsecureSkipVerify to es.tls.* CLI flags (#1473, @stefanvassilev)
• Return info msg for /health endpoint (#1465, @stefanvassilev)
• Add pprof endpoint to admin endpoint (#1375, @konradgaluszka)
• Add inbound transport as label to collector metrics #1446 (guanw)
• Sorted key/value store badger backed storage plugin (#760, @burmanm)
• Add Admin port and group all ports in one file (#1442, @yurishkuro)
• Adds support for agent level tag (#1396, @annanay25)
• Add a Downsampling writer that drop a percentage of spans (#1353, @guanw)

Bug fixes, Minor Improvements

• Sort traces in memory store to return most recent traces (#1394, @jacobmarble)
• Add span format tag for jaeger-collector (#1493, @guo0693)
• Upgrade gRPC to 1.20.1 (#1492, @guanw)
• Switch from counter to a gauge for partitions held (#1485, @bobrik)
• Add CORS handling for Zipkin collector service (#1463, @JonasVerhofste)
• Check elasticsearch nil response (#1467, @YEXINGZHE54)
• Disable sampling in logger - zap(#1460, @psinghal20)
• New layout for proto definitions and generated files (#1427, @annanay25)
• Upgrade Go to 1.12.1 (#1437 ,@yurishkuro)

UI Changes

• UI pinned to version 1.2.0. The changelog is available here v1.2.0

Release 1.11.0

Backend Changes

Breaking Changes

• Introduce kafka.producer and kafka.consumer flags to replace kafka flags (1360, @ledor473)

  The following flags have been deprecated in the Collector and the Ingester:

  --kafka.brokers
  --kafka.encoding
  --kafka.topic
  

  In the Collector, they are replaced by:

  --kafka.producer.brokers
  --kafka.producer.encoding
  --kafka.producer.topic
  

  In the Ingester, they are replaced by:

  --kafka.consumer.brokers
  --kafka.consumer.encoding
  --kafka.consumer.group-id
  

New Features

• Support secure gRPC channel between agent and collector (#1391, @ghouscht, @yurishkuro)
• Allow to use TLS with ES basic auth (#1388, @pavolloffay)

Bug fixes, Minor Improvements

• Make esRollover.py init idempotent (#1407 and #1408, @pavolloffay)
• Allow thrift reporter if grpc hosts are not provided (#1400, @pavolloffay)
• Deprecate colon in index prefix in ES dependency store (#1386, @pavolloffay)
• Make grpc reporter default and add retry (#1384, @pavolloffay)
• Use CQLSH_HOST in final call to cqlsh (#1372, @funny-falcon)

UI Changes

• UI pinned to version 1.1.0. The changelog is available here v1.1.0

Release 1.10.1

Backend Changes

• Discover dependencies table version automatically (#1364, @black-adder)

Breaking Changes

New Features

Bug fixes, Minor Improvements

• Separate query-service functionality from http handler (#1312, @annanay25)

UI Changes

Release 1.10.0

Backend Changes

Breaking Changes

• Remove cassandra SASI indices (#1328, @black-adder)

Migration Path:

1. Run plugin/storage/cassandra/schema/migration/v001tov002part1.sh which will copy dependencies into a csv, update the dependency UDT, create a new dependencies_v2 table, and write dependencies from the csv into the dependencies_v2 table.
2. Run the collector and query services with the cassandra flag cassandra.enable-dependencies-v2=true which will instruct jaeger to write and read to and from the new dependencies_v2 table.
3. Update spark job to write to the new dependencies_v2 table. The feature will be done in #58.
4. Run plugin/storage/cassandra/schema/migration/v001tov002part2.sh which will DELETE the old dependency table and the SASI index.

Users who wish to continue to use the v1 table don't have to do anything as the cassandra flag cassandra.enable-dependencies-v2 will default to false. Users may migrate on their own timeline however new features will be built solely on the dependencies_v2 table. In the future, we will remove support for v1 completely.

• Remove ErrorBusy metric, it essentially duplicates SpansDropped (#1091, @cstyan)

New Features

• Support certificates in elasticsearch scripts (#1339, @pavolloffay)
• Add ES Rollover support to main indices (#1309, @pavolloffay)
• Load ES auth token from file (#1319, @pavolloffay)
• Add username/password authentication to ES index cleaner (#1318, @gregoryfranklin)
• Add implementation of FindTraceIDs function for Elasticsearch reader (#1280, @vlamug)
• Support archive traces for ES storage (#1197, @pavolloffay)

Bug fixes, Minor Improvements

• Use Zipkin annotations if the timestamp is zero (#1341, @geobeau)
• Use GRPC round robin balancing even if only one hostname (#1329, @benley)
• Tolerate whitespaces in ES servers and kafka brokers (#1305, @verma-varsha)
• Let cassandra servers contain whitespace in config (#1301, @karlpokus)

UI Changes

Release 1.9.0

Backend Changes

Breaking Changes

• Change Elasticsearch index prefix from : to - (#1284, @pavolloffay)

Changed index prefix separator from : to - because Elasticsearch 7 does not allow : in index name.
Jaeger query still reads from old indices containing - as a separator, therefore no configuration or migration changes are required.

• Add CLI configurable es.max-num-spans while retrieving spans from ES (#1283, @annanay25)

The default value is set to 10000. Before no limit was applied.

• Update to jaeger-lib 2 and latest sha for jaeger-client-go, to pick up refactored metric names (#1282, @objectiser)

Update to latest version of jaeger-lib, which includes a change to the naming of counters exported to
prometheus, to follow the convention of using a _total suffix, e.g. jaeger_query_requests is now
jaeger_query_requests_total.

Jaeger go client metrics, previously under the namespace jaeger_client_jaeger_ are now under
jaeger_tracer_.

• Add gRPC metrics to agent (#1180, @pavolloffay)

The following metrics:

jaeger_agent_tchannel_reporter_batch_size{format="jaeger"} 0
jaeger_agent_tchannel_reporter_batch_size{format="zipkin"} 0
jaeger_agent_tchannel_reporter_batches_failures{format="jaeger"} 0
jaeger_agent_tchannel_reporter_batches_failures{format="zipkin"} 0
jaeger_agent_tchannel_reporter_batches_submitted{format="jaeger"} 0
jaeger_agent_tchannel_reporter_batches_submitted{format="zipkin"} 0
jaeger_agent_tchannel_reporter_spans_failures{format="jaeger"} 0
jaeger_agent_tchannel_reporter_spans_failures{format="zipkin"} 0
jaeger_agent_tchannel_reporter_spans_submitted{format="jaeger"} 0
jaeger_agent_tchannel_reporter_spans_submitted{format="zipkin"} 0

jaeger_agent_collector_proxy{endpoint="baggage",result="err"} 0
jaeger_agent_collector_proxy{endpoint="baggage",result="ok"} 0
jaeger_agent_collector_proxy{endpoint="sampling",result="err"} 0
jaeger_agent_collector_proxy{endpoint="sampling",result="ok"} 0

have been renamed to:

jaeger_agent_reporter_batch_size{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_batch_size{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_batches_failures{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_batches_failures{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_batches_submitted{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_batches_submitted{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_spans_failures{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_spans_failures{format="zipkin",protocol="tchannel"} 0
jaeger_agent_reporter_spans_submitted{format="jaeger",protocol="tchannel"} 0
jaeger_agent_reporter_spans_submitted{format="zipkin",protocol="tchannel"} 0

jaeger_agent_collector_proxy{endpoint="baggage",protocol="tchannel",result="err"} 0
jaeger_agent_collector_proxy{endpoint="baggage",protocol="tchannel",result="ok"} 0
jaeger_agent_collector_proxy{endpoint="sampling",protocol="tchannel",result="err"} 0
jaeger_agent_collector_proxy{endpoint="sampling",protocol="tchannel",result="ok"} 0

• Rename tcollector proxy metric in agent (#1182, @pavolloffay)

The following metric:

jaeger_http_server_errors{source="tcollector-proxy",status="5xx"}

has been renamed to:

jaeger_http_server_errors{source="collector-proxy",status="5xx"}

New Features

• Add tracegen utility for generating traces (#1245, @yurishkuro)
• Use DCAwareRoundRobinPolicy as fallback for TokenAwarePolicy (#1285, @vprithvi)
• Add Zipkin Thrift as kafka ingestion format (#1256, @geobeau)
• Add FindTraceID to the spanstore interface (#1246, @vprithvi)
• Migrate from glide to dep (#1240, @isaachier)
• Make tchannel timeout for reporting in agent configurable (#1034, @gouthamve)
• Add archive traces to all-in-one (#1189, @pavolloffay)
• Start moving components of adaptive sampling to OSS (#973, @black-adder)
• Add gRPC communication between agent and collector (#1165, #1187, #1181 and #1180, @pavolloffay)

Bug fixes, Minor Improvements

• Update exposed ports in ingester dockerfile (#1289, @objectiser)
• Upgrade Shopify/Sarama for proper handling newest kafka servers 2.x by ingester (#1248, @vprithvi)
• Fix sampling strategies overwriting service entry when no sampling type is specified (#1244, @objectiser)
• Fix dot replacement for int (#1272, @pavolloffay)
• Add C* query to error logs (#1250, @vprithvi)
• Add locking around partitionIDToState map accesses (#1239, @vprithvi)
• Reorganize config manager packages in agent (#1198, @pavolloffay)

UI Changes

• UI pinned to version 1.0.0. The changelog is available here v1.0.0