[Meta] Kibana support for ES aggregations

[rayafratkina]

This issue will track support for ES aggregations within Kibana. The list below is not in a priority order, just an inventory.

Bucket aggregations

Type	AggConfigs⁴	Visualize	Lens	TSVB	Known issues & notes
Adjacency matrix
Histogram	✔️	✔️	✔️
Date histogram	✔️	✔️	✔️	✔️	#3787
Auto-interval date histogram	🔹	🔹	🔹	🔹	¹ see Notes below
Children
Composite					#26528
Date range	✔️	✔️			#9916
Diversified sampler	✔️				#67388, #120788
Filter	✔️	✔️	✔️	✔️
Filters	✔️	✔️	✔️	✔️	#17674
Geo distance
GeoHash grid	✔️	✔️			Not used, remove? #123192
GeoTile grid	✔️
Global	🔹	🔹	🔹	🔹	² see Notes below
IP range	✔️	✔️
Missing					#67391
Nested
Reverse nested
Parent
Range	✔️	✔️	✔️
Sampler	✔️				#120788
Terms	✔️	✔️	✔️	✔️	#42467, #17454
Significant terms	✔️	✔️	#116421	✔️	#40368
Rare terms	✔️		✔️		#67393
Significant text	✔️				#67394
Multi terms	✔️		✔️	✔️	#77632
Variable width histogram
Categorize text (expr.)

Metrics aggregations

Type	AggConfigs⁴	Visualize	Lens	TSVB	Known issues & notes
Average/sum	✔️	✔️	✔️	✔️
Weighted average					#67399
Cardinality	✔️	✔️	✔️	✔️
Geo bounds	✔️
Geo centroid	✔️
Geo line
Max/min	✔️	✔️	✔️	✔️
Percentiles	✔️	✔️	✔️	✔️	#20084
Percentiles Rank	✔️	✔️	✔️	✔️
Scripted metric	⛔	⛔	⛔	⛔	#2646 (comment)
Stats
String stats					#51510
Extended stats			✔️
Top hits	✔️	✔️	✔️	✔️³	#26850
Top metrics	✔️
Value count			✔️	✔️	#67403
Median absolute deviation
Boxplot					#4157
Event rate and average rate					#82948 and #59843 Implements positive rate only on the clientside in TSVB and Lens (not using ES aggregation)
T-test

Matrix aggregations

Type	AggConfigs⁴	Visualize	Lens	TSVB	Known issues & notes
Matrix stats

Pipeline aggregations

Type	AggConfigs⁴	Visualize	Lens	TSVB	Known issues & notes
Avg bucket	✔️	✔️	Client side "collapse by"	✔️
Derivative	✔️	✔️	✔️	✔️
Max/min bucket	✔️	✔️	Client side "collapse by"	✔️
Sum bucket	✔️	✔️	Client side "collapse by"	✔️
Stats/extended stats bucket
Percentile bucket
Moving average	✔️	✔️	✔️	✔️
Moving function
Moving percentiles
Normalize			Client side "normalize by unit"		#67402
Cumulative sum	✔️	✔️	✔️	✔️
Cumulative cardinality					#70337
Bucket script				✔️	#67405
Bucket selector
Bucket sort					#123719
Serial differencing	✔️	✔️		✔️
Bucket sort
Bucket count K-S test correlation (expr.)
Bucket correlation (expr.)
Inference bucket	N/A

Other aggregation ideas (not directly supported by ES)

• Filter ratio metric aggregation: Add "Filter Ratio" metric aggregation #20515 [Lens] Add simple ratio calculation, similar to filter ratio #42385
• Cumulative distribution: Cumulative distribution function #3905
• Articulate approximate nature of aggregations in the UI: Display approximate nature of some aggregations #2363
• Rate [lens] Add "counter rate" for monotonically increasing numbers #46627
• Average event rate [Lens] Discuss types of rates #62375
• Percentage change from previous [Lens] Shortcut to formula percentage change as an option in differences  #62381
• Multi-field terms aggregation [data.search.aggs] Support for multi-terms aggregation #77632

Notes

1. There are no plans to support Auto-interval date histogram in Kibana, since we're supporting auto interval via an implementation in Kibana. Due to technical reasons we'll keep using that Kibana side implementation for auto intervals. Auto interval on all time fields #4312 for the enhancement to support auto interval on all fields (not only the primary time field of an index pattern).
2. The global aggregation is meant to make a part of an aggregation ignore the search query. This is useful if you want to return documents and an aggregation in the same request. Since Kibana doesn't have that use-case, and can freely define the query (or a filters aggregation at any level it wants), there is currently no generic use-case to support Global in Kibana.
3. TSVB only supports top hits on numeric fields, no string fields.
4. AggConfigs lives in the data plugin's search service, and is used behind the scenes for aggregations in Visualize and Lens. Work on AggConfigs is tracked in [Meta] Aggregations Roadmap #60126. A checkmark here means that we have the technical foundation to implement it into visualizations, but not that it's already available to the user anywhere.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[talevy]

Hi @lukeelmers and @ppisljar, I notice that there are no plans to leverage the auto-date histogram since Kibana has a workaround for this. Is there a similar perspective to the proposed auto-numeric histogram feature (elastic/elasticsearch#31828), in that there is a kibana workaround at the moment?

[polyfractal]

Related, can I ask why this is the case:

There are no plans to support Auto-interval date histogram in Kibana, since we're supporting auto interval via an implementation in Kibana. Due to technical reasons we'll keep using that Kibana side implementation for auto intervals.

My understanding is that auto-date-histo was prioritized and added in 7.0 because it was requested in part by Kibana (elastic/elasticsearch#9572 (comment), #3757, etc) precisely so that Kibana didn't have to worry about auto-interval logic anymore? Is there some context somewhere as to why Kibana wants to continue supporting it's own auto-interval?

[timroes]

@talevy Hi, there is currently no real workaround for creating auto intervals on numeric fields. I think you'll need to wait for us to implement the auto histogram for numeric fields (which would leverage the ES aggregation) for this to work. Sorry. :-/

[timroes]

@polyfractal I read through some of the linked issues, and it seems some of those comments are really old, so it could simply be that we thought at that time it might be better to have it in Elasticsearch (for auto (non-date) histogram) that's also most likely still true, but for the date histograms there are a couple of issues:

• I unfortunately don't find the issue where that was discussed anymore, but I remember that auto_date_histogram had problems with extended_bounds. So we could not set the extended bounds, and would get a reasonable interval based on that extended bounds. We use that within the charts, because we want the actual chart time range be used for reasonable interval calculations, so that if you e.g. look at a chart from the past year, but the data only exists within one week of that year, we still want that the interval is 100 (arbitrary number I just made up) over that one year and not over that 1 week that contains data. I also think there were (in the past) some problems with min_doc_count = 0 for auto date histograms?
• We are showing the interval that will be used for the query in a couple of places before making the request, which would become impossible if we don't calculate it client side and thus don't know it before doing the request. (Even if we would do it after the request, I think we'd need an explicit part in the response that tells us what interval was used, since there could be buckets missing, so we can't calculate it from the buckets we have. But still we need it before the request in a couple of places).
• Last but not least: it works with the clientside logic. There is currently for us simply no real need to replacing that (rather deep sitting) logic, which would be a huge huge properly fragile change.

I hope this could shine some light onto the reasoning here, and happy to help with any further details.

[desaianuj]

@rayafratkina are there any plans to add percentile aggregation in Kibana Lens?

[loretoparisi]

what's the status about nested entities visualization in kibana?

[ppisljar]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.

[rayafratkina]

Re-opening as we are still using this

[timductive]

Closing old meta issues in favor of more granular, current meta issues. Going forward, we expect aggregation support to come via ES|QL support

[loretoparisi]

Closing old meta issues in favor of more granular, current meta issues. Going forward, we expect aggregation support to come via ES|QL support

thank you, could you please clarify the ES|QL support? Thanks.