[TSVB] FR: Scripted Fields / Scripted Aggregations

[hetzer-fastec]

Feature Request
I am suggesting to include support for scripted aggregations (such as a scripted sum aggregation as described in the official documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-sum-aggregation.html) and/or scripted fields in the time series visual builder. Currently only the bucket script aggregation seems to be implemented (see #4707).

Right now, one can neither select a scripted field as the target of an aggregation nor supply a custom script in combination with a special aggregation in the time series visual builder, which is a drastic limitation, in my opinion.

I have a use case where I need to compute several bucket script aggregations over simple aggregations, which themselves are scripted / work on scripted fields. For example: I need to compute a sum of the form sum(f(x)), where f(x) = x, if y = 0 and f(x) = 0 else (x, y are document fields). Unfortunately I am not in a situation to alter the type in order to include an according precomputed field.

There seems to be no way, I can visualize these computations right now in Kibana although I can generally compute them.

[alexfrancoeur]

Thanks for filing @hetzer-fastec! I think the new Math aggregation we are working on #13681 will help with your use case. Would you mind reviewing the description to see if the functionality fully fits your use case. I believe it will.

cc: @simianhacker

[hetzer-fastec]

Thanks for your quick answer, @alexfrancoeur! I already saw #13681, but as far as I can see that one does not really solve my use case, it only partially helps.

The main problem I am having is that I cannot create a scripted aggregation, e.g. a scripted sum aggregation in the TSVB.

For example, suppose we have a type car, which has a state "s", and two numeric values "a" and "b". Now we want to have a scripted sum aggregation, which basically selects "a" as a summand if the state has a certain feature and "b" as a summand it this is not the case. Let's call this sum "L". Let's now consider that our car type has two other numerical fields "c" and "d", which are summed as well based on some condition regarding the state. Let's call this sum "P".

Now at the end we want to compute the result of "L" / "P", which we can do by a bucket-script aggregation.

While I can theoretically create two sums in the TSVB using the sum aggregation, I cannot create them in the way I described it as the sum aggregation does not allow to define a script, which the summation is based on.

While I really like the idea of the math aggregation, it does not solve my problem, as far as I understood the issue. Please correct me if I am wrong - I am more than happy to learn that my issue is resolved with the help of the math aggregation :).

[hetzer-fastec]

@alexfrancoeur Any news on this ? :)

[hetzer-fastec]

At least another reply would be great!

[alexfrancoeur]

@hetzer-fastec apologies, somehow I missed your comments from earlier. Thank you for your patience. I agree and do not believe that Math.js may not be able to handle the state-like logic you are looking for. @simianhacker, this is a pretty interesting use case. Any idea's how @hetzer-fastec can accomplish this? If TSVB natively supported scripted fields would this be possible?

[simianhacker]

At some point I'm going to integrate TSVB with Kibana Index Patterns (and scripted fields). There are no timelines (due to other forces of nature on my time :D)

[hetzer-fastec]

@simianhacker @alexfrancoeur Thanks for the update! I guess, I have to wait until its implemented then :).

[ebalmeida]

+1

[jehutywong]

+1

[Jerome2606]

+1

[inner-outer-space]

+1

[vinodradhakrishnaiyer]

+1

[ever4Kenny]

vote for this issue

[akamzin]

+1 I believe this is a serious limitation. Other tools have it but do not have bucket script.

[Paul-de-Barbeyrac]

+1

[cpiment]

+1

[linyaru]

+1

[ghost]

+1

[mvsousa]

+1

[iferreiro-ml]

+1

[sethyes]

+1

[ModestinoAndre]

+1

[amita-r]

+1
Any ETA on when the scripted fields would become available in TSVB?

[markwheal]

Scripted Fields support for TSVB would be a great new feature. +1

[rjso]

+1

[simianhacker]

Quick update on this issue, we've added this to the Metrics UI roadmap for 7.9. The goal is to allow users to define scripted fields ad hoc since there isn't a hard requirement for Kibana Saved Index Patterns in TSVB. I'm still working out what the best way to do this is. My initial approach will be to allow users to define scripted fields under the panel options so they can use them throughout the TSVB UI (via basic aggs and percentiles).

[simianhacker]

After playing around with this a bit today, instead of adding scripted fields under Panel Options I think a better approach will be to do something like this:

I've added a field named "Script" to all the field dropdowns, when this is selected it will display a text area for a script. Once you've entered a script, everything will just work like a normal field. The screen shot above is WIP and you should see a PR for this in the next couple of days.

[ihdavids]

I like the ability to run arbitrary scripts AND it would be nice to have support for scripted fields.
IMHO the ability to run arbitrary scripts here is a very powerful timesaver when searching for a problem using your data.

[ihdavids]

Any progress on this?

[Alsheh]

@simianhacker thanks for working on the scripted field feature in TSVB! 7.9 is already out and I'm wondering if we should still continue to expect this to be released in some 7.x version?

[simianhacker]

@Alsheh The PR was closed and this comment explains why:

#68332 (comment)

[ghost]

+1

[rayafratkina]

With #82438 TSVB can use runtime fields in Visualizations. I believe this closes this request.