Feature Request: template variable string manipulations

[CyBeRoni]

It would be nice if Grafana template variables could have some string manipulations applied. This could be useful to be able to refer to data about the same thing from various datasources from a variable, where that thing is named slightly differently for technical reasons. For instance, if you have a website that you refer to by the name "some_site" in your systems, but you have an AWS S3 bucket for it where you can't use underscores, you could transform that name to 'some-site', or 'somesite' instead.

The immediately obvious ideas for what kind of manipulations to support would be the various variable substitutions that bash supports. In bash I would do the above example by using ${var//_/-} or ${var//_}.

[tuukkamustonen]

Would be nice to be able to grab parts of variable, too, e.g. $(regex(some_variable, "/^([^-]+)\-([^-]+)$/", "foo: $1 bar:$2")).

This would allow showing much nicer labels with repeat panels/rows (no need to show full value like ew1-cosmos-prd-prod-admin_api-AutoScalingGroup-1FC4QCWSNZZFV but just admin_api (or even Admin API in combination also text transforms were supported).

[VivekDevops]

any work around for this request yet?

[davkal]

I'm trying to understand if this can be solved already via http://docs.grafana.org/reference/templating/#query-options
Could you describe your problem with a concrete example, and possibly a screenshot on where you hit a wall here?

[C-Duv]

In my case I have a $Host variable of type Query, whose possible values are my server host names: I use it in my queries to only fetch data of this/theses host(s) and in the Title or Description field.

But theses hostnames can be rather long: "srv42-nyc-prod.example.com" which can create line wrapping on small "Singlestat" panel for example.

The ability to perform a substring/regex operation on my $Host variable so that it strips the "-prod.example.com" and only keep "srv42-nyc" would be great.

[billyaustin84]

I have a similar use case to C-Duv.
I pull back a variable in the form "http://server1.example.com" and would like to use just the "server1" element of the string.

[marefr]

@billyaustin84 use the variable regex support for that

[davkal]

I'm still uncertain where the manipulated variables are being used: in panel queries or somewhere else? I screenshot would help immensely.

Also helpful, if @billyaustin84 and @C-Duv could confirm that they tried using http://docs.grafana.org/reference/templating/#query-options and failed.

[torkelo]

it would probably be used in series alias formatting & panel titles.

[DanCech]

When I initially added the support for ${varname:raw} syntax, my thought was that it could be extended to be a more general-purpose processing language for munging variables as needed, so a syntax like ${varname:replace(/(.*)_/,'$1 '):toLower} could represent a pipeline for modifying the value at interpolation time (as distinct from applying a regex to modify values at query time, since the same value might be used in different contexts within a dashboard).

[scaytrase]

For particular example I faced impossibility to just use simple conversions like [[variable:toLower]] or [[variable:toUpper]] since in different queries we need different flavours of variable value while keeping grouping (i.e for repeating rows by variable)

[pierrecdn]

I also have a use-case in which template variables represent applications, and these applications can be namespaced using / (slash) characters in a container orchestration platform (namely Marathon).

These characters are removed in the metrics collection process and replaced by _ (underscore).

But then if you want to propose dynamic links in text panels for ex. to the app / API / whatever, you've to do the opposite transformation, ie. s/_/\./.

[C-Duv]

@davkal, I already used regex option for my $Host variable of type "Query". The query fetches all Prometheus known instances and the regex strips the port part so my variable only contains the server's FQDN.

It works for querying data (using some_metric{instance=~"$Host"}).

But for labels in graphs's legend or in graph's title/description I sometimes want to display a shorter hostname (eg. srv42-nyc instead of srv42-nyc-prod.example.com).
So a Bash's variable mangling kind of manipulation is fine for me.

I know I could change the variable's regex so that it only contains srv42-nyc and use the following query some_metric{instance=~"$Host-prod.example.com}. But I should adapt all queries, and what if I change/add a domain?. Also feel a FQDN-role variable $Host is proper and variable content/meaning should not be changed for display purpose.

[tpolekhin]

my example:
i have a statsd-exporter for nginx-ingress-controller that changes all non-alphanumeric characters in metrics names and labels to underscore _

as a result i can't query both nginx metrics and kubernetes metrics in one dashboard,
because application deployment label is example-app and nginx handler label example_app

it would be very useful if we could have some basic string manipulation in variables section,
or right in a query string, for example change underscore _ to dash - in variable value for this query

[phrend]

When I initially added the support for ${varname:raw} syntax, my thought was that it could be extended to be a more general-purpose processing language for munging variables as needed, so a syntax like ${varname:replace(/(.*)_/,'$1 '):toLower} could represent a pipeline for modifying the value at interpolation time (as distinct from applying a regex to modify values at query time, since the same value might be used in different contexts within a dashboard).

This is exactly one of my use-cases. I have metrics with part of the path in upper-case, and part in lower-case. It would be great to be able to create a single variable that could be transformed to fit either use-case, on a given dashboard.

[SleepyBrett]

Also my usecase. I have some metrics that are tagged with the hosts IP address and some that are tagged with the hostname (which contains the ip address ip-x-x-x-x.blah) I'd like a way to use one variable to drive both queries.

[digital-infinity7]

My issue: I filter queries using lowercase environment names but I want environments to show in UPPERCASE in template panel titles. So generally something like ${variable:toUpper} would be great :-)
Other transformation ideas:

• from camelCase to underscore_case
• and opposite from underscore_case to camelCase
• first letter Upper case
• first letter lOWER case

[trevormac789]

My situation is I'm templating switches using the Zabbix plugin and what I need is the title will contain the interface name and description from the dashboard variable $ifalias.

The metrics will use the interface name extracted from the variable and catenate with strings like ":Bits received :Bits sent etc" and dynamically create a panel for every interface on the switch. The only piece missing is having the interface description in the title as so far I'm just using a variable with the interface name. If I could just drop everything after the first space in the variable. The full string will be: (Ethernet3/1 Alias myinterfacedescription). I can use "Interface ${ifalias regex manipulation}|: Bits received"

Is this possible now?

I'm going to redo the Zabbix item prototype to drop the prepended IF and space so I don't need to worry about that.

In the above example "item" field, replace $ifname with ${ifalias variable manipulation}

I'm so close it would be amazing if I could do this. We're about to roll out a lot of switches and this will be a huge time saver.

[creslinux]

I also have a need for this. I take a list or measurements from InfluxDB as my variable, i need the first two parts of the measurement name to build consistent panels.
By way of example:

compA_orders_west
compA_orders_east
compB_orders_west
compB_orders_east

I wish for my variable list to be ['compA_orders', 'compB_orders']
I will then add west and east to the query in 2 panels in dashboard

The result should be I can pick compA_orders and have two panels for west and east for this company.

!!!!!
EDIT -- Ignore me, Grafana already does this with Regex in variable load /^(.*?_.*?)_/ was my answer
!!!!!

[cpsubrian]

My use-case is repeating rows over 'tuples' of tag values. If I could create variables like:

action-controller,action-controller,action-controller...

And then use only part of the variable in right part of the query (two different tags). This would allow me to create the kind of repeating dashboards that I want to.

[waldner]

My use case is turning "host_example_com" to "host.example.com", something I thought would be easy to do...

[amanzag]

My use case is to be able to define a list of http endpoints to monitor, like:
endpoint=GET /path1, POST /path2
and then be able to create a repeated graph for each of the values, like:
http_server_requests_seconds_count{method="${endpoint[0]}", uri="${endpoint[1]}"}

[Fanduzi]

My use case is that the variables is ip_port , but we only want ip

see this the variable $host is 192.168.x.x_3307, but we want 192.168.x.x only . if there is a way like
'192.168.x.x_3307'.split('_')[0] it would be perfect!

[marefr]

@Fanduzi can't you use regex to remove the _<port> in the variable?

[jimmybigcommerce]

+1 the ability to manipulate the strings like this would be immensely useful!

[anthonysomerset]

For me, i'd like to maintain a single list of router/interface pairs e.g.

BYO-XXX-XXX_LZWXXXXXX,BYO-XXX-XXX_GigabitEthernet0/X/X.XXXX

I'd like to have a repeating panel on this variable list, that then uses the first part before _ to populate the Host field (Zabbix plugin) and the second part the Item field within a larger regex itself

if i can create nested variables that extract the parts of the parent variable to achieve this effect then that would work for me too

[pierrecdn]

I think it would be nice to have maintainers views on this instead of adding use-cases and use-cases year after year 😄
@davkal @marefr what do you think? Skeptical about validity of use-cases? Generally speaking, people don't always have the full control on their metric ingestion, metric transformation, metric history and would like to change their text representation in drop-down lists and panels.

[NeverUsedID]

@marefr

@Fanduzi can't you use regex to remove the _<port> in the variable?

How would one do that? Do You have an example?

EDIT: Just found the solution for me. Im using this regex in the query to get the variable:
'/(.*)_traffic_in/' select's everything without the "_traffic_in" from the metric.

[kogaba]

@marefr

@Fanduzi can't you use regex to remove the _<port> in the variable?

How would one do that? Do You have an example?

EDIT: Just found the solution for me. Im using this regex in the query to get the variable:
'/(.*)_traffic_in/' select's everything without the "_traffic_in" from the metric.

Thank you! Exactly what I was looking for!

[anitakrueger]

I needed to simply upperCase a variable today and found out I cannot do this in Grafana variables. The PR to fix it was closed 5 months ago with being unsure? #29754 (comment)

I thought Advanced variable format options are the answer?
Please provide a solution to this 4 year old request which so many people are asking for.

[sandeepdharembra]

I have a use case where I need to use custom variables like this -

key : value

Now I want the "value" such that I can split it and use the parts for different label values. Example -

Custom Variable name: pool
environment1 : "service1~servers"
environment2 : "service2~servers"

query :- metric_name{label1=~"{$pool/~.*/''}", label2=~"{$pool/^.*~/''}"}

The above kind of replace or some split functionality

Since we can't have chaining in custom variables, string manipulation would be helpful. I cannot pull environments from an datasource as those are not present in the metrics.

[cfstras]

Any updates on this? My specific usecase is a Grafana Cloud instance with both logs and metrics datasources -- and I want to include a textbox on a dashboard with a xyz-prom datasource selected that goes directly to the corresponding logs dashboard, but replacing &var-datasource=xyz-prom with &var-datasource=xyz-logs.

[yourbuddyconner]

Going to add my cry into the void for this seemingly-simple functionality! Would be a great gift if this were implemented, I literally just want to capitalize my dashboard titles with variables in them... It drives me nuts! 😵‍💫

[caroljung]

I just wanted to chime in here that this would be a really useful feature to have 🙏 pls consider implementing this grafana team, ty ❤️

[razvanpaul-mielcioiu]

I just wanted to add that while I see regex option in query vaiables a the solution for most of the use-cases here,
I'm using constant variables and multi-datasource panels, and this value manipulation would really help me managing those in a much easier way.

[lorenmh]

I would love to have this feature as well

ps @yourbuddyconner 👋

[mantydze]

I'm banging my head as well... I very much need this functionality

[mantydze]

As a temporary solution I use a hidden variable in between text box and actual query

raw_pattern - text box :  (user input)

pattern - Query (mysql) : SELECT REPLACE("$raw_pattern", "/", "_")

rows - Query : SHOW TAG VALUES FROM prod WITH KEY = 'my_key' WHERE "my_key" =~ /$pattern/

[somratdnutanix]

not sure why this feature request is such long time pending now.
I have an usecase where 'ip' comes as XX.XX.XX.XX:YYYY which needs to be formatted to ip-XX-XX-XX-XX for a query pattern.

[segfaultdoc]

Bump, would be useful for us as well. We have variables set in the format of ${THING-THONG} and would like to query using a regex expression that matches on THONG

[joseph-skupniewicz-sp]

👍 on this request. I needed to use prometheus' label_replace() as a clunky workaround, but this could be supported much better as a Grafana-native function

[hilalhmuhammed]

Wow, 6 years running... this might be some sort of record. I have been stretching my data source since a long time as this simple function is not available on the visualization front end..

[tangx]

It will be convenient for the dashboard of multiple data sources if this feature exists.

[skeagy]

Here's another specific real-world example where no solution is possible using regex from dashboard query variable or regex transform for panel legends... I have repeating rows for a variable specifying network interfaces. If these are virtual sub-interfaces (children of a parent physical interface), they are named like host_intf="router1_Eth0.123" where 123 is typically a VLAN ID. One property of these sub-interfaces is that there are no metrics related to errors or discards, but those metrics are associated with the underlying physical interface.

I want to process the repeating row variable to strip the dot and trailing numbers, which yields the underlying physical interface label. Then I can have panels in a row that show the virtual subinterface traffic and QoS drops, alongside the physical interface Ethernet discards and errors (using the row variable with string processing prior to interpolation in the query for the panel).

Without the straight-forward Grafana feature support for string processing a variable prior to interpolation, the only way I can do this is to redefine my label schema to include a reference to parent physical interface from each subinterface (either through the original metrics or through an "info" metric (e.g. * on() group_left() info_metric). But I have 2 million interfaces and 525,600 historical samples (1 year @ 1 min intervals). That's a big hassle to change, would take weeks just to load the data with 100% cpu even after I code it up, and it's millions extra in timeseries, etc.

I suppose I could also create a relational database to map these (ugly) and then chain it as a query variable... but that wouldn't work to align each instance of the virtual interface in a repeated row, with the associated physical interface variable in a panel in that row (and have the appropriate association for the panel in each repeated row).

There are many other examples- this is just a concrete one that doesn't need charts and numbers to understand (but it might need some network engineering domain specific knowledge to understand).

[tgriffitts-ns]

Another use case. I have a master/detail view in a dashboard. With a datalink in a column in the top panel which updates the rows in the detail panel. I use in the data link for the master panel:

/d/${__dashboard.uid}/${__dashboard}?var-alertnameselected=${__value}&${__all_variables} 

The problem is that var-alertnameselected is included TWICE because it is part of the builtin ${__all_variables} string. This works fine for single selection variables, as grafana seems to pick the first one. But when I change the variable to multi-select, it keeps adding more and more selected choices to the variable.

I want the datalink to be:

/d/${__dashboard.uid}/${__dashboard}?var-alertnameselected=${__value}&${__all_variables:regexp(s/var-alertnameselected=/var-ignoreme=/g)}

[jcsicard]

I too have been watching this issue for ages hoping for such a feature.
In the mean time, I came up with this hack to emulate the wanted behavior through a intermediate template variable of type query to a prometheus datasource which takes in a source (visible) template variable $OriginalVariable with RegexMatchPattern and outputs RegexReplacementPattern to be used as $RegexedReplacedVariable:

Variable name: RegexedReplacedVariable
Variable type: Query
Variable datasource: prometheus
Variable hide: variable
Variable query:

query_result(
	label_replace(
		label_replace(vector(0), "DummyLabel","$OriginalVariable", "__name__", ".*")
			, "DummyLabel", "RegexReplacementPattern", "DummyLabel", "RegexMatchPattern"
	)
)

Variable regex:
/\{DummyLabel="(.*)"\}.*/

Nasty workaround, but works...

[gelicia]

This has gone on a while, with some solutions made but nothing universal. I think the best next step for this ticket is to make feature requests for additional string manipulations that we currently don't support, and close this since we do support some for variables.

[marcojds]

I too have been watching this issue for ages hoping for such a feature. In the mean time, I came up with this hack to emulate the wanted behavior through a intermediate template variable of type query to a prometheus datasource which takes in a source (visible) template variable $OriginalVariable with RegexMatchPattern and outputs RegexReplacementPattern to be used as $RegexedReplacedVariable:

Variable name: RegexedReplacedVariable Variable type: Query Variable datasource: prometheus Variable hide: variable Variable query:

query_result(
	label_replace(
		label_replace(vector(0), "DummyLabel","$OriginalVariable", "__name__", ".*")
			, "DummyLabel", "RegexReplacementPattern", "DummyLabel", "RegexMatchPattern"
	)
)

Variable regex: /\{DummyLabel="(.*)"\}.*/

Nasty workaround, but works...

Could you share more about how this can be used within a metrics query? At least if this workaround works we have something to use in the meantime

[jcsicard]

I too have been watching this issue for ages hoping for such a feature. In the mean time, I came up with this hack to emulate the wanted behavior through a intermediate template variable of type query to a prometheus datasource which takes in a source (visible) template variable $OriginalVariable with RegexMatchPattern and outputs RegexReplacementPattern to be used as $RegexedReplacedVariable:
Variable name: RegexedReplacedVariable Variable type: Query Variable datasource: prometheus Variable hide: variable Variable query:

query_result(
	label_replace(
		label_replace(vector(0), "DummyLabel","$OriginalVariable", "__name__", ".*")
			, "DummyLabel", "RegexReplacementPattern", "DummyLabel", "RegexMatchPattern"
	)
)

Variable regex: /\{DummyLabel="(.*)"\}.*/
Nasty workaround, but works...

Could you share more about how this can be used within a metrics query? At least if this workaround works we have something to use in the meantime

So lets say you have a metric that has a label which is an interface's MAC address in a dashed format (aa-aa-aa-aa-aa-aa), and you want to filter that metric based on a MAC address from a templating variable (either from a label_values query, or just a user input text variable) but that variable's MAC address is in a colon form (aa:aa:aa:aa:aa:aa);

(For a final render I would hide the vMacDashed variable)

[mdvictor]

The supported variable formats are here. If your use-case is not supported please open a new separate feature request with the specific use case or open a contribution with a variable formatter.

I will close this issue as it has grown very large and contains quite a lot of use cases related to variable formats and encourage everyone to open issues for each separate case.

[CyBeRoni]

@mdvictor Those are string formatting options, not string manipulation options. Quite a difference between that and what I asked for almost 7 years ago.

But, this issue has 67 comments and 156 thumbs ups indicating support. If you would rather have a similar number of separate issues for the same thing instead of just one I won't stop you.

[mdvictor]

@CyBeRoni I understand your concerns and closing this issue doesn't mean that we are never going implement a solution for it or accept a PR around it. This issue has indeed been up for a while and has also collected quite a number of use cases, making it difficult to see what is supported now and what isn't.

At this moment we are also not looking into adding a universal solution for string manipulations in variables but would rather extend the format options linked above with ways to replace _ with - or adding toUpper/Lower options. This is why we are encouraging everyone to open specific issues or contributions per remaining use cases that will also help us better track the work around this.

[johnjeffers]

Ever spend hours on something you think will be simple, because it seems like such basic and obvious functionality that you're sure you must just be doing something wrong? You can't believe it's so difficult to get it working! Surely there's a solution that you're just missing, so you keep searching.

Then you finally come across a Github issue that's many years old, full of people asking for that exact thing, just as incredulous as you that this isn't implemented... and you scroll through, hoping to find an answer, only to have your hopes crushed when you see the maintainers close the issue with no resolution.

Yeah, I hate when that happens.

[johnjeffers]

Ok, so I'm not just complaining, I'll also add a solution I eventually came up with after lots of searching and cobbling together other, smarter people's ideas, in the hopes that it helps someone else.

Here's the problem I was trying to solve. I have a dashboard variable named namespace, which queries prometheus for kubernetes namespaces. We've prefixed all namespaces that have a particular app running in them. Let's say the prefix is xyz-. So for example we might have namespaces like xyz-foo-dev, xyz-bar-prod, etc.

Those namespaces might have an associated database named something like db-foo-dev or db-bar-prod. I wanted to get the value of namespace but chop off the xyz- so I could use only the bit after the prefix in a dashboard viz query. In other words, if I select xyz-foo-dev in the namespace var dropdown, I want to be able to get at just the foo-dev bit so I can use that as a filter in a query for the database where it needs to match db-foo-dev.

I start with a standard variable named namespace, using a prometheus datasource that looks like this:

I use that namespace var to do my standard namespace-based queries like so:

sum(max(rate(container_cpu_usage_seconds_total{cluster="$cluster", namespace=~"$namespace", container!=""}[3m0s])))

But the databases... well, I wish we would have just named them the same as the namespaces, but we didn't. So, here's how I'm getting this done.

I created another var named nsNoPrefix.

• Show on dashboard is set to Nothing because I don't want to see it as a dropdown.
• It filters on the active value of the namespace var so that's all it returns.
• Then it uses a regex to extract the substring (without the prefix) that I need.

It looks like this:

And here's that regex so you can copy/paste:

^xyz-(?!PLACEHOLDER_GARBAGE_THAT_WILL_NEVER_MATCH)(\S+)

Then I can use the nsNoPrefix var to query my database metrics, like so:

aws_rds_cpuutilization_average{dimension_DBInstanceIdentifier=~"db-$nsNoPrefix.*"}

I don't know if this is the best way to do this, or even a "right" way, but it seems to be working. 🤷

[aseure]

Any updates on this? The previous attempt #29754 was closed 3 years ago with the following reason:

We will think about an aternative approach, and get back to you when we came up with idea.

Is there any alternative being worked on? Or can we revive this old contribution to introduce some kind of string manipulation for variables? I'm just adding my +1 to all the reasons previously mentioned.