Errors with stackdriver ds and dashboard variables used in the Project field when alarm configured

[tomsucho]

What happened:
Dashboard has a variable that selects a GCP project id, which is then referenced in all charts in the dashboard to plot stats relevant only for the user selected project from the drop down menu on the dashboard.
I added a Stackdriver data source to poll agent metrics (instance memory stats) and since the query requires "Project" I have selected the "Templates Variables" from the drop down and then selected my variable $grafana_project_id which was listed there. So all fine and the chart for memory stats from agent metrics worked okay when switching the project id on the dashboard. However I noticed alarms (Telegram notifications) set based on this memory chart didn't work and I noticed the following errors in the grafana service logs:

grafana_1 | t=2020-05-06T15:55:01+0000 lvl=info msg=Requesting logger=data-proxy-log url=https://content-monitoring.googleapis.com/v3/projects/$grafana_project_id/timeSeries grafana_1 | t=2020-05-06T15:55:01+0000 lvl=eror msg="Request failed" logger=tsdb.stackdriver status="400 Bad Request" body="{\n \"error\": {\n \"code\": 400,\n \"message\": \"Invalid project ID.\",\n \"status\": \"INVALID_ARGUMENT\"\n }\n}\n"

so it can be seen that the variable was not interpolated in the URI.
When I remove the template variable $grafana_project_id from the Project field in the Query, the errors disappear.

What you expected to happen:
No errors in the logs and alarming to Telegram appearing okay.
How to reproduce it (as minimally and precisely as possible):
Please check description.
Anything else we need to know?:

Environment:

• Grafana version:
  image: grafana/grafana:6.7.2
• Data source type & version:
  Stackdriver to GCP
• OS Grafana is installed on:
  image: grafana/grafana:6.7.2
• User OS & Browser:
  MacOs Chrome
• Grafana plugins:
• Others:

[papagian]

duplicate of #6557