Cannot get Grafana to recognize Mimir monolitic instance, bare metal

[jnunezgts]

Hello all,

I’m getting started with Mimir and I’m trying to connect the following running on the same machine:

• Mimir mimir-2.11.0-1
• grafana-enterprise-10.4.1

I can see than Mimir is running and shows no errors (Console looks good http://localhost:9009/)
Grafana also looks good on http://localhost:3000/
And yet when I try to setup a new datasource pointing to my new Mimir instance I get an error saying:

401 Unauthorized - There was an error returned querying the Prometheus API.

I want to have applications pushing data to Mimir (for long term storage, aggregation). The applications will not use Prometheus but OpenTelemetry (but they have a way to push compatible metrics).
Then Grafana will be able to display time series, etc. from Mimir stored data.

I was also checking Grafana agent flow, but do you need to have now 3 components to forward the data? And that still doesn’t solve the issue of Grafana not being able to talk to Mimir.

Below are my settings:

Settings

/etc/mimir.config.yml

multitenancy_enabled: true

usage_stats:
  enabled: false

server:
  http_listen_port: 9009
  log_level: error

  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

limits:
  compactor_blocks_retention_period: 3m
  out_of_order_time_window: 10m

api:
  enable_otel_metadata_translation: true

distributor:
  ring:
    kvstore:
      store: memberlist
  pool:
    health_check_ingesters: true

ingester:
  ring:
    instance_interface_names:
      - vmx1.ZZZ
    instance_addr: 172.26.yy.xx

    min_ready_duration: 0s
    final_sleep: 0s
    num_tokens: 512

    kvstore:
      store: inmemory
    replication_factor: 1

blocks_storage:
  tsdb:
    dir: /scratch/grafana/tmp/mimir/tsdb

  bucket_store:
    sync_dir: /scratch/grafana/tmp/mimir/tsdb-sync

  filesystem:
    dir: /scratch/grafana/mimir/data/tsdb

compactor:
  data_dir: /scratch/grafana/tmp/mimir/compactor

ruler_storage:
  backend: local
  local:
    directory: /scratch/grafana/tmp/mimir/rules

/etc/grafana/grafana.ini

app_mode = production
instance_name = ${HOSTNAME}
[paths]
data = /scratch/grafana

[server]
protocol = http

domain = grafana.example.com

[analytics]
reporting_enabled = false
check_for_updates = true

[security]
admin_user = admin
admin_password = xxxx

[snapshots]

[dashboards]
versions_to_keep = 10
min_refresh_interval = 1m

[users]

[auth]

[auth.github]

name = GitHub
icon = github
enabled = false
allow_sign_up = true
auto_login = false
client_id = XXX
client_secret = YYYY
scopes = user:email,read:org
auth_url = https://ghe-myorg.example.com/login/oauth/authorize
token_url = https://ghe-myorg.example.com/login/oauth/access_token
api_url = https://ghe-myorg.example.com/api/v3/user

[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true

[smtp]
enabled = true
host = mail.example.com:25
from_address = myorgtechnologyus@example.com

[log]
[log.console]


[log.file]


log_rotate = true


max_days = 5

[alerting]
enabled = false

I went back and forth on the documentation and I don’t think I missed anything. Gladly I can paste my Mimir configuration here for review.

Thanks.

[56quarters]

To help troubleshoot, an you provide logs from Mimir when the request is made, the full response from Mimir, and the configuration in Grafana used for the Mimir datasource?

Can you make requests directly against the Mimir HTTP APIs?

[jnunezgts]

Hello Nick, thanks for looking into this.

Let gather those details and I will share them here.

[jnunezgts]

Slowly providing more details. So REST API seems to work:

root@myserver:~# curl http://myserver.example.com:9009/api/v1/status/config
{"status":"success","data":{}}root@myserver:~#

Logs coming up.

[jnunezgts]

Grafana.log

logger=cleanup t=2024-04-04T00:06:56.769097818-04:00 level=info msg="Completed cleanup jobs" duration=62.50355ms
logger=plugins.update.checker t=2024-04-04T00:06:56.877402452-04:00 level=info msg="Update check succeeded" duration=74.32209ms
logger=infra.usagestats t=2024-04-04T00:07:37.713415497-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T00:16:56.726849895-04:00 level=info msg="Completed cleanup jobs" duration=20.464111ms
logger=plugins.update.checker t=2024-04-04T00:16:56.875608201-04:00 level=info msg="Update check succeeded" duration=72.787448ms
logger=cleanup t=2024-04-04T00:26:56.723969752-04:00 level=info msg="Completed cleanup jobs" duration=17.154114ms
logger=plugins.update.checker t=2024-04-04T00:26:56.877536359-04:00 level=info msg="Update check succeeded" duration=74.195019ms
logger=licensing t=2024-04-04T00:36:56.721280887-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T00:36:56.721342363-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T00:36:56.742278398-04:00 level=info msg="Completed cleanup jobs" duration=36.10115ms
logger=plugins.update.checker t=2024-04-04T00:36:56.884303554-04:00 level=info msg="Update check succeeded" duration=80.78122ms
logger=infra.usagestats t=2024-04-04T00:37:37.712903585-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T00:46:56.821453873-04:00 level=info msg="Completed cleanup jobs" duration=115.245499ms
logger=plugins.update.checker t=2024-04-04T00:46:56.87056952-04:00 level=info msg="Update check succeeded" duration=67.152134ms
logger=cleanup t=2024-04-04T00:56:56.716293853-04:00 level=info msg="Completed cleanup jobs" duration=10.141478ms
logger=plugins.update.checker t=2024-04-04T00:56:56.91367398-04:00 level=info msg="Update check succeeded" duration=110.102649ms
logger=cleanup t=2024-04-04T01:06:56.737912408-04:00 level=info msg="Completed cleanup jobs" duration=31.056248ms
logger=plugins.update.checker t=2024-04-04T01:06:56.876999801-04:00 level=info msg="Update check succeeded" duration=73.508579ms
logger=infra.usagestats t=2024-04-04T01:07:37.713395448-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T01:16:56.722440093-04:00 level=info msg="Completed cleanup jobs" duration=16.183283ms
logger=plugins.update.checker t=2024-04-04T01:16:56.907745451-04:00 level=info msg="Update check succeeded" duration=104.07948ms
logger=cleanup t=2024-04-04T01:26:56.794308401-04:00 level=info msg="Completed cleanup jobs" duration=87.484664ms
logger=plugins.update.checker t=2024-04-04T01:26:56.883430241-04:00 level=info msg="Update check succeeded" duration=80.026108ms
logger=cleanup t=2024-04-04T01:36:56.724526856-04:00 level=info msg="Completed cleanup jobs" duration=18.415352ms
logger=licensing t=2024-04-04T01:36:56.755709956-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T01:36:56.755759096-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=plugins.update.checker t=2024-04-04T01:36:56.937775395-04:00 level=info msg="Update check succeeded" duration=134.834807ms
logger=infra.usagestats t=2024-04-04T01:37:37.713428419-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T01:46:56.72636422-04:00 level=info msg="Completed cleanup jobs" duration=19.997053ms
logger=plugins.update.checker t=2024-04-04T01:46:56.902113797-04:00 level=info msg="Update check succeeded" duration=99.095432ms
logger=cleanup t=2024-04-04T01:56:56.726024479-04:00 level=info msg="Completed cleanup jobs" duration=19.280771ms
logger=plugins.update.checker t=2024-04-04T01:56:56.868624002-04:00 level=info msg="Update check succeeded" duration=65.018787ms
logger=cleanup t=2024-04-04T02:06:56.72459332-04:00 level=info msg="Completed cleanup jobs" duration=18.195233ms
logger=plugins.update.checker t=2024-04-04T02:06:56.875503539-04:00 level=info msg="Update check succeeded" duration=72.530662ms
logger=infra.usagestats t=2024-04-04T02:07:37.713886828-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T02:16:56.726309056-04:00 level=info msg="Completed cleanup jobs" duration=20.118556ms
logger=plugins.update.checker t=2024-04-04T02:16:56.877901717-04:00 level=info msg="Update check succeeded" duration=74.403023ms
logger=cleanup t=2024-04-04T02:26:56.725024966-04:00 level=info msg="Completed cleanup jobs" duration=18.804279ms
logger=plugins.update.checker t=2024-04-04T02:26:56.865212596-04:00 level=info msg="Update check succeeded" duration=62.463441ms
logger=licensing t=2024-04-04T02:36:56.707390171-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T02:36:56.707458577-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T02:36:56.739953941-04:00 level=info msg="Completed cleanup jobs" duration=33.411012ms
logger=plugins.update.checker t=2024-04-04T02:36:56.871445956-04:00 level=info msg="Update check succeeded" duration=68.182358ms
logger=infra.usagestats t=2024-04-04T02:37:37.71277659-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T02:46:56.726583467-04:00 level=info msg="Completed cleanup jobs" duration=20.098874ms
logger=plugins.update.checker t=2024-04-04T02:46:56.869556876-04:00 level=info msg="Update check succeeded" duration=66.517087ms
logger=cleanup t=2024-04-04T02:56:56.716517223-04:00 level=info msg="Completed cleanup jobs" duration=10.198078ms
logger=plugins.update.checker t=2024-04-04T02:56:56.868214193-04:00 level=info msg="Update check succeeded" duration=64.580216ms
logger=cleanup t=2024-04-04T03:06:56.723484027-04:00 level=info msg="Completed cleanup jobs" duration=16.931767ms
logger=plugins.update.checker t=2024-04-04T03:06:56.909795421-04:00 level=info msg="Update check succeeded" duration=106.566955ms
logger=infra.usagestats t=2024-04-04T03:07:37.713233737-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T03:16:56.725663568-04:00 level=info msg="Completed cleanup jobs" duration=19.239899ms
logger=plugins.update.checker t=2024-04-04T03:16:56.866787856-04:00 level=info msg="Update check succeeded" duration=63.779394ms
logger=cleanup t=2024-04-04T03:26:56.735439639-04:00 level=info msg="Completed cleanup jobs" duration=29.314771ms
logger=plugins.update.checker t=2024-04-04T03:26:56.868853942-04:00 level=info msg="Update check succeeded" duration=65.875417ms
logger=cleanup t=2024-04-04T03:36:56.719031375-04:00 level=info msg="Completed cleanup jobs" duration=12.508318ms
logger=licensing t=2024-04-04T03:36:56.732538135-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T03:36:56.732590747-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=plugins.update.checker t=2024-04-04T03:36:56.920191681-04:00 level=info msg="Update check succeeded" duration=117.19417ms
logger=infra.usagestats t=2024-04-04T03:37:37.712636089-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T03:46:56.716316592-04:00 level=info msg="Completed cleanup jobs" duration=10.149473ms
logger=plugins.update.checker t=2024-04-04T03:46:56.902400546-04:00 level=info msg="Update check succeeded" duration=99.049739ms
logger=cleanup t=2024-04-04T03:56:56.725188451-04:00 level=info msg="Completed cleanup jobs" duration=19.008869ms
logger=plugins.update.checker t=2024-04-04T03:56:56.884305354-04:00 level=info msg="Update check succeeded" duration=80.593953ms
logger=cleanup t=2024-04-04T04:06:56.727708087-04:00 level=info msg="Completed cleanup jobs" duration=21.353634ms
logger=plugins.update.checker t=2024-04-04T04:06:56.886607257-04:00 level=info msg="Update check succeeded" duration=83.583105ms
logger=infra.usagestats t=2024-04-04T04:07:37.713303827-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T04:16:56.725312675-04:00 level=info msg="Completed cleanup jobs" duration=18.799799ms
logger=plugins.update.checker t=2024-04-04T04:16:56.891820824-04:00 level=info msg="Update check succeeded" duration=88.523228ms
logger=cleanup t=2024-04-04T04:26:56.722388578-04:00 level=info msg="Completed cleanup jobs" duration=16.146136ms
logger=plugins.update.checker t=2024-04-04T04:26:56.875731214-04:00 level=info msg="Update check succeeded" duration=72.062687ms
logger=licensing t=2024-04-04T04:36:56.707050791-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T04:36:56.707118557-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T04:36:56.724248467-04:00 level=info msg="Completed cleanup jobs" duration=18.008858ms
logger=plugins.update.checker t=2024-04-04T04:36:56.905678502-04:00 level=info msg="Update check succeeded" duration=102.780314ms
logger=infra.usagestats t=2024-04-04T04:37:37.713030024-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T04:46:56.722708184-04:00 level=info msg="Completed cleanup jobs" duration=16.167903ms
logger=plugins.update.checker t=2024-04-04T04:46:56.877893404-04:00 level=info msg="Update check succeeded" duration=74.776607ms
logger=cleanup t=2024-04-04T04:56:56.725226454-04:00 level=info msg="Completed cleanup jobs" duration=18.873049ms
logger=plugins.update.checker t=2024-04-04T04:56:56.890160861-04:00 level=info msg="Update check succeeded" duration=86.711378ms
logger=cleanup t=2024-04-04T05:06:56.736047675-04:00 level=info msg="Completed cleanup jobs" duration=29.631919ms
logger=plugins.update.checker t=2024-04-04T05:06:56.890398339-04:00 level=info msg="Update check succeeded" duration=87.555719ms
logger=infra.usagestats t=2024-04-04T05:07:37.714316929-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T05:16:56.717570451-04:00 level=info msg="Completed cleanup jobs" duration=11.335443ms
logger=plugins.update.checker t=2024-04-04T05:16:56.881708312-04:00 level=info msg="Update check succeeded" duration=78.31763ms
logger=cleanup t=2024-04-04T05:26:56.725247697-04:00 level=info msg="Completed cleanup jobs" duration=18.799408ms
logger=plugins.update.checker t=2024-04-04T05:26:57.031717054-04:00 level=info msg="Update check succeeded" duration=228.751847ms
logger=cleanup t=2024-04-04T05:36:56.737995104-04:00 level=info msg="Completed cleanup jobs" duration=31.666833ms
logger=licensing t=2024-04-04T05:36:56.745089077-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T05:36:56.745136317-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=plugins.update.checker t=2024-04-04T05:36:56.867153782-04:00 level=info msg="Update check succeeded" duration=64.106007ms
logger=infra.usagestats t=2024-04-04T05:37:37.712634495-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T05:46:56.725443866-04:00 level=info msg="Completed cleanup jobs" duration=18.937119ms
logger=plugins.update.checker t=2024-04-04T05:46:56.876155319-04:00 level=info msg="Update check succeeded" duration=73.108129ms
logger=cleanup t=2024-04-04T05:56:56.716357951-04:00 level=info msg="Completed cleanup jobs" duration=10.128726ms
logger=plugins.update.checker t=2024-04-04T05:56:56.865146649-04:00 level=info msg="Update check succeeded" duration=61.775135ms
logger=cleanup t=2024-04-04T06:06:56.722364778-04:00 level=info msg="Completed cleanup jobs" duration=15.855212ms
logger=plugins.update.checker t=2024-04-04T06:06:56.878758137-04:00 level=info msg="Update check succeeded" duration=75.662081ms
logger=infra.usagestats t=2024-04-04T06:07:37.712575074-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T06:16:56.723775717-04:00 level=info msg="Completed cleanup jobs" duration=17.122602ms
logger=plugins.update.checker t=2024-04-04T06:16:56.866155363-04:00 level=info msg="Update check succeeded" duration=62.790272ms
logger=cleanup t=2024-04-04T06:26:56.716440916-04:00 level=info msg="Completed cleanup jobs" duration=10.30599ms
logger=plugins.update.checker t=2024-04-04T06:26:56.889286258-04:00 level=info msg="Update check succeeded" duration=86.538385ms
logger=licensing t=2024-04-04T06:36:56.716245305-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T06:36:56.71630593-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T06:36:56.71759786-04:00 level=info msg="Completed cleanup jobs" duration=11.455634ms
logger=plugins.update.checker t=2024-04-04T06:36:56.881374844-04:00 level=info msg="Update check succeeded" duration=77.960062ms
logger=infra.usagestats t=2024-04-04T06:37:37.71353776-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T06:46:56.723522946-04:00 level=info msg="Completed cleanup jobs" duration=16.929298ms
logger=plugins.update.checker t=2024-04-04T06:46:56.86788666-04:00 level=info msg="Update check succeeded" duration=64.845442ms
logger=cleanup t=2024-04-04T06:56:56.716402162-04:00 level=info msg="Completed cleanup jobs" duration=10.198623ms
logger=plugins.update.checker t=2024-04-04T06:56:56.869961664-04:00 level=info msg="Update check succeeded" duration=66.479681ms
logger=cleanup t=2024-04-04T07:06:56.72520892-04:00 level=info msg="Completed cleanup jobs" duration=18.811822ms
logger=plugins.update.checker t=2024-04-04T07:06:56.914045225-04:00 level=info msg="Update check succeeded" duration=110.734103ms
logger=infra.usagestats t=2024-04-04T07:07:37.714585294-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T07:16:56.717691179-04:00 level=info msg="Completed cleanup jobs" duration=11.374521ms
logger=plugins.update.checker t=2024-04-04T07:16:57.11258344-04:00 level=info msg="Update check succeeded" duration=309.55967ms
logger=cleanup t=2024-04-04T07:26:56.716873182-04:00 level=info msg="Completed cleanup jobs" duration=10.461474ms
logger=plugins.update.checker t=2024-04-04T07:26:56.877743254-04:00 level=info msg="Update check succeeded" duration=74.918582ms
logger=licensing t=2024-04-04T07:36:56.723915829-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T07:36:56.724385427-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T07:36:56.726744531-04:00 level=info msg="Completed cleanup jobs" duration=20.388793ms
logger=plugins.update.checker t=2024-04-04T07:36:56.884841166-04:00 level=info msg="Update check succeeded" duration=81.923868ms
logger=infra.usagestats t=2024-04-04T07:37:37.713622524-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T07:46:56.712907095-04:00 level=info msg="Completed cleanup jobs" duration=6.810502ms
logger=plugins.update.checker t=2024-04-04T07:46:56.909165371-04:00 level=info msg="Update check succeeded" duration=105.552917ms
logger=cleanup t=2024-04-04T07:56:56.716435619-04:00 level=info msg="Completed cleanup jobs" duration=10.236825ms
logger=plugins.update.checker t=2024-04-04T07:56:56.883176554-04:00 level=info msg="Update check succeeded" duration=79.467333ms
logger=cleanup t=2024-04-04T08:06:56.751776798-04:00 level=info msg="Completed cleanup jobs" duration=44.764292ms
logger=plugins.update.checker t=2024-04-04T08:06:56.886133937-04:00 level=info msg="Update check succeeded" duration=82.479303ms
logger=infra.usagestats t=2024-04-04T08:07:37.713719575-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T08:16:56.727206641-04:00 level=info msg="Completed cleanup jobs" duration=20.896901ms
logger=plugins.update.checker t=2024-04-04T08:16:56.872714455-04:00 level=info msg="Update check succeeded" duration=69.918156ms
logger=cleanup t=2024-04-04T08:26:56.728725177-04:00 level=info msg="Completed cleanup jobs" duration=22.014717ms
logger=plugins.update.checker t=2024-04-04T08:26:56.876719317-04:00 level=info msg="Update check succeeded" duration=73.485445ms
logger=licensing t=2024-04-04T08:36:56.716186638-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T08:36:56.71625381-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T08:36:56.72552603-04:00 level=info msg="Completed cleanup jobs" duration=19.16846ms
logger=plugins.update.checker t=2024-04-04T08:36:56.941585229-04:00 level=info msg="Update check succeeded" duration=138.389734ms
logger=infra.usagestats t=2024-04-04T08:37:37.71329434-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T08:46:56.716670769-04:00 level=info msg="Completed cleanup jobs" duration=10.467006ms
logger=plugins.update.checker t=2024-04-04T08:46:56.92444484-04:00 level=info msg="Update check succeeded" duration=121.124506ms
logger=cleanup t=2024-04-04T08:56:56.725537171-04:00 level=info msg="Completed cleanup jobs" duration=19.284257ms
logger=plugins.update.checker t=2024-04-04T08:56:56.872283236-04:00 level=info msg="Update check succeeded" duration=68.500414ms
logger=cleanup t=2024-04-04T09:06:56.716377453-04:00 level=info msg="Completed cleanup jobs" duration=10.184545ms
logger=plugins.update.checker t=2024-04-04T09:06:56.893097748-04:00 level=info msg="Update check succeeded" duration=90.15813ms
logger=infra.usagestats t=2024-04-04T09:07:37.712841893-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T09:16:56.735030115-04:00 level=info msg="Completed cleanup jobs" duration=28.011309ms
logger=plugins.update.checker t=2024-04-04T09:16:57.038439424-04:00 level=info msg="Update check succeeded" duration=234.699072ms
logger=cleanup t=2024-04-04T09:26:56.725602698-04:00 level=info msg="Completed cleanup jobs" duration=19.118784ms
logger=plugins.update.checker t=2024-04-04T09:26:57.32915098-04:00 level=info msg="Update check succeeded" duration=525.980001ms
logger=cleanup t=2024-04-04T09:36:56.734298728-04:00 level=info msg="Completed cleanup jobs" duration=27.971358ms
logger=licensing t=2024-04-04T09:36:56.752079102-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T09:36:56.752268496-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=plugins.update.checker t=2024-04-04T09:36:56.918352859-04:00 level=info msg="Update check succeeded" duration=115.519335ms
logger=infra.usagestats t=2024-04-04T09:37:37.713364707-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T09:46:56.734949435-04:00 level=info msg="Completed cleanup jobs" duration=28.279511ms
logger=plugins.update.checker t=2024-04-04T09:46:56.893365539-04:00 level=info msg="Update check succeeded" duration=90.122565ms
logger=cleanup t=2024-04-04T09:56:56.72522906-04:00 level=info msg="Completed cleanup jobs" duration=18.763541ms
logger=plugins.update.checker t=2024-04-04T09:56:56.921866771-04:00 level=info msg="Update check succeeded" duration=118.883026ms
logger=cleanup t=2024-04-04T10:06:56.735939461-04:00 level=info msg="Completed cleanup jobs" duration=29.179429ms
logger=plugins.update.checker t=2024-04-04T10:06:56.975041417-04:00 level=info msg="Update check succeeded" duration=171.737021ms
logger=infra.usagestats t=2024-04-04T10:07:37.713101161-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T10:16:56.723710198-04:00 level=info msg="Completed cleanup jobs" duration=17.033845ms
logger=plugins.update.checker t=2024-04-04T10:16:56.885878873-04:00 level=info msg="Update check succeeded" duration=82.70595ms
logger=cleanup t=2024-04-04T10:26:56.717543065-04:00 level=info msg="Completed cleanup jobs" duration=11.351847ms
logger=plugins.update.checker t=2024-04-04T10:26:56.895610597-04:00 level=info msg="Update check succeeded" duration=92.168078ms
logger=plugins.update.checker t=2024-04-04T10:36:56.882335377-04:00 level=info msg="Update check succeeded" duration=78.944647ms
logger=licensing t=2024-04-04T10:36:58.046110863-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T10:36:58.046185844-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T10:36:58.048787905-04:00 level=info msg="Completed cleanup jobs" duration=1.341952994s
logger=infra.usagestats t=2024-04-04T10:37:37.713452092-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T10:46:56.725413725-04:00 level=info msg="Completed cleanup jobs" duration=19.183886ms
logger=plugins.update.checker t=2024-04-04T10:46:56.90587014-04:00 level=info msg="Update check succeeded" duration=102.09976ms
logger=cleanup t=2024-04-04T10:56:56.717628602-04:00 level=info msg="Completed cleanup jobs" duration=11.287015ms
logger=plugins.update.checker t=2024-04-04T10:56:56.905173136-04:00 level=info msg="Update check succeeded" duration=102.365729ms
logger=cleanup t=2024-04-04T11:06:56.735772022-04:00 level=info msg="Completed cleanup jobs" duration=29.387227ms
logger=plugins.update.checker t=2024-04-04T11:06:56.87737259-04:00 level=info msg="Update check succeeded" duration=74.571933ms
logger=infra.usagestats t=2024-04-04T11:07:37.713522893-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T11:16:56.725775469-04:00 level=info msg="Completed cleanup jobs" duration=19.496638ms
logger=plugins.update.checker t=2024-04-04T11:16:56.893143228-04:00 level=info msg="Update check succeeded" duration=89.69646ms
logger=cleanup t=2024-04-04T11:26:56.748280735-04:00 level=info msg="Completed cleanup jobs" duration=41.507063ms
logger=plugins.update.checker t=2024-04-04T11:26:56.879801129-04:00 level=info msg="Update check succeeded" duration=76.173005ms
logger=licensing t=2024-04-04T11:36:56.70801184-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T11:36:56.708078636-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T11:36:56.726189505-04:00 level=info msg="Completed cleanup jobs" duration=19.733404ms
logger=plugins.update.checker t=2024-04-04T11:36:56.876508093-04:00 level=info msg="Update check succeeded" duration=73.010017ms
logger=infra.usagestats t=2024-04-04T11:37:37.713045943-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T11:46:56.724545452-04:00 level=info msg="Completed cleanup jobs" duration=17.730542ms
logger=plugins.update.checker t=2024-04-04T11:46:56.934821846-04:00 level=info msg="Update check succeeded" duration=131.468652ms
logger=cleanup t=2024-04-04T11:56:56.820446247-04:00 level=info msg="Completed cleanup jobs" duration=114.346091ms
logger=plugins.update.checker t=2024-04-04T11:56:56.941554177-04:00 level=info msg="Update check succeeded" duration=137.889854ms
logger=cleanup t=2024-04-04T12:06:56.723821676-04:00 level=info msg="Completed cleanup jobs" duration=17.134146ms
logger=plugins.update.checker t=2024-04-04T12:06:57.663473757-04:00 level=info msg="Update check succeeded" duration=860.121893ms
logger=infra.usagestats t=2024-04-04T12:07:37.71319321-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T12:16:56.71664109-04:00 level=info msg="Completed cleanup jobs" duration=10.425347ms
logger=plugins.update.checker t=2024-04-04T12:16:56.881244424-04:00 level=info msg="Update check succeeded" duration=77.663101ms
logger=cleanup t=2024-04-04T12:26:56.748761369-04:00 level=info msg="Completed cleanup jobs" duration=42.167747ms
logger=plugins.update.checker t=2024-04-04T12:26:56.875891844-04:00 level=info msg="Update check succeeded" duration=72.555935ms
logger=cleanup t=2024-04-04T12:36:56.735013318-04:00 level=info msg="Completed cleanup jobs" duration=28.813996ms
logger=licensing t=2024-04-04T12:36:56.736808634-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T12:36:56.736846199-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=plugins.update.checker t=2024-04-04T12:36:57.739581549-04:00 level=info msg="Update check succeeded" duration=935.832104ms
logger=infra.usagestats t=2024-04-04T12:37:37.713280875-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T12:46:56.718481479-04:00 level=info msg="Completed cleanup jobs" duration=12.24279ms
logger=plugins.update.checker t=2024-04-04T12:46:56.907237244-04:00 level=info msg="Update check succeeded" duration=104.46147ms
logger=cleanup t=2024-04-04T12:56:56.722493232-04:00 level=info msg="Completed cleanup jobs" duration=15.82373ms
logger=plugins.update.checker t=2024-04-04T12:56:58.569193006-04:00 level=info msg="Update check succeeded" duration=1.76599696s
logger=cleanup t=2024-04-04T13:06:56.726063436-04:00 level=info msg="Completed cleanup jobs" duration=19.865308ms
logger=plugins.update.checker t=2024-04-04T13:06:56.891917868-04:00 level=info msg="Update check succeeded" duration=88.445966ms
logger=infra.usagestats t=2024-04-04T13:07:37.713115995-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T13:16:56.722576191-04:00 level=info msg="Completed cleanup jobs" duration=16.01688ms
logger=plugins.update.checker t=2024-04-04T13:16:56.88127318-04:00 level=info msg="Update check succeeded" duration=78.083763ms
logger=cleanup t=2024-04-04T13:26:56.725803292-04:00 level=info msg="Completed cleanup jobs" duration=19.424538ms
logger=plugins.update.checker t=2024-04-04T13:26:56.888630129-04:00 level=info msg="Update check succeeded" duration=85.671895ms
logger=licensing t=2024-04-04T13:36:56.707549616-04:00 level=info msg="Validated license token" appURL=http://myserver.example.com:3000/ source=disk status=NotFound
logger=licensing.renewal t=2024-04-04T13:36:56.707612412-04:00 level=warn msg="failed to load or validate token" err="license token file not found: /scratch/grafana/license.jwt"
logger=cleanup t=2024-04-04T13:36:56.731176975-04:00 level=info msg="Completed cleanup jobs" duration=25.071439ms
logger=plugins.update.checker t=2024-04-04T13:36:56.997484407-04:00 level=info msg="Update check succeeded" duration=193.972221ms
logger=grafana.update.checker t=2024-04-04T13:36:57.012794739-04:00 level=info msg="Update check succeeded" duration=200.901383ms
logger=infra.usagestats t=2024-04-04T13:37:37.713762286-04:00 level=info msg="Usage stats are ready to report"
logger=cleanup t=2024-04-04T13:46:56.725374831-04:00 level=info msg="Completed cleanup jobs" duration=19.153844ms
logger=plugins.update.checker t=2024-04-04T13:46:57.139959572-04:00 level=info msg="Update check succeeded" duration=336.26105ms
logger=cleanup t=2024-04-04T13:56:56.724286232-04:00 level=info msg="Completed cleanup jobs" duration=17.372736ms
logger=plugins.update.checker t=2024-04-04T13:56:57.827506796-04:00 level=info msg="Update check succeeded" duration=1.023956143s
logger=context userId=0 orgId=0 uname= t=2024-04-04T13:57:07.444012622-04:00 level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=192.168.237.8 time_ms=3 duration=3.003064ms size=24 referer= handler=/login status_source=server
logger=context userId=0 orgId=0 uname= t=2024-04-04T13:57:07.455647217-04:00 level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.237.8 time_ms=0 duration=454.751µs size=47 referer= handler=/ status_source=server
logger=context userId=0 orgId=0 uname= t=2024-04-04T13:57:07.46583692-04:00 level=info msg="Request Completed" method=GET path=/user/auth-tokens/rotate status=302 remote_addr=192.168.237.8 time_ms=2 duration=2.782993ms size=24 referer= handler=/user/auth-tokens/rotate status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:57:08.185578821-04:00 level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.237.8 time_ms=1 duration=1.120103ms size=0 referer= handler=/api/live/ws status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:57:30.662708376-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.464926ms size=176 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:57:44.52948942-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=1 duration=1.800559ms size=226 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:58:24.501812525-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.412433ms size=173 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:59:18.986339458-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.723644ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:59:19.968981083-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.3781ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:59:20.151065819-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.191872ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:59:20.349171102-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.205988ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:59:33.18445521-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.324488ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T13:59:33.869101021-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.465687ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T14:00:54.052453853-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.389476ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=context userId=2 orgId=1 uname=jnunez t=2024-04-04T14:00:54.79508995-04:00 level=info msg="Request Completed" method=GET path=/api/datasources/uid/fdgrc69ynw4xsc/health status=400 remote_addr=192.168.237.8 time_ms=2 duration=2.19587ms size=103 referer=http://st-avtdev-vm01:3000/connections/datasources/edit/fdgrc69ynw4xsc handler=/api/datasources/uid/:uid/health status_source=server
logger=cleanup t=2024-04-04T14:06:56.724808542-04:00 level=info msg="Completed cleanup jobs" duration=18.647143ms
logger=plugins.update.checker t=2024-04-04T14:06:57.397942205-04:00 level=info msg="Update check succeeded" duration=594.487269ms
logger=infra.usagestats t=2024-04-04T14:07:37.713114733-04:00 level=info msg="Usage stats are ready to report"

[jnunezgts]

Mimir (journalctl):

rg/mimir-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 27 16:11:57 st-avtdev-vm01.striketechnologies.com mimir[6568]: ts=2024-03-27T20:11:57.066581892Z caller=reporter.go:174 level=info msg="failed to send anonymous usage stats report" err="send the report to the stats server: Post \"https://stats.grafana.org/mimir-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 27 16:12:57 st-avtdev-vm01.striketechnologies.com mimir[6568]: ts=2024-03-27T20:12:57.067996617Z caller=reporter.go:174 level=info msg="failed to send anonymous usage stats report" err="send the report to the stats server: Post \"https://stats.grafana.org/mimir-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Mar 27 18:25:13 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Deactivated successfully.
Mar 27 18:25:13 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Consumed 8min 32.145s CPU time.
Mar 27 18:25:13 st-avtdev-vm01.striketechnologies.com mimir[28789]: ts=2024-03-27T22:25:13.466643414Z caller=modules.go:583 level=info msg="The querier worker has not been configured with either the query-frontend or query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler address for querier worker." address=127.0.0.1:9095
Mar 27 18:25:14 st-avtdev-vm01.striketechnologies.com mimir[28789]: ts=2024-03-27T22:25:14.652593467Z caller=mimir.go:850 level=info msg="Application started"
Mar 27 18:33:31 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Deactivated successfully.
Mar 27 18:33:31 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Consumed 1.548s CPU time.
Mar 27 18:33:31 st-avtdev-vm01.striketechnologies.com mimir[28878]: ts=2024-03-27T22:33:31.795709226Z caller=modules.go:583 level=info msg="The querier worker has not been configured with either the query-frontend or query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler address for querier worker." address=127.0.0.1:9095
Mar 27 18:33:33 st-avtdev-vm01.striketechnologies.com mimir[28878]: ts=2024-03-27T22:33:33.114125273Z caller=mimir.go:850 level=info msg="Application started"
Mar 27 18:40:19 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Deactivated successfully.
Mar 27 18:40:19 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Consumed 1.327s CPU time.
Mar 27 18:40:19 st-avtdev-vm01.striketechnologies.com mimir[28964]: ts=2024-03-27T22:40:19.103312919Z caller=modules.go:583 level=info msg="The querier worker has not been configured with either the query-frontend or query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler address for querier worker." address=127.0.0.1:9095
Mar 27 18:40:20 st-avtdev-vm01.striketechnologies.com mimir[28964]: ts=2024-03-27T22:40:20.45468141Z caller=mimir.go:850 level=info msg="Application started"
Mar 27 18:43:20 st-avtdev-vm01.striketechnologies.com systemd[1]: mimir.service: Deactivated successfully.
Mar 27 18:43:20 st-avtdev-vm01.striketechnologies.com mimir[28992]: ts=2024-03-27T22:43:20.270447102Z caller=modules.go:583 level=info msg="The querier worker has not been configured with either the query-frontend or query-scheduler address. Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration. If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler address for querier worker." address=127.0.0.1:9095
Mar 27 18:43:21 st-avtdev-vm01.striketechnologies.com mimir[28992]: ts=2024-03-27T22:43:21.627693707Z caller=mimir.go:850 level=info msg="Application started"

[56quarters]

It looks like you have multi-tenancy enabled in Mimir (multitenancy_enabled: true). When enabled, you need to pass a header to Mimir to identify which tenant to write metrics for or to query: docs. Hard to say without the datasource configuration but I suspect this is what's missing.

[jnunezgts]

Yeah, I did not populate that. Let me try that and see if that works.

[jnunezgts]

I added the HTTP Header as I have multi-tenant enabled but the real issue what the URL. So I changed the port to use the default, on Mimir, and also the end URL on the Grafana datasource:

http://myserver.example:8080/prometheus

And it works now!