Integration with Grafana doesn't work

[ssveta7ak]

What happened?

Integration with Grafana doesn't work.
The user couldn't log in with IdP.

What did you expect to happen?

How'd it happen?

1. Follow the guide to set up Grafana:
   https://www.pomerium.com/docs/guides/grafana
2. Log into Grafana as an Admin and create a new user linked to IdP.
3. Log out.
4. Attempt to log back in.
   The IdP login window doesn't appear.
   The user is presented with the Grafana login page.
   

What's your environment like?

pomerium: 0.22.2

What's your config.yaml?

  - from: https://grafana.localhost.pomerium.io
    to: http://grafana:3000
    host_rewrite_header: true
    pass_identity_headers: true
    policy:
      - allow:
         or:
          - domain:
                is: pomerium.com

What's your docker-compose.yaml?

 grafana:
    image: grafana/grafana:latest
    user: "1000:1000"
    networks:
      main: {}
    ports:
      - 3000:3000
    environment:
      - GF_AUTH_SIGNOUT_REDIRECT_URL=https://grafana.localhost.pomerium.io/.pomerium/sign_out
      - GF_AUTH_JWT_ENABLED=true
      - GF_AUTH_JWT_HEADER_NAME=X_POMERIUM_JWT_ASSERTION
      - GF_AUTH_JWT_EMAIL_CLAIM=email
      - GF_AUTH_JWT_JWK_SET_URL=https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json
      - GF_AUTH_JWT_CACHE_TTL=60m
      - GF_AUTH_JWT_AUTO_SIGN_UP=true
    volumes:
      - ./grafana-storage:/var/lib/grafana/plugins:rw

What did you see in the logs?

grafana_core-pomerium-1  | {"level":"info","service":"authorize","request-id":"06985ae7-cb6e-4894-a25d-02b638db3c0b","check-request-id":"06985ae7-cb6e-4894-a25d-02b638db3c0b","method":"GET","path":"/","host":"grafana.localhost.pomerium.io","query":"","ip":"172.31.0.1","session-id":"38dab363-cc2d-474d-a7c9-3a3a7560d430","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"google-oauth2|104583909997507305030","email":"srekuts@pomerium.com","time":"2023-08-09T17:03:29Z","message":"authorize check"}
grafana_core-grafana-1   | logger=context userId=0 orgId=0 uname= t=2023-08-09T17:03:29.569601649Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=172.31.0.1 time_ms=0 duration=487.182µs size=29 referer= handler=/

Additional context

Add any other context about the problem here.

[kenjenkins]

Do we need to also add GF_AUTH_DISABLE_LOGIN_FORM=true to the Grafana container environment variables?

[desimone]

@ssveta7ak will you see if the suggested helps?

[ssveta7ak]

@desimone, @kenjenkins
GF_AUTH_DISABLE_LOGIN_FORM=true doesn't help.
The IdP Log in still doesn't work.
Without this parameter, it's impossible to log into Grafana.

[pschiffe]

GF_AUTH_JWT_HEADER_NAME must be set to X-Pomerium-Jwt-Assertion, it's case sensitive. I had the same issue: https://discuss.pomerium.com/t/bugs-in-the-grafana-guide/293/2

[ssveta7ak]

logs

grafana_core-pomerium-1  | {"level":"info","service":"authorize","request-id":"5318e70e-028d-4824-b605-50093150f5dd","check-request-id":"5318e70e-028d-4824-b605-50093150f5dd","method":"GET","path":"/","host":"grafana.localhost.pomerium.io","query":"","ip":"172.23.0.1","session-id":"8b05c98c-a488-4e54-88c4-fea3a1365fa7","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"google-oauth2|104583909997507305030","email":"user@pomerium.com","time":"2023-08-15T18:42:08Z","message":"authorize check"}
grafana_core-grafana-1   | logger=context t=2023-08-15T18:42:08.466371111Z level=warn msg="Invalid JWT" error="Get \"https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json\": tls: failed to verify certificate: x509: certificate signed by unknown authority" traceID=
grafana_core-grafana-1   | logger=context userId=0 orgId=0 uname= t=2023-08-15T18:42:08.466503141Z level=info msg="Request Completed" method=GET path=/ status=401 remote_addr=172.23.0.1 time_ms=3 duration=3.42436ms size=39 referer=
grafana_core-pomerium-1  | {"level":"info","type":"type.googleapis.com/session.Session","query":"","offset":0,"limit":1,"filter":{"$or":[{"id":"8b05c98c-a488-4e54-88c4-fea3a1365fa7"},{"$index":"8b05c98c-a488-4e54-88c4-fea3a1365fa7"}]},"time":"2023-08-15T18:42:08Z","message":"query"}
grafana_core-pomerium-1  | {"level":"info","service":"authorize","request-id":"c16f27cb-3324-4ef4-840a-f1febbdfa140","check-request-id":"c16f27cb-3324-4ef4-840a-f1febbdfa140","method":"GET","path":"/favicon.ico","host":"grafana.localhost.pomerium.io","query":"","ip":"172.23.0.1","session-id":"8b05c98c-a488-4e54-88c4-fea3a1365fa7","allow":true,"allow-why-true":["domain-ok"],"deny":false,"deny-why-false":["valid-client-certificate-or-none-required"],"user":"google-oauth2|104583909997507305030","email":"user@pomerium.com","time":"2023-08-15T18:42:08Z","message":"authorize check"}
grafana_core-grafana-1   | logger=context t=2023-08-15T18:42:08.508146822Z level=warn msg="Invalid JWT" error="Get \"https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json\": tls: failed to verify certificate: x509: certificate signed by unknown authority" traceID=
grafana_core-grafana-1   | logger=context userId=0 orgId=0 uname= t=2023-08-15T18:42:08.50827224Z level=info msg="Request Completed" method=GET path=/favicon.ico status=401 remote_addr=172.23.0.1 time_ms=3 duration=3.026852ms size=39 referer=https://grafana.localhost.pomerium.io/ handler=public-assets

docker-compose.yaml

version: "3"
networks:
  main: {}
services:
  pomerium:
    image: pomerium/pomerium:latest
    volumes:
      - ./config.yaml:/pomerium/config.yaml:ro
      - /home/sveta/certificates/:/pomerium/certs/:ro
      - /home/sveta/.local/share/mkcert/rootCA.pem:/pomerium/rootCA.pem:ro
    ports:
      - 443:443
    networks:
      main:
        aliases:
          - authenticate.localhost.pomerium.io  
          - grafana.localhost.pomerium.io  

  grafana:
    image: grafana/grafana:latest
    user: "1000:1000"
    networks:
      main: {}
    ports:
      - 3000:3000
    environment:
      - GF_AUTH_SIGNOUT_REDIRECT_URL=https://grafana.localhost.pomerium.io/.pomerium/sign_out
      - GF_AUTH_JWT_ENABLED=true
      - GF_AUTH_JWT_HEADER_NAME=X-Pomerium-Jwt-Assertion
      - GF_AUTH_JWT_EMAIL_CLAIM=email
      - GF_AUTH_JWT_JWK_SET_URL=https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json
      - GF_AUTH_JWT_CACHE_TTL=60m
      - GF_AUTH_JWT_AUTO_SIGN_UP=true
    volumes:
      - ./grafana-storage:/var/lib/grafana/plugins:rw

config.yaml

  - from: https://grafana.localhost.pomerium.io
    to: http://grafana:3000
    host_rewrite_header: true
    pass_identity_headers: true
    policy:
      - allow:
         or:
          - domain:
                is: pomerium.com

[kenjenkins]

I think I understand:

• the guide does not have instructions for configuring a TLS certificate for Pomerium
• consequently Grafana cannot fetch the jwks.json file:

  tls: failed to verify certificate: x509: certificate signed by unknown authority

I see two options:

1. Add instructions for issuing a locally-trusted certificate (e.g. using mkcert), and figure out how to include the CA in the Grafana container. This may be more complicated than we want this guide to be.
2. See if it is feasible to run Pomerium without TLS for this guide (using the insecure_server setting), adding a clear warning that this is not suitable for a production deployment.

[kenjenkins]

Quick update: option (2) would require some code changes within Pomerium, as the OAuth2 callback URL is hard-coded to use "https" in a few places; one of them is here:

pomerium/authorize/check_response.go

Lines 210 to 212 in a83375d

	// always assume https scheme
	checkRequestURL := getCheckRequestURL(in)
	checkRequestURL.Scheme = "https"

---

I wonder if it's worth considering another option:

3. Set up some tunnel (ngrok?) using a public domain, so that Pomerium can provision its own certificates using the Autocert feature.

[pavankumar-go]

      - GF_AUTH_JWT_JWK_SET_URL=https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json

shouldn't this be the pomerium authenticate service endpoint ? Grafana does not expose that endpoint