setup grafana for binder.pangeo.io

[jhamman]

mybinder.org has https://grafana.mybinder.org. It would be good to combine this config with pangeo-data/pangeo#359 once that is finalized.

@jacobtomlinson - if you can help guide me in the right direction, I'm ready to give this a try.

[jhamman]

side note, as of 8/30, we have google analytics tracking on binder.pangeo.io setup so at least we have some basic diagnostics.

[jacobtomlinson]

You need to install the prometheus and grafana helm charts.

There is some example config for pangeo.pydata.org in with PR you mentioned.

[jhamman]

@jacobtomlinson - that was pretty straightforward. I deployed this using the configs here: 95242a8. This is live here (for now): http://35.238.151.129/

However, the pangeo dashboard isn't playing nice. Any thoughts on what's going wrong here?

[jacobtomlinson]

It is probably a namespace thing. The dashboard assumes the kubernetes namespace your Pangeo is in is called pangeo. We should fix this.

[jhamman]

Yes, my namespace is pangeo-binder. I'll take a look.

…

[jacobtomlinson]

If you edit each graph you should see where the namespace is specified in the query. Updating this should fix it, but its rather labor intensive sorry!

[jhamman]

Thanks @jacobtomlinson! This is mostly working now. We'll want to make some minor changes but this seems to be working.

[rabernat]

Possibly off topic question: can we change the skin of the grafana dashboard? I can't stand the black background.

[jhamman]

I will leave this open until I have a chance to clean up the dashboard and move the ip address to grafana.binder.pangeo.io.

@rabernat - I don't know for sure but I would image this is possible.

[jhamman]

cross referencing to jupyterhub/mybinder.org-deploy#726. It seems like we'll need to scrape the mybinder.org configs from their grafana site.

[salvis2]

I'm looking to enable Prometheus / Grafana for the AWS BinderHub, so it'll happen for the GCP BinderHub as well.

[salvis2]

@TomAugspurger looks like the CI deployment failed: https://app.circleci.com/pipelines/github/pangeo-data/pangeo-binder/152/workflows/7e61c31d-0a15-45fa-a9e7-4ca381b2a7b9/jobs/159

I think that error is from prometheus-operator, I've seen GitHub issues talking about it for the CustomResourceDefinitions: helm/charts#23413. They list a fix in the readme, but that's only supposed to apply for helm > v2.14. Might need the six lines of kubectl apply -f ... in .circleci/config.yaml.

[TomAugspurger]

Thanks, looking now.

[TomAugspurger]

I'm pretty stumped right now. I wonder if kube-lego is causing issues, if it hasn't been updated to not use extensions/v1beta1 since it's deprecated?

[salvis2]

Did we need kube-lego? I thought ingress-nginx was replacing it a bit.

For the rest, 4 of the 5 it couldn't find in the GCP deployment are from prometheus-operator. The 4 in the AWS deployment that it couldn't find are the same as the GCP ones. I can add the CRD install lines to the CI Action.