Running the prometheus server on gokrazy

[stapelberg]

          Some things are really straight forward and worked like a breeze. Basically just `gok add` these:

• github.com/prometheus/alertmanager/cmd/alertmanager
• github.com/prometheus/node_exporter
• github.com/prometheus/blackbox_exporter

And add options to the config.json for WaitForClock, ExtraFilePaths, and CommandLineFlags for each of them to use a provided config file, e.g. --config.file=/etc/alertmanager/alertmanager.yml.

Unfortunately, prometheus server is the hard part, if you want the GUI to work. This requires the assets to be built first and I could not find a way to do this with the upstream module, only with a local copy.

git clone git@github.com:prometheus/prometheus.git
cd prometheus
make build  # this is more than necessary (assets, assets-compress) but the easiest for a PoC
rm -rf web/ui/node_modules/  # not sure why these make trouble on go update
gok add ./cmd/prometheus
cat <<EOF # add build flag to config.json to include (this command only shows what to add)
"github.com/prometheus/prometheus/cmd/prometheus": {
    "GoBuildFlags": [
        "-a",
         "-tags=netgo,builtinassets,stringlabels"
     ],
}
EOF
gok update

Some notes:

• -a might not be necessary, but I did not check. It is what is used by the promu build tool.
• options for mapping config files need to be added

So this works for now, but I would prefer to not need to clone and update the prometheus repository manually.

If you have an idea how I could reliably get the assets for the build so I can use gok add github.com/prometheus/prometheus/cmd/prometheus, please let me know!

Originally posted by @Syphdias in #51 (comment)

[stapelberg]

Hey @Syphdias 👋

I split this question into a separate issue because it’s not really related to the userguide.

Does prometheus not use go:embed for its assets? Could/should it?

[Syphdias]

I split this question into a separate issue because it’s not really related to the userguide.

Hopefully it will be, if I write it down after I understood it.

Does prometheus not use go:embed for its assets? Could/should it?

I’m quite the beginner on go, so this will be a learning opportunity.

I figured since you explicitly mentioned prometheus in your recent GPN talk, you already run it on gokrazy and just didn’t get to writing it down. Sounds like you didn’t.

It took me quite a while to figure out why I got an error when just adding upstream and why the precompiled binary through breakglass worked. When using make build in the prometheus repository it uses npm first to generate some assets (make assets stage) and then compressed them and echos the list of files to the embed.go which is probably what you are referring to (script is here). This is why I went through make build to get all the necessary files first.

I have no idea if it should use go:embed. Yes, it already does but can I trigger this from the go build command somehow?

PS: Something of note might be that the go module version is different from the Prometheus version. Not sure if that matters in any way.

[stapelberg]

I figured since you explicitly mentioned prometheus in your recent GPN talk, you already run it on gokrazy and just didn’t get to writing it down. Sounds like you didn’t.

I run the node exporter and blackbox exporter regularly. I ran the prometheus server (built using make) for a few days for an event network, but I don’t do that on a permanent basis.

Sorry that the nuance of “how to deploy” wasn’t included in my talk for the prometheus example. I mainly wanted to point out that it’s possible, i.e. that prometheus runs in a gokrazy environment once you get it installed. I’ll be more careful regarding phrasing in the next iteration.

When using make build in the prometheus repository it uses npm first to generate some assets (make assets stage) and then compressed them and echos the list of files to the embed.go which is probably what you are referring to (script is here). This is why I went through make build to get all the necessary files first.

I have no idea if it should use go:embed. Yes, it already does but can I trigger this from the go build command somehow?

I see. In Go, generation of any sort (typically with the go generate command) is supposed to be done by the developers, with the result committed to source control. Apparently prometheus doesn’t commit the web ui assets to source control, so that’s what would need changing to make prometheus buildable with the go tool.

I think this would need a discussion upstream in prometheus. Haven’t checked the issues if buildability with the go tool was already discussed, though, so that would be the next step.

[Syphdias]

Sorry that the nuance of “how to deploy” wasn’t included in my talk for the prometheus example. I mainly wanted to point out that it’s possible, i.e. that prometheus runs in a gokrazy environment once you get it installed. I’ll be more careful regarding phrasing in the next iteration.

I really enjoyed your talk. How to deploy in detail was not the focus of the talk of the talk after all. And to be fair. I did get prometheus to run on gokrazy in two different ways even. One was to use the precompiled binary, the other was what you did prior to your talk. I was wondering how it should be done though.

I found prometheus/prometheus#7517 which refers to the mailing list post from 2020 where someone explains that "the new React UI requires some separate build steps that are not Go-based, and the result of that is not checked in (because it's hard to make completely reproducible builds that wouldn't create diffs every time the React UI is built)."
It then continues to link to where the magic happens: a dead link and lines in the Makefile but on master instead of a commit.
I also found prometheus/prometheus#548 which suggests that in 2015 the needed assets were available.

I guess it is unlikely that they will change their build process for our convenience. If the could use go generate, they probably would have.

So I see two approaches to deal with prometheus (and possibly other go written tools that require extra stuff:

1. Use the compiled binary the project provides. It only needs to be copied over and executed. It would be nice to be able to provide a URL where to get it from but since often releases come as archives. So that would require some sort of selecting the binary, e.g. http://some/prometheus/release.tar:prometheus.
2. Have a PreBuildCommand or somthing like that to do arbitrary things like make build before running go build.

1 could be accomplished with a small go programs to wrap the logic for getting the binary either from disk (ExtraFilePaths) or remote URL (potentially extracting tar). If done like this I think only one program can run like this.
I could also imagine this as gokrazy feature itself something like a dummy module using PreBuildBinary (not a existing feature). This might be a direction you do not want the project to go in, though.

2 would need to be implemented on gokrazy level unless there is a go build argument to run something arbitrary before doing the actual work,e.g. fake command: go build -prebuild "cd prometheus-path && make build" -tags=netgo,builtinassets,stringlabels

Do you have any ideas how to work around the asset generation problem?

[BrunoTeixeira1996]

Hi guys, I think a cool feature to add here (when the UI for the prometheus is working) is to add the Grafana stack.

[Syphdias]

Hi guys, I think a cool feature to add here (when the UI for the prometheus is working) is to add the Grafana stack.

@BrunoTeixeira1996, to be clear. Prometheus is absolutely working. I just didn't find a "main stream" way to install it. Check out the first post in this thread. The best way would be to make prometheus (the upstream project) include the assets when building. I think if go install github.com/prometheus/cmd/prometheus would work and provide a "asset filled" binary, then it would work for gokrazy out of the box.

There is already a guide to Grafana you can find here: https://gokrazy.org/development/non-go/

[stapelberg]

Thanks for digging through the history.

I guess it is unlikely that they will change their build process for our convenience. If the could use go generate, they probably would have.

Well, it can’t hurt to ask again. The discussions you linked to happened in 2015 and 2020, so maybe enough time has passed that they might reconsider.

Regarding the problem of reproducibility: Whether the submitted assets will be used could be controlled by a build tag. There’s already the builtinassets tag, so maybe we could have one more. That said, I’m not entirely sure what aspect of reproducibility is problematic. The Go build will be reproducible, even if the NPM part isn’t. If that’s problematic, I don’t see how submitting files changes the equation, as users won’t be able to reproduce the full prometheus install either way.

So I see two approaches to deal with prometheus (and possibly other go written tools that require extra stuff:

The general approach I follow with such special cases is that the working copy is managed outside of gokrazy, to the point that when the gokrazy image is built, a go build is all that’s needed. (I use the replace directive to point gokrazy to my working copy.)

For example, to include plugins in the Caddy webserver, the recommended approach is to underscore import the plugins and run the Caddy logic. So, instead of installing caddy directly in gokrazy, I instead have the following main.go:

package main

import (
	caddycmd "github.com/caddyserver/caddy/v2/cmd"
	_ "github.com/caddyserver/caddy/v2/modules/standard"
	_ "github.com/kirsch33/realip"
	_ "github.com/caddyserver/transform-encoder"
)

func main() {
	caddycmd.Main()
}

In terms of updating this setup, I need to run go get github.com/caddyserver/caddy/v2 myself, either on-demand, or as part of my update script.

We should probably add this more explicitly to https://gokrazy.org/userguide/automation/, but that page already lists a few recipes for how to accomplish things outside the scope of gokrazy (for example: configuration templating).

So, in other words, I think custom installation, update or build steps, or otherwise massaging a piece of software into a consumable state for gokrazy, should be done before running the gok CLI. That clearly separates the responsibility and concerns, and keeps gokrazy small.

[Syphdias]

Well, it can’t hurt to ask again. The discussions you linked to happened in 2015 and 2020, so maybe enough time has passed that they might reconsider.

Done.

For example, to include plugins in the Caddy webserver, the recommended approach is to underscore import the plugins and run the Caddy logic. So, instead of installing caddy directly in gokrazy, I instead have the following main.go:

So, in other words, I think custom installation, update or build steps, or otherwise massaging a piece of software into a consumable state for gokrazy, should be done before running the gok CLI. That clearly separates the responsibility and concerns, and keeps gokrazy small.

You could use upstream prometheus and map any assets via ExtraFilePaths since prometheus does having them in the file system — shoot, I just realised you cannot map them to /perm/home/prometheus, but it could still be copied to it. As long as the react app stays compatible with the versions the assets do not need to change. Update path would then be to generate assets and copy them via breakglass.