Testing best practices

[andig]

I'm once more working on packaging https://evcc.io. One key challenge for me (Mac developer) ist user-testing the final application. Operating a Raspi, remote-updating etc. is too cumbersome for my taste.
Are there best practices how I could practically test without a target device, maybe by running a packaged (if testing-only) gokrazy-enabled application inside Docker? I'd only be testing application logic, persistence and functionality of the bundled modules (e.g. breakglass), but obviously not Linux kernels etc.

[damdo]

Hey @andig I'm using qemu to emulate various hardware devices (amd64, arm64, ...) on which I test the final gokrazy instance bundle.
You can find that setup here: https://github.com/damdo/gokrazy-on-qemu

[andig]

Thanks @damdo. As far as I understand your approach you're basically creating an entire (vagrant) machine running the packer-generated image? Im wondering if that's necessary though: shouldn't it suffice (at least for testing hardware-independent modules) to run the compiled binary (that ends up in the image) as root process inside Docker?

[damdo]

Yes I'm running it within Vagrant but that's because of how my dev setup is defined (as code).
You should be able to run qemu natively on macOS if you set it up correctly. Also if your purpose is testing you can use this qemu setup in a CI pipeline (Github Actions or other) with ease.

Not sure about running it as a Linux Container, haven't tried that yet.

[stapelberg]

One key challenge for me (Mac developer) ist user-testing the final application. Operating a Raspi, remote-updating etc. is too cumbersome for my taste.

Even when just replacing your program with gok run on a running gokrazy instance? That copies the program into RAM, which is much faster than writing to (typically) an SD card. Give it a try if you haven’t.

If there are any remaining performance bottlenecks, we can look into addressing them.

Are there best practices how I could practically test without a target device, maybe by running a packaged (if testing-only) gokrazy-enabled application inside Docker? I'd only be testing application logic, persistence and functionality of the bundled modules (e.g. breakglass), but obviously not Linux kernels etc.

As @damdo mentioned, using qemu is a pretty good way to run gokrazy, in particular because it not only runs the gokrazy kernel, but also provides a full block device. It looks like qemu is available for Mac, too, but I’m not sure how well it works.

I hadn’t really explored running gokrazy in Docker so far.

Here’s how to do it:

% GOARCH=amd64 gokr-packer -overwrite_root=/tmp/hello.squashfs
% mkdir /tmp/rootfs
% cd /tmp/rootfs
% unsquashfs /tmp/hello.squashfs
% tar -c . | docker import - gokrazy/gokrazy:exp

% docker run -t -i gokrazy/gokrazy:exp /gokrazy/init
gokrazy build timestamp 2022-11-09T18:50:26+01:00
2022/11/09 18:50:57 init.go:21: tmpfs on /tmp: operation not permitted

% docker run --privileged -t -i gokrazy/gokrazy:exp /gokrazy/init 
gokrazy build timestamp 2022-11-09T18:50:26+01:00
2022/11/09 18:52:41 gokrazy.go:76: found hardware watchdog "iTCO_wdt" with timeout 30s, pinging...
panic: rootdev.find: kernel command line "BOOT_IMAGE=/vmlinuz-linux root=/dev/mapper/cryptroot acpi_enforce_resources=lax enforcing=0 nvidia.NVreg_RegistryDwords=OverrideMaxPerf=0x2 nvidia-drm.modeset=1 rd.luks.options=discard ibt=off loglevel=3 quiet" did not match (?:root|ubd0)=(/dev/(?:mmcblk[01]p|sda|loop0p|nvme0n1p))([23])

goroutine 1 [running]:
github.com/gokrazy/internal/rootdev.findRaw()
	/home/michael/go/src/github.com/gokrazy/internal/rootdev/rootdev.go:150 +0x1ab
github.com/gokrazy/internal/rootdev.findDev()
	/home/michael/go/src/github.com/gokrazy/internal/rootdev/rootdev.go:158 +0x1d
github.com/gokrazy/internal/rootdev.Partition(0x81340a?)
	/home/michael/go/src/github.com/gokrazy/internal/rootdev/rootdev.go:219 +0x25
github.com/gokrazy/gokrazy.mountfs()
	/home/michael/go/src/github.com/gokrazy/gokrazy/mount.go:109 +0x573
github.com/gokrazy/gokrazy.Boot({0x82006a, 0x19})
	/home/michael/go/src/github.com/gokrazy/gokrazy/gokrazy.go:225 +0x8a
main.main()
	/tmp/gokr-packer4138136765/init.go:20 +0xad

As you can see, the gokrazy init system either lacks permission to mount /tmp, or, when running with --privileged, would not find the expected partition layout on the root block device (of the host machine due to --privileged!)

So it doesn’t work right now, but with enough changes it could probably be made work. Be careful to not accidentally overwrite your hard disk while playing around with this :)

[andig]

Even when just replacing your program with gok run on a running gokrazy instance? That copies the program into RAM, which is much faster than writing to (typically) an SD card. Give it a try if you haven’t.

Good advise, I wasn't aware of that. But still- I don't even want to have a Raspi running. Especially when I'm travelling.

I hadn’t really explored running gokrazy in Docker so far.

Maybe my approach is not feasible. I'm still wondering: at the end of the build we have a compiled application plus linux kernel. The compiled application consists of client app plus gokrazy environment. Shouldn't it be possible to reduce the gokrazy environment to the bare minimum necessary? The part without handling block devices, loading the kernel etc?

I'd be happy to contribute but would need pointers where to start.

[stapelberg]

Shouldn't it be possible to reduce the gokrazy environment to the bare minimum necessary? The part without handling block devices, loading the kernel etc?

Yes, but then the question is: what are you still getting from gokrazy? At the point when all you want to do is run your application, why not just build it using the Go tool and then run it on the host or in docker? Why add gokrazy into the mix here?

(I’m not saying I’m against making gokrazy work better in Docker. But I do wonder whether it’s valuable to make it work, or whether it just adds more moving pieces to a situation that would be better off with out it :)

[damdo]

Hey @andig,
I've updated https://github.com/damdo/gokrazy-on-qemu
You can now run qemu natively on both macOS and Linux directly without needing Vagrant.
I recorded a video where I show how I run it here: https://youtu.be/AZmTsyzf34Y

[andig]

@damdo thank you! I'm following along, but not successful yet:

GOOS=linux ./build.sh
gokrazy ensuring version: 'latest'
go: downloading github.com/gokrazy/gokrazy v0.0.0-20221113114523-dd415c9ee654
go: downloading golang.org/x/sys v0.0.0-20220817070843-5a390386f1f2
gokr-packer version '' is not the desired one 'latest'
fetching 'latest'..
go: cannot install cross-compiled binaries when GOBIN is set
go: downloading golang.org/x/crypto v0.0.0-20220214200702-86341886e292
go: downloading github.com/creack/pty v1.1.9
go: downloading github.com/prometheus/node_exporter v1.3.1-0.20220425111307-5ea0a936cbd0
go: downloading github.com/prometheus/common v0.33.0
go: downloading github.com/prometheus/client_golang v1.12.1
go: downloading github.com/prometheus/exporter-toolkit v0.7.1
go: downloading gopkg.in/alecthomas/kingpin.v2 v2.2.6
go: downloading github.com/ema/qdisc v0.0.0-20200603082823-62d0308e3e00
go: downloading github.com/godbus/dbus v0.0.0-20190402143921-271e53dc4968
go: downloading github.com/hashicorp/go-envparse v0.0.0-20200406174449-d9cfd743a15e
go: downloading github.com/hodgesds/perf-utils v0.5.1
go: downloading github.com/illumos/go-kstat v0.0.0-20210513183136-173c9b0a9973
go: downloading github.com/jsimonetti/rtnetlink v1.1.1
go: downloading github.com/lufia/iostat v1.2.1
go: downloading github.com/mattn/go-xmlrpc v0.0.3
go: downloading github.com/mdlayher/wifi v0.0.0-20220320220353-954ff73a19a5
go: downloading github.com/prometheus/procfs v0.7.4-0.20211011103944-1a7a2bd3279f
go: downloading github.com/safchain/ethtool v0.2.0
go: downloading github.com/soundcloud/go-runit v0.0.0-20150630195641-06ad41a06c4a
go: downloading golang.org/x/net v0.0.0-20220225172249-27dd8689420f
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d
go: downloading github.com/google/go-cmp v0.5.7
go: downloading golang.org/x/oauth2 v0.0.0-20220223155221-ee480838109b
go: downloading google.golang.org/appengine v1.6.6
go: downloading github.com/rtr7/kernel v0.0.0-20221117035324-989bbb15761e
cat: /Users/andig/.config/gokrazy/http-password.txt: No such file or directory
building grokrazy img(s) in 'single' mode
./build.sh: line 143: ./gokr-packer: No such file or directory

======
OUTPUT
======
'single' mode: drive.img has been generated.

At this point, the image has not been created. After setting the http password:

GOOS=linux ./build.sh
gokrazy ensuring version: 'latest'
gokr-packer version '' is not the desired one 'latest'
fetching 'latest'..
go: cannot install cross-compiled binaries when GOBIN is set
building grokrazy img(s) in 'single' mode
./build.sh: line 143: ./gokr-packer: No such file or directory

======
OUTPUT
======

'single' mode: drive.img has been generated.

Seems there is a problem with installing the packer that I couldn't figure out yet. Seems the problem is that the packer gets cross-compiled, too which it shouldn't?

Note: I'm running on M1 Mac (arm64). Re-running with GOARCH=arm64 doesn't help either.

Update

Seems I can make progress by doing

os="${GOOS_:=linux}"
arch="${GOARCH_:=amd64}"

and removing the forced amd64- continuing to investigate.

[damdo]

Oh yes that might happen! When I put together the scripts I didn't take into account The new M1 Macs (arm64) (I have an amd64 one), so we might need to update the build script to do so. It shouldn't be too hard though :) I'll take a look today or tomorrow 👍

[damdo]

@andig This PR should make things more flexible to support macOS arm64. Let me know if this works for you. Thanks

[andig]

This is really great. Solved by @damdo without Vagrant now. Close?

[stapelberg]

Cool stuff! Maybe we should add a userguide article to explain how to use this (similar to the video, but in textual form)? @damdo, would you be up for contributing such an article?

Thanks

[andig]

@damdo follow-up problem:

[    5.186064] IPv6: eth0: IPv6 duplicate address fd07:aded:3c98:e493:5054:ff:fe12:3456 used by 52:54:00:12:34:56 detected!

I'm unsure how to stop the qemu instance and (Ctrl-C doesn't work) and release the resources for another test.

[stapelberg]

I'm unsure how to stop the qemu instance and (Ctrl-C doesn't work)

Try Ctrl-a x

[damdo]

Cool stuff! Maybe we should add a userguide article to explain how to use this (similar to the video, but in textual form)? @damdo, would you be up for contributing such an article?

Sure I'd be up for it! I'm actually working on a Go rewrite of the scripts at https://github.com/damdo/gokrazy-on-qemu, removing the build logic in favour of a more unified approach with the new instance-centric design and focusing on an improved machine emulation cli experience for playing/testing with gokrazy (a "gokrazy-machine" tool that wraps qemu).

I'll keep you posted on my findings and then we can decide if we want to double down on the rewrite and document it on the website or keep the script approach and describe that on the docs.

[andig]

I have already removed the entire build part after I figured out that my few lines are doing the same for what I needed. Also takes care of the env var thing. The run part is definitely unique- if that could be part of a gok test command…

[damdo]

@andig

The run part is definitely unique- if that could be part of a gok test command…

Yes, that could be interesting. For now I'm trying to create a prototype as a separate cli tool, to prove its potential usefulness. But then we could evaluate :)

I'll let you know when I have a working prototype and you can let me know what you think.

[damdo]

Hey @andig so here is the prototype of the cli tool I was working on.

It's mainly a way to test gokrazy instances before deploying them on real hardware, but it can aid during development, CI, and more. I've added some basic features and some more experimental ones (OCI registries pull support).

Let me know what you think: https://github.com/damdo/gokrazy-machine