Docker: ARM images doesn't work since v6.4.x #19585

holgerpieta · 2019-10-02T19:13:23Z

What happened:
Grafana 6.4.X ARM in Docker does not work on Raspbian Buster.
Error message when trying to run the container:
/run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory

What you expected to happen:
Obvious: Grafana runs without error.

How to reproduce it (as minimally and precisely as possible):
docker run grafana/grafana

Anything else we need to know?:
Error is caused by a lib-c mismatch: Grafana is build with ld-linux-armhf.so, but Alpine base image contains only ld-musl-armv7.so.

Environment:

Grafana version: 6.4.X
Data source type & version: InfluxDB
OS Grafana is installed on: Raspbian Buster with Docker
User OS & Browser: Windows / Firefox
Grafana plugins: None
Others: None

hugohaggmark · 2019-10-03T07:33:30Z

Thanks for reporting this @theWaldschrat we'll investigate it further

marefr · 2019-10-03T08:05:15Z

@theWaldschrat what device are you using? Is that 32- or 64-bit architecture (armv6m armv7, armv8 etc)?

marefr · 2019-10-03T08:25:01Z

Maybe we need to include https://pkgs.alpinelinux.org/package/edge/main/armhf/libc6-compat in Grafana docker image?

apk add --no-cache --repository=http://dl-cdn.alpinelinux.org/alpine/edge/main libc6-compat

@theWaldschrat can you confirm that above solves the prioblem? Harder for us to verify without access to an actual ARM device.Docker based arm image maybe a possibility, but appreciate if you can help us out here. Thanks

holgerpieta · 2019-10-03T10:19:44Z

The device is a Raspberry Pi 4B. Technically it's an ARM64v8, but Raspbian by default runs a 32 bit kernel and userland, so ARM32v7.

uname -a
Host OS:
Linux raspberrypi 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l GNU/Linux
Grafana 6.3.6 image:
Linux 97f0bb9a456d 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l armv7l armv7l GNU/Linux
Grafana 6.4.X (latest) image:
Linux 84a01cb75816 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l Linux

I didn't build a lot of Docker images yet, so I'm not able to try above command in a new image, at least not quickly. But what I did do: Run a grafana:latest container with entry point bash and user root:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana
Here's the result:

Manually running /run.sh gives the same error message, so that's confirmed.
Run above command: No error.
Again run /run.sh: New error messages:

Error relocating /usr/share/grafana/bin/grafana-server: __memset_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __memcpy_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __vfprintf_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __fprintf_chk: symbol not found

Running ldd /usr/share/grafana/bin/grafana-server doesn't complain about missing libraries anymore, but gives the same results as above.

I'm not an expert, but I guess the lib-c are still not compatible.

marefr · 2019-10-03T10:38:42Z

@theWaldschrat thank you very helpful.

Just to verify some additional things can you specifically try these to verify that you get the same problem:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm32v7-linux:6.4.1

docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm32v7-linux:6.4.0-beta1

Just to be sure, can you also try to run and start grafana-server:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm64v8-linux:6.4.1

holgerpieta · 2019-10-03T11:17:23Z

First two all do the same as previously described.
Running /run.sh or directly /usr/share/grafana/bin/grafana-server doesn't make a difference.

The third one doesn't even start with an arch mismatch:
standard_init_linux.go:211: exec user process caused "exec format error"

ata4 · 2019-10-03T11:24:18Z

I have the same issue and had to downgrade to version 6.3.6, so it seems that all Alpine-based 6.4.x images are broken for ARMv7.

marefr · 2019-10-03T12:01:23Z

Thanks. After you entered bash can you try and install the musl-dev packade using apk add?

ata4 · 2019-10-03T12:32:45Z

musl-dev installs fine for me, but doesn't have an impact on the problem, with or without libc6-compat.

ata4 · 2019-10-03T12:53:16Z

Installing the glibc apks from https://github.com/armhf-docker-library/alpine-pkg-glibc/releases allows grafana-server to start. If I understand the issue correctly, it's better to statically link the binaries with musl, though.

holgerpieta · 2019-10-03T14:52:13Z

It's the idea of Alpine to statically link to musl instead of dynamic glibc linking. It's faster, smaller, more stable and potentially more secure. At least that's what they say.
But as far as I can see, Grafana is build outside the target image linking against glibc, so it's probably the best idea to either install glibc as above or use a different base image that already includes glibc.

protik77 · 2019-10-05T18:12:57Z

Considering this change effectively broke the docker image for ARM devices, I was expecting something better than a tag of "needs investigation".

Sigh! The curse of "agile development", I guess.

aknuds1 · 2019-10-09T14:10:39Z

I can reproduce the error in OS X, even though it looks a bit different from yours:

$ docker run --platform arm grafana/grafana
/lib/ld-linux-armhf.so.3: No such file or directory

I will see if I can fix it.

aknuds1 · 2019-10-09T19:22:10Z

I might have a lead on the root cause for this issue, hoping I can fix it by tomorrow.

aknuds1 · 2019-10-10T15:16:09Z

Working to solve this by building musl binaries in addition to glibc ones.

JasonSwindle · 2019-10-22T00:03:53Z

Just ran into this as well. My system is aarch64 (RockPro64) and I'm getting the same error:

/run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory

iwittkau · 2019-10-22T15:42:52Z

$ uname -a
Linux black-pearl 4.14.70-hypriotos-v7+ #1 SMP Sat Sep 22 05:54:18 UTC 2018 armv7l GNU/Linux

LGTM running on a Raspberry 3B

Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors.
If possible, try to add/connect to a datasource and render a dashboard/panel
Optional: Verify that you can download/install plugins

JochenLutz · 2019-10-22T16:12:56Z

SBC: Cubietruck (aka CubieBoard 3)

$ uname -a
Linux fernia 4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l armv7l armv7l GNU/Linux

LGTM

Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors.
If possible, try to add/connect to a datasource and render a dashboard/panel
Optional: Verify that you can download/install plugins

marefr · 2019-10-22T17:05:55Z

Thanks a lot for the quick responses and help. Very appreciated.

aknuds1 · 2019-10-22T17:26:24Z

Agreed, thanks for helping test!

…

On Tue, Oct 22, 2019, 19:05 Marcus Efraimsson ***@***.***> wrote: Thanks a lot for the quick responses and help. Very appreciated. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#19585?email_source=notifications&email_token=AACEVV4YMCESH5G7XWTY3QLQP4XHPA5CNFSM4I42J4CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6PL5A#issuecomment-545060340>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACEVV3OBIAWAV3ZNAP4XEDQP4XHPANCNFSM4I42J4CA> .

gcgarner · 2019-10-23T11:48:40Z

uname -a
Linux raspberrypi4 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

raspberry pi 4b

docker -v
Docker version 19.03.4, build 9013bf5

Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb with graph panel
Optional: Verify that you can download/install plugins

pgolm · 2019-10-23T21:03:52Z

LGTM: raspberry pi 4

$ uname -a
Linux worker-3 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb, prometheus, loki)
Verify that you can download/install plugins (raintank-worldping-app, grafana-kubernetes-app, devopsprodigy-kubegraf-app, grafana-piechart-panel)

…afana/grafana#19585

krystian-wojtas · 2019-10-26T16:31:33Z

LGTM

rockchip rock64

$ uname -a
Linux rock64 4.4.132-1072-rockchip-ayufan-ga1d27dba5a2e #1 SMP Sat Jul 21 20:18:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb, prometheus, loki)
Verify that you can download/install plugins (raintank-worldping-app, grafana-kubernetes-app, devopsprodigy-kubegraf-app, grafana-piechart-panel)

SySfRaMe · 2019-10-27T17:19:01Z

Docker skill level
Noob
Grafana skill level
Noob
-uname -a
Linux SwingerPictureServer 4.19.75-v7+ #1270 SMP ........... armv71 GNU/Linux
HW
Raspberry pi 3B
compose.sh file:
docker run
--name Grafana_test
-p 3001:3001
-e "GF_SERVER_ROOT_URL=http://:3001"
-e "GF_SECURITY_ADMIN_PASSOWRD="
--mount type=bind,source="/home/pi/DockerConf/Grafana/test/config",target="/etc/grafana":ro
grafana/grafana-arm32v7-linux:dev-musl

LOG file:
warn: msg="phantomJS is deprecated and will be removed in the future release...

marefr · 2019-10-28T14:32:02Z

Thanks all. We've merged this to master, but decided to include this fix in Grafana v6.5.0, to be released in a couple of weeks. Until then you may use nightly builds if you want to run Grafana v6.5-pre ARM-compatible Grafana docker images by using the grafana/grafana:master tag.

herm · 2019-11-10T11:14:08Z

Please add a note to docker hub so it is easier to find this issue. If you are currently pulling grafana/grafana you still get a non-functional image on armhf.

mhansen · 2020-01-11T03:08:12Z

Just commenting to note that grafana/grafana-arm32v7-linux:latest now works fine for me (this image), so I've unpinned versions 👍

pedroetb · 2020-01-11T12:55:22Z

@mhansen you can use directly the base image (grafana/grafana:latest), it's multiarch :)

I'm currently using grafana/grafana:6.5.1@sha256:befcd84da2c1f3310b23d93ba9eec4a80df4c86c04bd39455623ac632fbcefdd in an ARM cluster.

aknuds1 · 2020-02-13T15:00:41Z

@theWaldschrat @pedroetb @mhansen @herm @SySfRaMe @krystian-wojtas @pgolm @gcgarner @JochenLutz @iwittkau @JasonSwindle @protik77 @ata4 we could use some help testing new builds (Docker images and tar archives) on various ARM architectures, might you lend a hand? We would appreciate it!

The Docker images in question

grafana/grafana-arm64v8-linux:master-df1d43167af035c6819923ecce135056f37c79c2-new-pipeline
grafana/grafana-arm64v8-linux:master-df1d43167af035c6819923ecce135056f37c79c2-ubuntu-new-pipeline
grafana/grafana-arm32v7-linux:master-df1d43167af035c6819923ecce135056f37c79c2-new-pipeline
grafana/grafana-arm32v7-linux:master-df1d43167af035c6819923ecce135056f37c79c2-ubuntu-new-pipeline

The tar archives in question

MUSL archives are for Alpine Linux, GLIBC ones are for regular Linux distros:

RPMs

ARM64
ARMv7

volschin · 2020-02-14T09:02:33Z

Docker image
grafana/grafana-arm64v8-linux:master-df1d43167af035c6819923ecce135056f37c79c2-new-pipeline works fine on Raspberry Pi 4B with Kernel 4.19.97-v8+ and Docker CE 19.03.5.

aknuds1 · 2020-02-14T09:05:06Z

Thanks @volschin!

volschin · 2020-02-15T09:41:55Z

Had a problem with the container today after running about 24h (no templating init). This is nothing that happend in the last months. So perhaps there is a stability problem.

aknuds1 · 2020-02-15T19:02:31Z

Had a problem with the container today after running about 24h (no templating init). This is nothing that happend in the last months. So perhaps there is a stability problem.

What kind of problem did you see exactly @volschin?

iwittkau · 2020-03-02T10:32:02Z

@aknuds1 sorry, I did not get around to fully testing the new docker arm images yet. Is there a way to automate testing?

aknuds1 · 2020-03-03T07:43:46Z

I don't have any automated method, sorry @iwittkau.

mhansen · 2020-03-12T10:46:11Z

I'm no longer seeing grafana/grafana:latest as being multiarch, only amd64/linux.

$ docker run --rm mplatform/mquery grafana/grafana
Image: grafana/grafana
 * Manifest List: No
 * Supports: amd64/linux

I've switched to grafana/grafana:master

aknuds1 · 2020-03-12T11:03:35Z

I'm no longer seeing grafana/grafana:latest as being multiarch, only amd64/linux.
$ docker run --rm mplatform/mquery grafana/grafana
Image: grafana/grafana
 * Manifest List: No
 * Supports: amd64/linux
I've switched to grafana/grafana:master

@mhansen Interesting, thanks for the heads up. I will have to check on this.

protik77 · 2020-03-22T03:30:55Z

For what it's worth, I am using grafana/grafana-arm32v7-linux:latest for now. Though it installed 6.7.1.

hugohaggmark added the needs investigation for unconfirmed bugs. use type/bug for confirmed bugs, even if they "need" more investigating label Oct 3, 2019

adriankumpf mentioned this issue Oct 6, 2019

Grafana not loading in v1.10.0 teslamate-org/teslamate#193

Closed

marefr added area/docker type/bug labels Oct 7, 2019

aknuds1 added this to To do in Backend Platform Squad Oct 7, 2019

aknuds1 self-assigned this Oct 7, 2019

xlson mentioned this issue Oct 8, 2019

Grafana 6.4.1 armhf Docker Image does not start #19709

Closed

aknuds1 moved this from To do to In progress in Backend Platform Squad Oct 9, 2019

aknuds1 mentioned this issue Oct 14, 2019

Docker: Build and use musl-based binaries in alpine images to resolve glibc incompatibility issues #19798

Merged

aknuds1 moved this from In progress to Under review in Backend Platform Squad Oct 16, 2019

marefr mentioned this issue Oct 21, 2019

arm64 docker images since 6.4.0 non-functional #19931

Closed

aheendwhz added a commit to aheendwhz/smart-cottage that referenced this issue Oct 21, 2019

changed grafana to 6.3.6 due to github.com/grafana/grafana/issues/19585

cdf75e2

marefr added this to the 6.4.4 milestone Oct 22, 2019

marefr removed the needs investigation for unconfirmed bugs. use type/bug for confirmed bugs, even if they "need" more investigating label Oct 22, 2019

aknuds1 closed this as completed in #19798 Oct 24, 2019

Backend Platform Squad automation moved this from Under review to Done Oct 24, 2019

jmb12686 added a commit to jmb12686/raspi-docker-stacks that referenced this issue Oct 25, 2019

use grafana version 6.3.6, regression found in 6.4.x. see issue at gr…

69b73df

…afana/grafana#19585

marefr modified the milestones: 6.4.4, 6.5 Oct 28, 2019

nemchik mentioned this issue Nov 11, 2019

Add ARM images for grafana GhostWriters/DockSTARTer#828

Merged

4 tasks

marefr mentioned this issue Nov 13, 2019

Grafana docker container restart loop #20356

Closed

lbrito1 mentioned this issue Jun 21, 2020

Package request: grafana termux/termux-packages#4801

Closed

Docker: ARM images doesn't work since v6.4.x #19585

Docker: ARM images doesn't work since v6.4.x #19585

Comments

holgerpieta commented Oct 2, 2019

hugohaggmark commented Oct 3, 2019

marefr commented Oct 3, 2019 • edited

marefr commented Oct 3, 2019

holgerpieta commented Oct 3, 2019

marefr commented Oct 3, 2019 • edited

holgerpieta commented Oct 3, 2019

ata4 commented Oct 3, 2019 • edited

marefr commented Oct 3, 2019

ata4 commented Oct 3, 2019

ata4 commented Oct 3, 2019

holgerpieta commented Oct 3, 2019

protik77 commented Oct 5, 2019

aknuds1 commented Oct 9, 2019

aknuds1 commented Oct 9, 2019

aknuds1 commented Oct 10, 2019

JasonSwindle commented Oct 22, 2019

iwittkau commented Oct 22, 2019 • edited

JochenLutz commented Oct 22, 2019

marefr commented Oct 22, 2019

aknuds1 commented Oct 22, 2019 via email

gcgarner commented Oct 23, 2019 • edited

pgolm commented Oct 23, 2019

krystian-wojtas commented Oct 26, 2019 • edited

SySfRaMe commented Oct 27, 2019

marefr commented Oct 28, 2019

herm commented Nov 10, 2019

mhansen commented Jan 11, 2020

pedroetb commented Jan 11, 2020

aknuds1 commented Feb 13, 2020

The Docker images in question

The tar archives in question

RPMs

volschin commented Feb 14, 2020

aknuds1 commented Feb 14, 2020

volschin commented Feb 15, 2020

aknuds1 commented Feb 15, 2020 • edited

iwittkau commented Mar 2, 2020

aknuds1 commented Mar 3, 2020

mhansen commented Mar 12, 2020 • edited

aknuds1 commented Mar 12, 2020

protik77 commented Mar 22, 2020

marefr commented Oct 3, 2019 •

edited

marefr commented Oct 3, 2019 •

edited

ata4 commented Oct 3, 2019 •

edited

iwittkau commented Oct 22, 2019 •

edited

gcgarner commented Oct 23, 2019 •

edited

krystian-wojtas commented Oct 26, 2019 •

edited

aknuds1 commented Feb 15, 2020 •

edited

mhansen commented Mar 12, 2020 •

edited