fix(docker): run in docker, and build scripts for docker #1291

egidijus · 2020-12-09T09:07:08Z

no reason to grant excessive permissions to your docker container, this runs headless, no errors from chromium or puppetteer.

Description

This also prints the config for puppetteer, makes it easier to troubleshoot:

streetmerchant-shop-bot    | [8:08:40 PM] warn :: ℹ printing puppeteer configs:: {
streetmerchant-shop-bot    |   "0": "--disable-dev-shm-usage",
streetmerchant-shop-bot    |   "1": "--no-sandbox",
streetmerchant-shop-bot    |   "2": "--disable-setuid-sandbox",
streetmerchant-shop-bot    |   "3": "--headless",
streetmerchant-shop-bot    |   "4": "--disable-gpu"
streetmerchant-shop-bot    | }

no more this error when running in docker

streetmerchant-shop-bot    | }
streetmerchant-shop-bot    | [8:05:39 PM] error :: ✖ something bad happened, resetting streetmerchant in 5 seconds Failed to launch the browser process!
streetmerchant-shop-bot    | Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted
streetmerchant-shop-bot    | [1208/200539.568829:FATAL:zygote_host_impl_linux.cc(191)] Check failed: ReceiveFixedMessage(fds[0], kZygoteBootMessage, sizeof(kZygoteBootMessage), &boot_pid). 
streetmerchant-shop-bot    | Received signal 6
streetmerchant-shop-bot    |   r8: 00007ddd4844  r9: 00007f3d3f6f40 r10: 0000000000000008 r11: 0000000000000246
streetmerchant-shop-bot    |  r12: 00007ffd80 r13: 00007ffddbc r14: 00007f30c4ad2170 r15: 00000000000000a0
streetmerchant-shop-bot    |   di: 0000000000000002  si: 00007dd0  bp: 00007ddd4710  bx: 0000000000000000
streetmerchant-shop-bot    |   dx: 0000000000000000  ax: 0000000000000000  cx: 00007f30cd8a71  sp: 00007ffc55d708
streetmerchant-shop-bot    |   ip: 00007d8a71 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
streetmerchant-shop-bot    |  trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
streetmerchant-shop-bot    | [end of stack trace]
streetmerchant-shop-bot    | Calling _exit(1). Core file will not be generated.
streetmerchant-shop-bot    | 
streetmerchant-shop-bot    | 
streetmerchant-shop-bot    | TROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md
streetmerchant-shop-bot    |  {
streetmerchant-shop-bot    |   "stack": "Error: Failed to launch the browser process!\nFailed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted\n[1208/200539d8829:FATAL:zygote_host_impl_linux.cc(191)] Check failed: ReceiveFixedMessage(fds[0], kZygoteBootMessage, sizeof(kZygoteBootMessage), &boot_pid). \nReceived signal 6\n  r8: 00007d52d4844  r9: 00007d63f6f40 r10: 0000000000000008 r11: 0000000000000246\n r12: 0000d80 r13: 00007d4dbc r14: 0000d170 r15: 00000000000000a0\n  di: 0000000000000002  si: 0000d4710  bp: 00007d4710  bx: 0000000000000000\n  dx: 0000000000000000  ax: 0000000000000000  cx: 00007d71  sp: 0000d8\n  ip: 0000d8a71 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000\n trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000\n[end of stack trace]\nCalling _exit(1). Core file will not be generated.\n\n\nTROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md\n\n    at onClose (/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:193:20)\n    at ChildProcess.<anonymous> (/app/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserRunner.js:184:79)\n    at ChildProcess.emit (events.js:327:22)\n    at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)"
streetmerchant-shop-bot    | }

Testing

Running make will do everything for you

New dependencies

Make
docker-compose

…is runs headless, no errors from chromium or puppetteer

jef · 2020-12-09T18:12:41Z

Love this! Does docker-compose come with installs on Windows? I know for macOS and Linux you need to install separately, so it would need to be a prerequisite.

joshstubble · 2020-12-09T18:20:45Z

Docker Compose does come installed in the docker desktop app on windows

Dockerfile

docker-compose.yml

jef · 2020-12-09T19:52:21Z

On second thought, I'm a little hesitant on the docker-compose. Not that it's bad to have or even gives a quality of life, but I think the Makefile alone is sufficient. And that may even be something I don't necessarily want to maintain either, along with docker-compose.

I'd also like to add that it's somewhat verbose given we can use env-file with docker-run. The main benefit of using docker-compose is utilizing multiple containers and/or advanced volumes and networks -- neither of which streetmerchant has. That's my knee jerk reaction.

That being said, I really like the changes and it seems to be working great on my end. I appreciate your contribution and would like to hear your thoughts!

Thanks.

…oving

…stening, setting it as disabled default in docker-compose, and adding docs for how to build docker

egidijus · 2020-12-10T08:45:40Z

On second thought, I'm a little hesitant on the docker-compose. Not that it's bad to have or even gives a quality of life, but I think the Makefile alone is sufficient. And that may even be something I don't necessarily want to maintain either, along with docker-compose.

I'd also like to add that it's somewhat verbose given we can use env-file with docker-run. The main benefit of using docker-compose is utilizing multiple containers and/or advanced volumes and networks -- neither of which streetmerchant has. That's my knee jerk reaction.

That being said, I really like the changes and it seems to be working great on my end. I appreciate your contribution and would like to hear your thoughts!

Thanks.

🤷‍♀️
In my experience if you are working locally, even if it's a single container docker-compose build/up > plain docker run .

I could change the docker-compose.yml to v 2.0, you could use the docker-compose.yml to spin things up on portainer, some people run that on their NAS devices.

Obviously I like docker-compose and Makefiles, because it makes it super easy to get started without even reading any docs 😁

I see makefiles and docker-compose as build tools, once you get them going, they kinda work without much attention (caveat: if you have complex systems, your build tools will get complex, this is not complex).

jef

Okay, I see where you're coming from. If you don't mind though, I'll make a couple suggestions and you let me know what you think.

I really appreciate your contribution and your time for the input and feedback!

Makefile

jef · 2020-12-11T05:53:54Z

docker-compose.yml

@@ -0,0 +1,14 @@
+version: '3.0'


Suggested change

version: '3.0'

version: '3.8'

Let's pin to the latest version.

I haven't tested if this docker-compose is compatible with v3.8 compose file.
these versions are for the docker-compose.yml file, across the versions some add more features at the cost of mandatory complexity.

We can stick with v3.0 of docker-compose file until the end of days (caveat: if we do not need any functionality from greater versions, and do not mind slight change of syntax).

https://docs.docker.com/compose/compose-file/

Okay! I can get with that. Since we're pinning to 3.0, that makes sense to me.

docker-compose.yml

Co-authored-by: Jef LeCompte <jeffreylec@gmail.com>

jef

Amazing work!!! I really appreciate your feedback and the time you took to implement this.

Updates unnecessary need to grant special privileges when running in Docker.

rchenzheng · 2020-12-12T14:47:09Z

Thanks @jef @egidijus

TDCroPower · 2020-12-19T04:37:49Z

the Docker image was unfortunately only built for amd64 and therefore does not work on my Helios64 with rockchip64 (arm64/v8) architecture.
Do you have a tip how I can rewrite the Dockerfile so that I can build it on the Helios64 as well?

here the output from Helios64...

root@helios64:/opt/docker/streetmerchant# docker run -it --rm \
>   --env-file ./dotenv \
>   ghcr.io/jef/streetmerchant:nightly
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: exec format error
root@helios64:/opt/docker/streetmerchant# uname -a
Linux helios64 5.9.14-rockchip64 #20.11.4 SMP PREEMPT Tue Dec 15 08:52:20 CET 2020 aarch64 GNU/Linux
root@helios64:/opt/docker/streetmerchant#

edit:
I have customized your Dockerfile with another node image and copied the files manually into the build directory.

My customized Dockerfile with "node:14.15.3-buster" instead of "node:14.15.0-alpine3.12".
is the change necessary?

FROM node:14.15.3-buster AS builder

LABEL org.opencontainers.image.source="https://github.com/jef/streetmerchant"

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true

WORKDIR /build

COPY package.json package.json
COPY package-lock.json package-lock.json
COPY tsconfig.json tsconfig.json
RUN npm ci

COPY src/ src/
RUN npm run build
RUN npm prune --production

FROM node:14.15.3-buster

RUN apk add --no-cache chromium

ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser \
	DOCKER=true

RUN addgroup -S appuser && adduser -S -g appuser appuser \
	&& mkdir -p /home/appuser/Downloads /app \
	&& chown -R appuser:appuser /home/appuser \
	&& chown -R appuser:appuser /app

USER appuser

WORKDIR /app

COPY --from=builder /build/node_modules/ node_modules/
COPY --from=builder /build/build/ build/
COPY web/ web/
COPY package.json package.json
COPY version.txt version.txt

ENTRYPOINT ["npm", "run"]
CMD ["start:production"]

however there seems to be a problem installing chromium or?

root@helios64:/opt/docker/streetmerchant# docker build -t streetmerchant .
Sending build context to Docker daemon  1.111MB
Step 1/24 : FROM node:14.15.3-buster AS builder
 ---> 0fff5459f31f
Step 2/24 : LABEL org.opencontainers.image.source="https://github.com/jef/streetmerchant"
 ---> Using cache
 ---> 0af35811e896
Step 3/24 : ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
 ---> Using cache
 ---> eab7baf56d05
Step 4/24 : WORKDIR /build
 ---> Using cache
 ---> 263f477ff905
Step 5/24 : COPY package.json package.json
 ---> 948ec20ecd9a
Step 6/24 : COPY package-lock.json package-lock.json
 ---> afe96990d670
Step 7/24 : COPY tsconfig.json tsconfig.json
 ---> d423520b99e7
Step 8/24 : RUN npm ci
 ---> Running in de41b35e2907

> twitch@4.3.6 postinstall /build/node_modules/twitch
> node ./scripts/npm/postinstall.js


> top-user-agents@1.0.19 postinstall /build/node_modules/top-user-agents
> node scripts/postinstall

TypeError: Cannot read property 'host' of undefined
    at got.extend.hooks.beforeRequest (/build/node_modules/hooman/hooman.js:38:48)
    at Immediate.<anonymous> (/build/node_modules/got/source/request-as-event-emitter.js:300:11)

> puppeteer@5.5.0 install /build/node_modules/puppeteer
> node install.js

FROM node:14.15.3-buster AS builder

LABEL org.opencontainers.image.source="https://github.com/jef/streetmerchant"

ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true

WORKDIR /build

COPY package.json package.json
COPY package-lock.json package-lock.json
COPY tsconfig.json tsconfig.json
RUN npm ci

COPY src/ src/
RUN npm run build
RUN npm prune --production

FROM node:14.15.3-buster

RUN apk add --no-cache chromium

ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium-browser \
        DOCKER=true

RUN addgroup -S appuser && adduser -S -g appuser appuser \
        && mkdir -p /home/appuser/Downloads /app \
        && chown -R appuser:appuser /home/appuser \
        && chown -R appuser:appuser /app

USER appuser

WORKDIR /app

COPY --from=builder /build/node_modules/ node_modules/
COPY --from=builder /build/build/ build/
COPY web/ web/
COPY package.json package.json
COPY version.txt version.txt

ENTRYPOINT ["npm", "run"]
CMD ["start:production"]
~
~
"Dockerfile" 41L, 918C                                                                                                                      1,24          All
**INFO** Skipping browser download. "PUPPETEER_SKIP_CHROMIUM_DOWNLOAD" environment variable was found.

> nodemon@2.0.6 postinstall /build/node_modules/nodemon
> node bin/postinstall || exit 0

Love nodemon? You can now support the project via the open collective:
 > https://opencollective.com/nodemon/donate

added 1236 packages in 38.771s
Removing intermediate container de41b35e2907
 ---> f012d754ab62
Step 9/24 : COPY src/ src/
 ---> 0032adf94bdc
Step 10/24 : RUN npm run build
 ---> Running in e57c956e2f37

> streetmerchant@ build /build
> tsc

Removing intermediate container e57c956e2f37
 ---> 1a183b3784a8
Step 11/24 : RUN npm prune --production
 ---> Running in 70889104db9a
npm WARN hooman@1.2.6 requires a peer of got@>=11.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN twilio@3.54.1 requires a peer of @types/express@^4.17.7 but none is installed. You must install peer dependencies yourself.
npm WARN twilio@3.54.1 requires a peer of @types/qs@6.9.4 but none is installed. You must install peer dependencies yourself.

removed 672 packages, updated 251 packages and audited 818 packages in 31.192s

63 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

Removing intermediate container 70889104db9a
 ---> abed1acbf70c
Step 12/24 : FROM node:14.15.3-buster
 ---> 0fff5459f31f
Step 13/24 : RUN apk add --no-cache chromium
 ---> Running in 9f776797238b
/bin/sh: 1: apk: not found
The command '/bin/sh -c apk add --no-cache chromium' returned a non-zero code: 127
root@helios64:/opt/docker/streetmerchant#

edit3:
ok it works with Docker + Portainer (for the docker-compose file) on Helios64.
I have build the image with your Dockerfile and set this into the Stacks...

version: "2"
services:
  streetmerchant:
    image: streetmerchant:latest
    container_name: streetmerchant
    volumes:
      - /opt/docker/streetmerchant/dotenv:/app/dotenv
    network_mode: host
    restart: unless-stopped

and here the log of the running container...

> streetmerchant@ start:production /app
> node build/index.js

ＳＴＲＥＥＴＭＥＲＣＨＡＮＴ
3.3.0

[5:34:25 AM] info :: ℹ selected stores: alternate, amazon-de, amazon-de-warehouse, amazon-es, amazon-fr, amazon-it, cyberport, euronics-de, expert, otto, proshop-de, saturn, spielegrotte
[5:34:25 AM] info :: ℹ selected series: sonyps5c, sonyps5de
[5:34:29 AM] info :: ℹ puppeteer config:  {
  "0": "--disable-dev-shm-usage",
  "1": "--no-sandbox",
  "2": "--disable-setuid-sandbox",
  "3": "--headless",
  "4": "--disable-gpu"
}

[5:34:36 AM] warn :: ✖ [amazon-de-warehouse] [sony (sonyps5c)] ps5 console :: STATUS CODE ERROR 503
[5:34:37 AM] info :: ✖ [spielegrotte] [sony (sonyps5c)] ps5 console :: OUT OF STOCK
[5:34:39 AM] info :: ✖ [amazon-es] [sony (sonyps5c)] ps5 console :: OUT OF STOCK
[5:34:41 AM] warn :: ✖ [amazon-it] [sony (sonyps5c)] ps5 console :: CAPTCHA
[5:34:42 AM] info :: ✖ [spielegrotte] [sony (sonyps5de)] ps5 digital :: OUT OF STOCK
[5:34:43 AM] info :: ✖ [amazon-fr] [sony (sonyps5c)] ps5 console :: OUT OF STOCK
[5:34:44 AM] warn :: ✖ [amazon-de] [sony (sonyps5c)] ps5 console :: CAPTCHA
[5:34:44 AM] info :: ✖ [amazon-es] [sony (sonyps5de)] ps5 digital :: OUT OF STOCK
[5:34:46 AM] info :: ✖ [amazon-fr] [sony (sonyps5de)] ps5 digital :: OUT OF STOCK

kiowadriver · 2020-12-21T16:16:53Z

I am able to run fine without the SYS_ADMIN parameter (less of a security risk).

I have included some build instructions in my fork which i am keeping up to date with the main branch with some minor tweaks for unraid usage:
https://github.com/kiowadriver/streetmerchant/blob/main/README.md

rchenzheng · 2020-12-21T18:19:51Z

I am able to run fine without the SYS_ADMIN parameter (less of a security risk).

I have included some build instructions in my fork which i am keeping up to date with the main branch with some minor tweaks for unraid usage:
https://github.com/kiowadriver/streetmerchant/blob/main/README.md

It has already been merged

/close

TDCroPower · 2020-12-21T21:12:35Z

i first had the problem on my helios with aarch64 and was able to work around it.
Now I wanted to switch to the OrangePi to run the script on the side, but there is again the problem with the architecture because this uses armv7l and there is no chromium in the alpine package manager...

aarch64...
https://pkgs.alpinelinux.org/packages?name=chromium&branch=edge&arch=aarch64

armv7l...
https://pkgs.alpinelinux.org/packages?name=chromium&branch=edge&arch=armv7

should I rather create a new issue for this or can we discuss it further here?

jef · 2020-12-21T21:20:10Z

If you're having problems with install the Chromium browser, it's because of the package manager used in alpine is apk and debian uses apt. You'll need to replace those lines.

I prefer that we keep this image using alpine, as it should work on most systems.

If you'd like make this work for your ARM architecture, I think you should change the image to use arm64v8/node:14.15.0-alpine3.12.

TDCroPower · 2020-12-21T21:40:33Z

@jef the problem no longer exists for the Helios64 arm64v8, there it worked in the end, because for this there is the Chromium package at alpine.
However, there is no Chromium for armv7l at alpine, see link above.
Would need an alternative image or another variant to install Chromium in the image.

I am currently testing to rewrite the Dockerfile so that I work with buster-slim, since there should be the Chromium package on all architectures.

no reason to grant excessive permissions to your docker container, th…

526e36d

…is runs headless, no errors from chromium or puppetteer

egidijus requested a review from jef as a code owner December 9, 2020 09:07

egidijus changed the title ~~no reason to grant excessive permissions to your docker container, th…~~ fix: run in docker, and build scripts for docker Dec 9, 2020

jef reviewed Dec 9, 2020

View reviewed changes

Dockerfile Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

docker-compose.yml Outdated Show resolved Hide resolved

jef mentioned this pull request Dec 9, 2020

Docker-compose.yml #1282

Closed

egidijus and others added 3 commits December 10, 2020 08:29

cleanup: tested the groups, it appears we do not need to do this, rem…

e704ab9

…oving

fix/docs/cleanup: regular users are unlikely to need the webserver li…

2972996

…stening, setting it as disabled default in docker-compose, and adding docs for how to build docker

Merge branch 'main' into run-in-docker-no-error

61bb57a

Merge branch 'main' into run-in-docker-no-error

7dc1e5b

jef reviewed Dec 11, 2020

View reviewed changes

egidijus and others added 3 commits December 11, 2020 09:13

removing vars and mounts

ab29ba3

Co-authored-by: Jef LeCompte <jeffreylec@gmail.com>

Update docker-compose.yml

d7723a2

Co-authored-by: Jef LeCompte <jeffreylec@gmail.com>

Update Makefile

b6ded48

Co-authored-by: Jef LeCompte <jeffreylec@gmail.com>

jef approved these changes Dec 11, 2020

View reviewed changes

jef changed the title ~~fix: run in docker, and build scripts for docker~~ fix(docker): run in docker, and build scripts for docker Dec 11, 2020

jef merged commit c74ea28 into jef:main Dec 11, 2020

ven0ms99 pushed a commit to ven0ms99/streetmerchant that referenced this pull request Dec 12, 2020

fix(docker): run in docker, and build scripts for docker (jef#1291)

273d1b0

Updates unnecessary need to grant special privileges when running in Docker.

github-actions bot mentioned this pull request Oct 21, 2022

chore(main): release 3.7.0 #3060

Closed

vutting4221 mentioned this pull request Nov 27, 2023

[Snyk] Security upgrade mqtt from 4.2.6 to 4.2.7 vutting4221/streetmerchant#88

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(docker): run in docker, and build scripts for docker #1291

fix(docker): run in docker, and build scripts for docker #1291

egidijus commented Dec 9, 2020 •

edited

jef commented Dec 9, 2020

joshstubble commented Dec 9, 2020

jef commented Dec 9, 2020

egidijus commented Dec 10, 2020

jef left a comment

jef Dec 11, 2020

egidijus Dec 11, 2020

jef Dec 11, 2020

jef left a comment

rchenzheng commented Dec 12, 2020

TDCroPower commented Dec 19, 2020 •

edited

kiowadriver commented Dec 21, 2020

rchenzheng commented Dec 21, 2020

TDCroPower commented Dec 21, 2020 •

edited

jef commented Dec 21, 2020

TDCroPower commented Dec 21, 2020 •

edited

fix(docker): run in docker, and build scripts for docker #1291

fix(docker): run in docker, and build scripts for docker #1291

Conversation

egidijus commented Dec 9, 2020 • edited

Description

Testing

New dependencies

jef commented Dec 9, 2020

joshstubble commented Dec 9, 2020

jef commented Dec 9, 2020

egidijus commented Dec 10, 2020

jef left a comment

Choose a reason for hiding this comment

jef Dec 11, 2020

Choose a reason for hiding this comment

egidijus Dec 11, 2020

Choose a reason for hiding this comment

jef Dec 11, 2020

Choose a reason for hiding this comment

jef left a comment

Choose a reason for hiding this comment

rchenzheng commented Dec 12, 2020

TDCroPower commented Dec 19, 2020 • edited

kiowadriver commented Dec 21, 2020

rchenzheng commented Dec 21, 2020

TDCroPower commented Dec 21, 2020 • edited

jef commented Dec 21, 2020

TDCroPower commented Dec 21, 2020 • edited

egidijus commented Dec 9, 2020 •

edited

TDCroPower commented Dec 19, 2020 •

edited

TDCroPower commented Dec 21, 2020 •

edited

TDCroPower commented Dec 21, 2020 •

edited