Infinite loop in backend? #223

alexislefebvre · 2021-03-31T19:32:49Z

🐛 Bug Report

I am including my log files in zip format.
I have looked for similar issues among all the former issues including closed ones.

What Operating system and version is LibrePhotos running on:

Debian, last Docker release.

What architecture is LibrePhotos running on:

x64

How is LibrePhotos installed:

Docker

If running via Docker or Kubernets please list version including docker-compose:

Click to expand!

# DO NOT EDIT
# The .env file has everything you need to edit.
# Run options:
# 1. Use prebuilt images (preferred method):
#   run cmd: docker-compose up -d
# 2. Build images on your own machine:
#   build cmd: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build
#   run cmd: docker-compose up -d

version: '3.3'
services:
proxy:
  image: reallibrephotos/librephotos-proxy:${tag}
  restart: unless-stopped
  volumes:
    - ${myPhotos}:/data
    - ${proMedia}:/protected_media
  ports:
    - ${httpPort}:80
  depends_on:
    - backend
    - frontend

db:
  image: postgres:13
  restart: unless-stopped
  environment:
    - POSTGRES_USER=${dbUser}
    - POSTGRES_PASSWORD=${dbPass}
    - POSTGRES_DB=${dbName}
  volumes:
    - ${dbLocation}:/var/lib/postgresql/data
  command: postgres -c fsync=off -c synchronous_commit=off -c full_page_writes=off -c random_page_cost=1.0

frontend:
  image: reallibrephotos/librephotos-frontend:${tag}
  restart: always
  depends_on:
    - backend

backend:
  image: reallibrephotos/librephotos:${tag}
  restart: unless-stopped
  volumes:
    - ${myPhotos}:/data:ro
    - ${proMedia}:/protected_media
    - ${logLocation}:/logs
    - ${cachedir}:/root/.cache

  environment:
    - SECRET_KEY=${shhhhKey}
    - BACKEND_HOST=backend
    - ADMIN_EMAIL=${adminEmail}
    - ADMIN_USERNAME=${userName}
    - ADMIN_PASSWORD=${userPass}
    - DB_BACKEND=postgresql
    - DB_NAME=${dbName}
    - DB_USER=${dbUser}
    - DB_PASS=${dbPass}
    - DB_HOST=db
    - DB_PORT=5432
    - REDIS_HOST=redis
    - REDIS_PORT=6379
    - MAPBOX_API_KEY=${mapApiKey}
    - TIME_ZONE=${timeZone}
    - WEB_CONCURRENCY=${gunniWorkers}
    - SKIP_PATTERNS=${skipPatterns}
    - DEBUG=0

  # Wait for Postgres
  depends_on:
    - db

redis:
  image: redis:6
  restart: unless-stopped

The difference with default conf is - ${myPhotos}:/data:ro, my filed are mounted on a read-only mount point.

Are you running LibrePhotos on a virtual machine if so please list:

How is you picture library mounted on the host (or in the virtual machine):

Local file system (ext4)

Description of issue:

cd /var/librephotos/ ; docker-compose logs --tail 20 backend
Attaching to librephotos_backend_1
backend_1   | [2021-03-31 19:26:58 +0000] [313] [INFO] Booting worker with pid: 313
backend_1   | [2021-03-31 19:27:03 +0000] [320] [INFO] Booting worker with pid: 320
backend_1   | [2021-03-31 19:27:06 +0000] [323] [INFO] Booting worker with pid: 323
backend_1   | [2021-03-31 19:27:11 +0000] [328] [INFO] Booting worker with pid: 328
backend_1   | [2021-03-31 19:27:15 +0000] [333] [INFO] Booting worker with pid: 333
backend_1   | [2021-03-31 19:27:19 +0000] [338] [INFO] Booting worker with pid: 338
backend_1   | [2021-03-31 19:27:23 +0000] [343] [INFO] Booting worker with pid: 343
backend_1   | [2021-03-31 19:27:27 +0000] [348] [INFO] Booting worker with pid: 348
backend_1   | [2021-03-31 19:27:30 +0000] [353] [INFO] Booting worker with pid: 353
backend_1   | [2021-03-31 19:27:35 +0000] [358] [INFO] Booting worker with pid: 358
backend_1   | [2021-03-31 19:27:39 +0000] [363] [INFO] Booting worker with pid: 363
backend_1   | [2021-03-31 19:27:43 +0000] [368] [INFO] Booting worker with pid: 368
backend_1   | [2021-03-31 19:27:46 +0000] [373] [INFO] Booting worker with pid: 373
backend_1   | [2021-03-31 19:27:51 +0000] [378] [INFO] Booting worker with pid: 378
backend_1   | [2021-03-31 19:27:55 +0000] [383] [INFO] Booting worker with pid: 383
backend_1   | [2021-03-31 19:27:59 +0000] [388] [INFO] Booting worker with pid: 388
backend_1   | [2021-03-31 19:28:02 +0000] [393] [INFO] Booting worker with pid: 393
backend_1   | [2021-03-31 19:28:06 +0000] [398] [INFO] Booting worker with pid: 398
backend_1   | [2021-03-31 19:28:11 +0000] [403] [INFO] Booting worker with pid: 403
backend_1   | [2021-03-31 19:28:14 +0000] [408] [INFO] Booting worker with pid: 408

cat …/librephotos_logs/gunicorn_django.log
[2021-03-31 19:27:19 +0000] [338] [INFO] Booting worker with pid: 338
[2021-03-31 19:27:23 +0000] [343] [INFO] Booting worker with pid: 343
[2021-03-31 19:27:27 +0000] [348] [INFO] Booting worker with pid: 348
[2021-03-31 19:27:30 +0000] [353] [INFO] Booting worker with pid: 353
[2021-03-31 19:27:35 +0000] [358] [INFO] Booting worker with pid: 358
[2021-03-31 19:27:39 +0000] [363] [INFO] Booting worker with pid: 363
[2021-03-31 19:27:43 +0000] [368] [INFO] Booting worker with pid: 368
[2021-03-31 19:27:46 +0000] [373] [INFO] Booting worker with pid: 373
[2021-03-31 19:27:51 +0000] [378] [INFO] Booting worker with pid: 378
[2021-03-31 19:27:55 +0000] [383] [INFO] Booting worker with pid: 383
[2021-03-31 19:27:59 +0000] [388] [INFO] Booting worker with pid: 388
[2021-03-31 19:28:02 +0000] [393] [INFO] Booting worker with pid: 393
[2021-03-31 19:28:06 +0000] [398] [INFO] Booting worker with pid: 398
[2021-03-31 19:28:11 +0000] [403] [INFO] Booting worker with pid: 403
[2021-03-31 19:28:14 +0000] [408] [INFO] Booting worker with pid: 408
[2021-03-31 19:28:18 +0000] [413] [INFO] Booting worker with pid: 413
[2021-03-31 19:28:22 +0000] [418] [INFO] Booting worker with pid: 418
[2021-03-31 19:28:26 +0000] [423] [INFO] Booting worker with pid: 423
[2021-03-31 19:28:30 +0000] [428] [INFO] Booting worker with pid: 428
[2021-03-31 19:28:34 +0000] [433] [INFO] Booting worker with pid: 433
[2021-03-31 19:28:38 +0000] [438] [INFO] Booting worker with pid: 438
[2021-03-31 19:28:42 +0000] [443] [INFO] Booting worker with pid: 443
[2021-03-31 19:28:45 +0000] [448] [INFO] Booting worker with pid: 448
[2021-03-31 19:28:49 +0000] [453] [INFO] Booting worker with pid: 453
[2021-03-31 19:28:54 +0000] [458] [INFO] Booting worker with pid: 458
[2021-03-31 19:28:58 +0000] [463] [INFO] Booting worker with pid: 463
[2021-03-31 19:29:02 +0000] [468] [INFO] Booting worker with pid: 468
[2021-03-31 19:29:05 +0000] [473] [INFO] Booting worker with pid: 473
[2021-03-31 19:29:10 +0000] [480] [INFO] Booting worker with pid: 480
[2021-03-31 19:29:13 +0000] [483] [INFO] Booting worker with pid: 483
[2021-03-31 19:29:17 +0000] [488] [INFO] Booting worker with pid: 488
[2021-03-31 19:29:21 +0000] [493] [INFO] Booting worker with pid: 493
[2021-03-31 19:29:25 +0000] [498] [INFO] Booting worker with pid: 498
[2021-03-31 19:29:29 +0000] [503] [INFO] Booting worker with pid: 503
[2021-03-31 19:29:33 +0000] [508] [INFO] Booting worker with pid: 508
[2021-03-31 19:29:37 +0000] [513] [INFO] Booting worker with pid: 513
[2021-03-31 19:29:41 +0000] [518] [INFO] Booting worker with pid: 518
[2021-03-31 19:29:45 +0000] [523] [INFO] Booting worker with pid: 523
[2021-03-31 19:29:49 +0000] [528] [INFO] Booting worker with pid: 528
[2021-03-31 19:29:52 +0000] [533] [INFO] Booting worker with pid: 533
[2021-03-31 19:29:56 +0000] [538] [INFO] Booting worker with pid: 538
[2021-03-31 19:30:00 +0000] [543] [INFO] Booting worker with pid: 543
[2021-03-31 19:30:04 +0000] [548] [INFO] Booting worker with pid: 548
[2021-03-31 19:30:09 +0000] [553] [INFO] Booting worker with pid: 553
[2021-03-31 19:30:13 +0000] [558] [INFO] Booting worker with pid: 558
[2021-03-31 19:30:17 +0000] [563] [INFO] Booting worker with pid: 563
[2021-03-31 19:30:22 +0000] [568] [INFO] Booting worker with pid: 568
[2021-03-31 19:30:27 +0000] [573] [INFO] Booting worker with pid: 573
[2021-03-31 19:30:33 +0000] [580] [INFO] Booting worker with pid: 580
[2021-03-31 19:30:35 +0000] [583] [INFO] Booting worker with pid: 583

My CPU has too much usage, I have to stop LibrePhotos.

How can we reproduce it:

I don't know.

Additional Information:

The text was updated successfully, but these errors were encountered:

derneuere · 2021-03-31T19:53:09Z

In this issue it looks like it might be memory related: benoitc/gunicorn#1663
Could you reduce you gunicorn worker to 1 in your config?

alexislefebvre · 2021-03-31T22:57:59Z

Thanks for your answer. The behaviour is the same with one worker. I have 8 GB of RAM.

darrepac · 2021-04-02T21:09:36Z

@alexislefebvre you are not the only one with this bug... some others reported it through discord... let see if @derneuere can do some magic here to solve it ;)

alexislefebvre · 2021-04-12T15:17:13Z

The CPU of my server is “Intel(R) Celeron(R) CPU 1037U @ 1.80GHz”. It doesn't support AVX.

alexislefebvre · 2021-04-12T15:29:26Z

See also LibrePhotos/librephotos-docker#7

alexislefebvre · 2021-04-15T18:31:57Z

With the last update, with this fix:

docker-compose logs --tail 20 --follow backend
Attaching to librephotos-ayu_backend_1
backend_1   | statd: unrecognized service
backend_1   | /entrypoint.sh: line 10:    48 Illegal instruction     (core dumped) python manage.py clear_cache
backend_1   | Running backend server...
backend_1   | production backend starting
backend_1   | [2021-04-15 18:28:28 +0000] [63] [INFO] Starting gunicorn 20.1.0
backend_1   | [2021-04-15 18:28:28 +0000] [63] [INFO] Listening at: http://172.23.0.4:8001 (63)
backend_1   | [2021-04-15 18:28:28 +0000] [63] [INFO] Using worker: gevent
backend_1   | [2021-04-15 18:28:28 +0000] [67] [INFO] Booting worker with pid: 67
backend_1   | [2021-04-15 18:28:39 +0000] [63] [WARNING] Worker with pid 67 was terminated due to signal 4
backend_1   | [2021-04-15 18:28:39 +0000] [76] [INFO] Booting worker with pid: 76
backend_1   | [2021-04-15 18:28:46 +0000] [63] [WARNING] Worker with pid 76 was terminated due to signal 4
backend_1   | [2021-04-15 18:28:46 +0000] [81] [INFO] Booting worker with pid: 81
backend_1   | [2021-04-15 18:28:54 +0000] [63] [WARNING] Worker with pid 81 was terminated due to signal 4
backend_1   | [2021-04-15 18:28:54 +0000] [86] [INFO] Booting worker with pid: 86
backend_1   | [2021-04-15 18:29:01 +0000] [63] [WARNING] Worker with pid 86 was terminated due to signal 4
backend_1   | [2021-04-15 18:29:01 +0000] [91] [INFO] Booting worker with pid: 91

And so on.

deliantwo · 2021-04-15T20:52:37Z

The last update did not work for me (same error as before).
I've rebuilt the backend docker image on the server (with a CPU without the AVX support) and I can confirm that it works! 🎉

derneuere · 2021-04-15T20:54:11Z

@deliantwo Could you upload your backend image to dockerhub? Just to check if it resolves @alexislefebvre problems too.

darrepac · 2021-04-15T21:04:46Z

eager to test too! or tell me how to rebuild the backend docker image

alexislefebvre · 2021-04-15T21:31:32Z

@darrepac you can try this:

# 2. Build images on your own machine:
#   build cmd: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build
#   run cmd: docker-compose up -d

Source: https://github.com/LibrePhotos/librephotos-docker/blob/9fb707b9e653f2e9bbabfe3df364a5cc1c9fd8c4/docker-compose.yml#L6-L8

alexislefebvre · 2021-04-15T21:36:14Z

I must be missing something:

$ export COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 ; docker-compose build
db uses an image, skipping
backend uses an image, skipping
frontend uses an image, skipping
proxy uses an image, skipping
redis uses an image, skipping
WARNING: Native build is an experimental feature and could change at any time
$ docker-compose -v
docker-compose version 1.25.0, build 0a186604

alexislefebvre · 2021-04-15T21:51:56Z

The docker-compose build command can't work because the docker-compose.yml file use a image node instead of build with context.

deliantwo · 2021-04-15T22:17:02Z

@deliantwo Could you upload your backend image to dockerhub? Just to check if it resolves @alexislefebvre problems too.

Sure, no problem. I'm pushing it right now but I'm on a slow network and it's 4.32GB... have faith 😄
When the push ends you will find it here

darrepac · 2021-04-16T12:57:46Z

Sorry for the noob question but what do I have to put in yml file to use your image instead of the current one?

deliantwo · 2021-04-16T13:29:34Z

@darrepac Just change the backend image repository. I've pushed the same tag dev, so no additional change is needed

@@ -39,7 +39,7 @@
       - backend
 
   backend:
-    image: reallibrephotos/librephotos:${tag}
+    image: delian2/librephotos:${tag}
     restart: always
     volumes:
       - ${myPhotos}:/data

alexislefebvre · 2021-04-16T14:35:16Z

Thanks a lot @deliantwo, it works! 🎉

derneuere · 2021-04-16T20:20:04Z

All right, I will write a tutorial on how to build the backend image yourself for users who use CPUs without AVX support.

darrepac · 2021-04-17T19:05:59Z

@deliantwo well done, solve the problem on my side too

derneuere · 2021-04-26T19:34:12Z

Wrote docs about it: https://docs.librephotos.com/1/old_processors/

darrepac · 2021-05-13T07:53:20Z

@darrepac Just change the backend image repository. I've pushed the same tag dev, so no additional change is needed
@@ -39,7 +39,7 @@
       - backend
 
   backend:
-    image: reallibrephotos/librephotos:${tag}
+    image: delian2/librephotos:${tag}
     restart: always
     volumes:
       - ${myPhotos}:/data

@deliantwo do you plan to maintain the image or not?

thx

…norepo Update dependency @babel/eslint-parser to v7.22.9

alexislefebvre added the bug Something isn't working label Mar 31, 2021

alexislefebvre mentioned this issue Apr 12, 2021

Worker process keeps being terminated due to signal 4 #232

Closed

derneuere closed this as completed Apr 26, 2021

AnkurPrabhu pushed a commit to AnkurPrabhu/librephotos that referenced this issue Oct 22, 2023

Merge pull request LibrePhotos#223 from LibrePhotos/renovate/babel-mo…

de3d16c

…norepo Update dependency @babel/eslint-parser to v7.22.9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Infinite loop in backend? #223

Infinite loop in backend? #223

alexislefebvre commented Mar 31, 2021 •

edited

derneuere commented Mar 31, 2021

alexislefebvre commented Mar 31, 2021 •

edited

darrepac commented Apr 2, 2021

alexislefebvre commented Apr 12, 2021 •

edited

alexislefebvre commented Apr 12, 2021

alexislefebvre commented Apr 15, 2021

deliantwo commented Apr 15, 2021 •

edited

derneuere commented Apr 15, 2021

darrepac commented Apr 15, 2021

alexislefebvre commented Apr 15, 2021 •

edited

alexislefebvre commented Apr 15, 2021 •

edited

alexislefebvre commented Apr 15, 2021 •

edited

deliantwo commented Apr 15, 2021 •

edited

darrepac commented Apr 16, 2021

deliantwo commented Apr 16, 2021 •

edited

alexislefebvre commented Apr 16, 2021

derneuere commented Apr 16, 2021

darrepac commented Apr 17, 2021

derneuere commented Apr 26, 2021

darrepac commented May 13, 2021

Infinite loop in backend? #223

Infinite loop in backend? #223

Comments

alexislefebvre commented Mar 31, 2021 • edited

🐛 Bug Report

What Operating system and version is LibrePhotos running on:

What architecture is LibrePhotos running on:

How is LibrePhotos installed:

If running via Docker or Kubernets please list version including docker-compose:

Are you running LibrePhotos on a virtual machine if so please list:

How is you picture library mounted on the host (or in the virtual machine):

Description of issue:

How can we reproduce it:

Additional Information:

derneuere commented Mar 31, 2021

alexislefebvre commented Mar 31, 2021 • edited

darrepac commented Apr 2, 2021

alexislefebvre commented Apr 12, 2021 • edited

alexislefebvre commented Apr 12, 2021

alexislefebvre commented Apr 15, 2021

deliantwo commented Apr 15, 2021 • edited

derneuere commented Apr 15, 2021

darrepac commented Apr 15, 2021

alexislefebvre commented Apr 15, 2021 • edited

alexislefebvre commented Apr 15, 2021 • edited

alexislefebvre commented Apr 15, 2021 • edited

deliantwo commented Apr 15, 2021 • edited

darrepac commented Apr 16, 2021

deliantwo commented Apr 16, 2021 • edited

alexislefebvre commented Apr 16, 2021

derneuere commented Apr 16, 2021

darrepac commented Apr 17, 2021

derneuere commented Apr 26, 2021

darrepac commented May 13, 2021

alexislefebvre commented Mar 31, 2021 •

edited

alexislefebvre commented Mar 31, 2021 •

edited

alexislefebvre commented Apr 12, 2021 •

edited

deliantwo commented Apr 15, 2021 •

edited

alexislefebvre commented Apr 15, 2021 •

edited

alexislefebvre commented Apr 15, 2021 •

edited

alexislefebvre commented Apr 15, 2021 •

edited

deliantwo commented Apr 15, 2021 •

edited

deliantwo commented Apr 16, 2021 •

edited