Broken after restart #210

janaxhell · 2024-04-20T16:07:54Z

Today I deployed the container on a x64 machine, all wente well, I imported three dictionaries (several hours for Japanese), added DeepL API key, then I tried to use it. At that point it looked borked and the server in general became sluggish. Since it was taking forever to stop the service, I rebooted the machine. When I tried to restart Linguacafe, I got this repeated error from linguacafe-webserver:

In Connection.php line 712:
                                                                               
  SQLSTATE[HY000] [2002] Connection refused (SQL: select * from information_s  
  chema.tables where table_schema = linguacafe and table_name = migrations an  
  d table_type = 'BASE TABLE')                                                 
                                                                               
In Connector.php line 70:
                                             
  SQLSTATE[HY000] [2002] Connection refused

I haven't touched anything, just rebooted, what did I break?

The text was updated successfully, but these errors were encountered:

simjanos-dev · 2024-04-20T16:40:45Z

Before you do anything, I think you should backup your linguacafe folder to prevent any data loss if possible.

several hours for Japanese

That sounds very extreme. It takes less than 20 minutes on my slow laptop for JMDict, which is the largest I think. Do you use an HDD or SSD for the linguacafe folder? What system specifications do you have?

I haven't touched anything, just rebooted, what did I break?

It seems like the database is not working, and I think it is corrupted. I don't know if it can be fixed. I also don't know if it's the database content itself, or the server files. Can you please also copy the log from the database server as well, to see what it says when you start the server?

@sergiolaverde0 Have any idea about what can be done with the database?

janaxhell · 2024-04-20T16:57:33Z

Yes, it took at least 3 hours for jmdict and another couple of hours for japanese wikictionary and italian wikictionary.
The server has 16 Gb of RAM, system is OMV6 running from USB pendrive with /config and /data partitions on 2 separate EXT4 SSDs.

When started, this is the log from database and CPU is maxed out to 100%.

2024-04-20 16:46:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.36-1.el8 started.
2024-04-20 16:46:53+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-04-20 16:46:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.36-1.el8 started.
'/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock'
2024-04-20T16:46:54.089500Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-04-20T16:46:54.091808Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.36) starting as process 1
2024-04-20T16:46:54.101026Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

On the same server I'm running Nextcloud, Immich, Photoprism and many others, with no problem.
But I have deployed them all from Portainer setting folders as cache/data volumes, not "volumes".
I have tried to do the same with Linguacafe, but it didn't work, so I used the plain docker-compose from the git and just changed the network, because all the slots of the bridge are in use. It looks like this:

version: "3"
name: linguacafe

networks:
    omv2:
      external: true

services:
    webserver:
        image: ghcr.io/simjanos-dev/linguacafe-webserver:${VERSION:-latest}
        container_name: linguacafe-webserver
        restart: unless-stopped
        depends_on:
            - mysql
        volumes:
            - ./storage:/var/www/html/storage
        environment:
            DB_DATABASE: ${DB_DATABASE:-linguacafe}
            DB_USERNAME: ${DB_USERNAME:-linguacafe}
            DB_PASSWORD: ${DB_PASSWORD:-linguacafe}
        ports:
            - "${PORT:-9191}:80"
        networks:
            - omv2
        extra_hosts:
            - "host.docker.internal:host-gateway"
    mysql:
        image: mysql:8.0
        container_name: linguacafe-database
        restart: unless-stopped
        tty: true
        volumes:
            - ./database:/var/lib/mysql
        environment:
            MYSQL_DATABASE: ${DB_DATABASE:-linguacafe}
            MYSQL_USER: ${DB_USERNAME:-linguacafe}
            MYSQL_PASSWORD: ${DB_PASSWORD:-linguacafe}
            MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD:-linguacafe}
            SERVICE_NAME: linguacafe
        command: mysqld --general-log=1 --general-log-file=/var/lib/mysql/general-log.log
        networks:
            - omv2
    python:
        container_name: linguacafe-python-service
        command: "python3 /app/tokenizer.py"
        restart: unless-stopped
        tty: true
        image: ghcr.io/simjanos-dev/linguacafe-python-service:${VERSION:-latest}
        environment:
            PYTHONPATH: "/var/www/html/storage/app/model"
        volumes:
            - ./storage:/var/www/html/storage
        networks:
            - omv2
        platform: ${PLATFORM:-}

EDIT One thing I just remembered, /etc/docker/daemon.json is set as

{
  "data-root": "/srv/dev-disk-by-uuid-8998116d-e145-4319-8c9f-41609747f657/docker"
}

because when I installed the SSD I think I tried to get the most speed I could.

simjanos-dev · 2024-04-20T17:20:11Z

The server has 16 Gb of RAM, system is OMV6 running from USB pendrive with /config and /data partitions on 2 separate EXT4 SSDs.

{
"data-root": "/srv/dev-disk-by-uuid-8998116d-e145-4319-8c9f-41609747f657/docker"
}

Honestly, I'm not familiar with those, but it does not sound slow at all. I am using an old laptop, with an old ssd and importing JMDict took less than 20 minutes. It is a bit slow, but not several hours slow.

If you have just started using it, I think it would be better to just try a fresh install, and figure out why is it slow if it's okay with you. It will be much faster to import dictionaries again.

so I used the plain docker-compose from the git and just changed the network

I don't think this is something related to the network. Dictionary import just reads data from the storage, and saves it in the database after a minimal text processing. If you can import any text, then the network part should be fine.

2024-04-20T16:46:54.101026Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

It seems like from the log that the database is just starting up.

I have no idea what could cause the slowdown.

sergiolaverde0 · 2024-04-20T17:49:02Z

On the same server I'm running Nextcloud, Immich, Photoprism and many others, with no problem

I used the plain docker-compose from the git and just changed the network

Are the other containers running on the same network? Do any of them use a different MySQL container? Did you set any of the variables for the Database container (username, passowrd, ect)?

janaxhell · 2024-04-20T18:04:42Z

Are the other containers running on the same network?

Yes, of course. Portainer counts 89 running containers (with a few more stopped for various reasons). At some point when I was adding them, the system refused to add more, because there were no more non-overlapping IPs (from what I understood by the error popouts), so I created a new one and set all subsequent containers to "omv2" network.

Do any of them use a different MySQL container?

Most likely, but I'd have to check them one by one... Anyway they are always separated. If a container uses a db, I mount it in a folder that is usually /config/[container-name]/db:/db

Did you set any of the variables for the Database container (username, passowrd, ect)?

If you mean in the Linguacafe db, no, I left all settings to default with idea of changing them after checking it was working.
If you mean other containers db, yes, usually I set a user and pass different then admin/admin or things like that.

janaxhell · 2024-04-20T18:07:27Z

If you have just started using it, I think it would be better to just try a fresh install, and figure out why is it slow if it's okay with you. It will be much faster to import dictionaries again.

No problem, just tell me what to do/delete.

2024-04-20T16:46:54.101026Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

It seems like from the log that the database is just starting up.

I have no idea what could cause the slowdown.

The first time I started the LinguaCafe, everything was smooth, it slowed down as soon as I started importing JMCT dictionary.
After that, it was just slow and refused to stop/restart, or at least it was so slow that after like 10 mins I rebooted OMV.

janaxhell · 2024-04-20T18:16:05Z

Mmm I just noticed that Ghost uses MySQL (I installed this yesterday, still haven't used it):

version: '3.1'

services:

  ghost:
    image: ghost:5-alpine
    restart: always
    ports:
      - 2222:2368
    environment:
      # see https://ghost.org/docs/config/#configuration-options
      database__client: mysql
      database__connection__host: db
      database__connection__user: xxxxxxx
      database__connection__password: xxxxxxxxx
      database__connection__database: ghost
      # this url value is just an example, and is likely wrong for your environment!
      url: http://192.168.1.164:2222
      # contrary to the default mentioned in the linked documentation, this image defaults to NODE_ENV=production (so development mode needs to be explicitly specified if desired)
      #NODE_ENV: development
    volumes:
      - /srv/dev-disk-by-uuid-8998116d-e145-4319-8c9f-41609747f657/Configs/ghost/data:/var/lib/ghost/content

  db:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: xxxxxxxx

janaxhell · 2024-04-20T18:18:37Z

And Seafile as well:

version: '2.0'
services:
  db:
    image: mariadb:10.5
    container_name: seafile-mysql
    environment:
      - MYSQL_ROOT_PASSWORD=xxxxxxxxx  # Requested, set the root's password of MySQL service.
      - MYSQL_LOG_CONSOLE=true
    volumes:
      - /srv/dev-disk-by-uuid-8998116d-e145-4319-8c9f-41609747f657/Configs/Seafile/db:/var/lib/mysql  # Requested, specifies the path to MySQL data persistent store.
    networks:
      - seafile-net

  memcached:
    image: memcached:1.6
    container_name: seafile-memcached
    entrypoint: memcached -m 256
    networks:
      - seafile-net
          
  seafile:
    image: seafileltd/seafile-mc:latest
    container_name: seafile
    ports:
      - "8480:80"
  #   - "493:443"  # If https is enabled, cancel the comment.
    volumes:
      - /srv/dev-disk-by-uuid-ba4514c7-4c36-4c08-9827-3f3f584ab6b0/data/seafile:/shared   # Requested, specifies the path to Seafile data persistent store.
    environment:
      - DB_HOST=db
      - DB_ROOT_PASSWD=xxxxxxxxx  # Requested, the value shuold be root's password of MySQL service.
      - TIME_ZONE=Europe/Rome  # Optional, default is UTC. Should be uncomment and set to your local time zone.
      - SEAFILE_ADMIN_EMAIL=xxxxxxxxxx # Specifies Seafile admin user, default is 'me@example.com'.
      - SEAFILE_ADMIN_PASSWORD=xxxxxxxxxxx     # Specifies Seafile admin password, default is 'asecret'.
      - SEAFILE_SERVER_LETSENCRYPT=false   # Whether to use https or not.
      - SEAFILE_SERVER_HOSTNAME=xxxxxxxxxxx # Specifies your host name if https is enabled.
    depends_on:
      - db
      - memcached
    networks:
      - seafile-net

networks:
  seafile-net:

sergiolaverde0 · 2024-04-20T18:23:44Z

Do any of them use a different MySQL container?

Most likely, but I'd have to check them one by one... Anyway they are always separated. If a container uses a db, I mount it in a folder that is usually /config/[container-name]/db:/db

Good practice, but I suspect the webserver is trying to knock on a different container's door as that is the most likely explanation of the SQLSTATE[HY000] [2002] Connection refused error you got. Neither Linguacafe or those two services you showed us specify a port where their MySQL should bound to, and my hypothesis is that after restart it tried to connect to a database it had no access to.

Running Linguacafe in its own network is probably the best solution right now.

simjanos-dev · 2024-04-21T22:06:23Z

Did you have any success figuring out what's wrong?

janaxhell · 2024-04-22T09:09:21Z

I will try again tomorrow afternoon, I'm busy with university. I need a fat timeslot to handle this kind of things ;-)

simjanos-dev · 2024-04-22T09:12:29Z

Oh, okay. Didn't mean to rush you, I was just curious if I could help with something.

janaxhell · 2024-04-23T16:15:54Z

So, I've started from scratch and used the docker-compose as-is, nothing touched. I've been able to set it up and running. But again the dictionary import is slow. After 40 minutes, JMDict is at 28%. One thing that I didn't mention previously is that in Admin Settings/Dictionaries I've set target for JMDict and Japanese wikictionary to Italian and Japanese as target to Italian wikictionary. I've noticed that the import popup window for JMDict kept English as target. Is it possible that that' the reason of this slowness?

simjanos-dev · 2024-04-23T16:29:28Z

I'm not sure I understand completely. I just realized that you can change JMDict source language, which is a bug.

Source language: The language that you want to use the dictionary for. So if you have a Japanese - English dictionary, the source language should be japanese.

Target language: It's just for visual help, it does not change anything. It's for cases when you use multi lingual dictionaries for one source language. For example you learn Japanese, then you use a Japanese - English, and Japanese - German dictionary at the same time. Changing this does not change the content of the dictionary.

Is it possible that that' the reason of this slowness?

In any case, it should not be related to the slowness. I really have no idea what causes this. I asked on discord if anyone had similar experiences.

janaxhell · 2024-04-23T16:42:38Z

Actually it seems completely stuck. I just checked it and it's still at 28%. Last time was slow, but didn't get stuck.
webserver is repeating

192.168.1.21 - - [23/Apr/2024:16:01:24 +0000] "GET /dictionaries/get-record-count/dict_jp_jmdict HTTP/1.1" 200 1268 "http://192.168.1.164:9191/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
192.168.1.21 - - [23/Apr/2024:16:01:36 +0000] "GET /dictionaries/get-record-count/dict_jp_jmdict HTTP/1.1" 200 1268 "http://192.168.1.164:9191/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
192.168.1.21 - - [23/Apr/2024:16:01:48 +0000] "GET /dictionaries/get-record-count/dict_jp_jmdict HTTP/1.1" 200 1268 "http://192.168.1.164:9191/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
192.168.1.21 - - [23/Apr/2024:16:02:07 +0000] "GET /dictionaries/get-record-count/dict_jp_jmdict HTTP/1.1" 200 1268 "http://192.168.1.164:9191/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
192.168.1.21 - - [23/Apr/2024:16:02:22 +0000] "GET /dictionaries/get-record-count/dict_jp_jmdict HTTP/1.1" 200 1268 "http://192.168.1.164:9191/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
192.168.1.21 - - [23/Apr/2024:16:02:34 +0000] "GET /dictionaries/get-record-count/dict_jp_jmdict HTTP/1.1" 200 1268 "http://192.168.1.164:9191/admin" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

simjanos-dev · 2024-04-23T17:09:36Z

These are requests to ask for the current database record count, so it can update the progress bar.

janaxhell · 2024-04-23T17:20:14Z

Since it was still at 28%, I refreshed the page and started over. Will see what happens.
EDIT This time it took about 2 hours, but it didn't get stuck. Now I will try to import the other 2 dictionaries.

sergiolaverde0 · 2024-04-24T23:52:32Z

in Admin Settings/Dictionaries I've set target for JMDict and Japanese wikictionary to Italian and Japanese as target to Italian wikictionary

I... I didn't know you could do that. After checking the files of JMDict directly I'm quite certain that you shouldn't do that or at least won't accomplish anything with it. Yet, I mimicked it and it still imported normally.

While your Linguacafe stack is having a seizure do the rest of your services work normally or is it propagating to them too? Does the all the CPU usage really come from these containers?

janaxhell · 2024-04-25T09:34:15Z

The third diictionary I imported, Italian wikictionary actually took about 20 mins. Basically each one I imported took less than the previous, so about 2 hours, 1 hour, 20 mintues.

These are Grafana graphs of the past 2 days. I have screenshotted both the whole set of containers running and the single separate Linguacafe containers. Check the middle/lower left frames.
Orange line in lower frame is the whole Linguacafe stack, the middle frame is the various components isolated from the rest of containers, except last 2 screenshots. The swap partition is always filled by Nextcloud AIO, I don't know why, as there's always plenty of RAM available.

WEBSERVER

PYTHON SERVICE

DATABASE

ALL CONTAINERS

ALL CONTAINERS - UPPER PART OF GRAPHS

simjanos-dev · 2024-04-25T10:12:18Z

The third diictionary I imported, Italian wikictionary actually took about 20 mins. Basically each one I imported took less than the previous, so about 2 hours, 1 hour, 20 mintues.

Every dictionary takes a different amount of time based on what kind of dictionary file it is, and how many record it contains.

Italian wiktionary took me 1:30 minutes.

My laptop: Intel i5-8250u CPU, 8GB RAM, 128GB 5+ year old SSD.

janaxhell · 2024-04-25T10:23:45Z

I don't know what to say. I just deployed the compose as-is and that is how it behaves on my machine.

Intel i3-7100T CPU @ 3.40GHz, 32GB RAM, 512GB 1 year SSD

The server works fine with everything else.

janaxhell · 2024-05-18T13:33:37Z

Just updated to v0.12.1
I had to recreate user and re-import dictionaries, but this time import took about 7 minutes for all of them.

simjanos-dev · 2024-05-21T23:58:18Z

I have no idea what happened, but I am happy its working now. :)

simjanos-dev closed this as completed May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Broken after restart #210

Broken after restart #210

janaxhell commented Apr 20, 2024 •

edited

simjanos-dev commented Apr 20, 2024 •

edited

janaxhell commented Apr 20, 2024 •

edited

simjanos-dev commented Apr 20, 2024

sergiolaverde0 commented Apr 20, 2024

janaxhell commented Apr 20, 2024 •

edited

janaxhell commented Apr 20, 2024

janaxhell commented Apr 20, 2024

janaxhell commented Apr 20, 2024

sergiolaverde0 commented Apr 20, 2024

simjanos-dev commented Apr 21, 2024

janaxhell commented Apr 22, 2024

simjanos-dev commented Apr 22, 2024

janaxhell commented Apr 23, 2024

simjanos-dev commented Apr 23, 2024 •

edited

janaxhell commented Apr 23, 2024

simjanos-dev commented Apr 23, 2024

janaxhell commented Apr 23, 2024 •

edited

sergiolaverde0 commented Apr 24, 2024

janaxhell commented Apr 25, 2024

simjanos-dev commented Apr 25, 2024

janaxhell commented Apr 25, 2024

janaxhell commented May 18, 2024

simjanos-dev commented May 21, 2024

Broken after restart #210

Broken after restart #210

Comments

janaxhell commented Apr 20, 2024 • edited

simjanos-dev commented Apr 20, 2024 • edited

janaxhell commented Apr 20, 2024 • edited

simjanos-dev commented Apr 20, 2024

sergiolaverde0 commented Apr 20, 2024

janaxhell commented Apr 20, 2024 • edited

janaxhell commented Apr 20, 2024

janaxhell commented Apr 20, 2024

janaxhell commented Apr 20, 2024

sergiolaverde0 commented Apr 20, 2024

simjanos-dev commented Apr 21, 2024

janaxhell commented Apr 22, 2024

simjanos-dev commented Apr 22, 2024

janaxhell commented Apr 23, 2024

simjanos-dev commented Apr 23, 2024 • edited

janaxhell commented Apr 23, 2024

simjanos-dev commented Apr 23, 2024

janaxhell commented Apr 23, 2024 • edited

sergiolaverde0 commented Apr 24, 2024

janaxhell commented Apr 25, 2024

simjanos-dev commented Apr 25, 2024

janaxhell commented Apr 25, 2024

janaxhell commented May 18, 2024

simjanos-dev commented May 21, 2024

janaxhell commented Apr 20, 2024 •

edited

simjanos-dev commented Apr 20, 2024 •

edited

janaxhell commented Apr 20, 2024 •

edited

janaxhell commented Apr 20, 2024 •

edited

simjanos-dev commented Apr 23, 2024 •

edited

janaxhell commented Apr 23, 2024 •

edited