install new stt server error (on Celeron CPU or certain VMs) #218

royrogermcfreely · 2022-12-10T11:20:40Z

hey,
i run sepia on a debian vm with 4 cores and 16gb ram (for stt testing)

i tried to download/install the new stt server via docker commands from the wiki. but when i run
sudo docker run --rm --name=sepia-stt -p 20741:20741 -it sepia/stt-server:latest

i get this error:

SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch

the same happens when i try to install the stt server with this command

wget https://raw.githubusercontent.com/SEPIA-Framework/sepia-stt-server/master/scripts/install.sh
bash install.sh -y
bash run.sh

then i get this error:

Running SEPIA STT-Server
run.sh: Zeile 23: 27702 Ungültiger Maschinenbefehl   nohup python3 -u -m launch &>> ../log.out
curl: (7) Failed to connect to localhost port 20741: Verbindungsaufbau abgelehnt

2022_12_10_11:58:56 - Start
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...

i removed all old docker stt images and cleaned the folder where i downloaded the files from install.sh script.

the old stt server was working.

could you give me a hint how i get the new stt server working?

/roy

The text was updated successfully, but these errors were encountered:

fquirin · 2022-12-10T22:18:29Z

Hey roy,

this is the x86 container right?
Just tried it with the same command and a fresh load from DockerHUB and its working for me 🤔 .

Failed to connect to localhost port 20741: Verbindungsaufbau abgelehnt

I wonder if there is something blocking the port maybe?

vanseforge · 2022-12-20T17:29:41Z

Hi,

I have the same problem like roy: I tried to install the stt server on a Synology DS920+ (CPU Intel Celeron J4125 -> x86 container) via docker an get also the following message:

sepia-stt     | SEPIA STT Server - Settings file used: './server.conf'
sepia-stt     | SEPIA STT Server - Settings file tag: 'Dynamic server settings'
sepia-stt     | SEPIA STT Server - Starting...
sepia-stt     | on-docker.sh: line 2:     7 Illegal instruction     (core dumped) python3 -m launch
sepia-stt exited with code 132

PlanetSmasher · 2022-12-21T00:09:08Z

Hi Florian,

I have the same issues on my standard "docker" host but aswell as on my testing machine where this image was never pulled before.
It always tells me the following.

sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     7 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1  | on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
sepia-stt-server-sepia-1 exited with code 132

I am running Debian GNU/Linux 11 (bullseye) in a Proxmox VM on an Intel NUC
so it is amd64

As I do not use any custom models yet I just did not set up a custom settings.conf

fquirin · 2022-12-21T19:03:51Z

Ok, so clearly something is going on here :-/. Unfortunately I cannot reproduce the issue and wonder what could make the same Docker image fail on certain machines and work on others 🤔, after all this is exactly what should NOT happen with Docker 🙈 .

@royrogermcfreely , @PlanetSmasher , @vanseforge could you all run the following commands for me once plz:

uname -a
docker --version

Are you using the Docker containers in the default network mode (bridge I think) or in host mode without network isolation? (or any other mode?)

I'll try to spin up more test systems in the meantime to see if any of them show the same error.

PlanetSmasher · 2022-12-21T20:52:03Z

I am running Debian 11

user@systemname:~$ uname -a
Linux systemname 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
user@systemname:~$ docker --version
Docker version 20.10.21, build baeda1f

For easier use I wrote a quick docker-compose.yml :

version: '3'

services:
    sepia:
#       image: sepia/stt-server:vosk_amd64 #2018 model
#       image: sepia/stt-server:latest #2021 rewrite
       image: sepia/stt-server:dynamic_v1.0.0_amd64
       ports:
          - 20741:20741
       volumes:
           - './sepia-stt-share:/apps/share'
           - './server.conf:/home/admin/sepia-stt/share/my.conf'
       restart: unless-stopped
       environment:
           - SEPIA_STT_SETTINGS=/home/admin/sepia-stt/share/my.conf

fquirin · 2022-12-21T21:35:37Z

That looks perfectly fine, as far as I can tell. sepia/stt-server:latest and sepia/stt-server:dynamic_v1.0.0_amd64 should point to the same image btw at the moment.
Can you run the same command without ports, volumes and environment just to see what happens plz. Or simply: sudo docker run --rm --name=sepia-stt -it sepia/stt-server:dynamic_v1.0.0_amd64

PlanetSmasher · 2022-12-21T22:57:59Z

here it comes

root@someserver:/opt/docker/sepia-stt-server# sudo docker run --rm --name=sepia-stt -it sepia/stt-server:dynamic_v1.0.0_amd64
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
on-docker.sh: line 2:     8 Illegal instruction     (core dumped) python3 -m launch
root@someserver:/opt/docker/sepia-stt-server#

what I've found out so far:
The on-docker.sh runs the python3 -m launch command which tries to start the vosk stuff

I started the docker with the default docker run and added /bin/bash so I can get to the console.
now i changed to the ~/sepia-stt/server directory and executed the launch command again

->

> admin@a6b00abb49f1:~/sepia-stt/server$ python3 -m launch
> SEPIA STT Server - Settings file used: './server.conf'
> SEPIA STT Server - Settings file tag: 'Dynamic server settings'
> SEPIA STT Server - Starting...
> Illegal instruction (core dumped)
> admin@a6b00abb49f1:~/sepia-stt/server$

fquirin · 2022-12-22T12:31:42Z

The on-docker.sh runs the python3 -m launch command which tries to start the vosk stuff

Correct, the weird thing is that the only situation I've ever seen this error is when calling a x86 command from arm64 or the other way around.

I think we have to go step by step through the Python script and see where it breaks. Could you do the following:

Enter Docker image with bash: sudo docker run --rm --name=sepia-stt2 -it sepia/stt-server:latest /bin/bash
Go to cd sepia-stt/server
Try once to run the server directly (w/o launch script): python3 -m server
If you get the same error open Python with python3 and import the script line by line until it breaks:

from fastapi import FastAPI, Response, WebSocket, status
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from fastapi.responses import RedirectResponse

from settings import SERVER_NAME, SERVER_VERSION
from launch_setup import settings
from http_api import HttpApiEndpoint, SettingsRequest
from socket_api import WebsocketApiEndpoint

app = FastAPI()
app.mount("/www", StaticFiles(directory="www"))

app.add_middleware(CORSMiddleware, allow_origins=settings.cors_origins, allow_credentials=True, allow_headers=["*"], allow_methods=["*"])

http_endpoint = HttpApiEndpoint()
socket_endpoint = WebsocketApiEndpoint()

fquirin · 2022-12-22T12:58:42Z

I found something interesting that might be related to this problem on Stackoverflow. The post claims:

In summary, VBox Guest VM is not able to run tensorflow inside the docker-engine while HyperV Guest VM is able to run.

and:

deploying the containers in Synology NAS 920+, I am experiencing the same problem with tensorflow. In this case maybe is normal as this model uses Intel Celeron J4125 which does not support AVX and AVX2 instructions

The SEPIA STT-Server currently does not use Tensorflow, but maybe Numpy or something related has the same issue. Weird thing is that these libraries are not loaded at start immediately but only when the model is used first time 🤔

PlanetSmasher · 2022-12-22T13:02:39Z

It looks like Websocket is the bad guy here

admin@a6b00abb49f1:~/sepia-stt/server$ python3 -m server
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$

admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from socket_api import WebsocketApiEndpoint
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$

fquirin · 2022-12-22T13:08:26Z

ok interesting, I'll dig a bit deeper, brb ^^.

fquirin · 2022-12-22T13:14:50Z

Can you please skip from socket_api import WebsocketApiEndpoint and instead continue with these lines:

from fastapi import WebSocket, WebSocketDisconnect
from starlette.websockets import WebSocketState
from pydantic import ValidationError

from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)

import time
import asyncio

from chunk_processor import ChunkProcessor

and if that breaks at chunk_processor try:

from engine_vosk import VoskProcessor
from engine_coqui import CoquiProcessor

PlanetSmasher · 2022-12-22T13:26:49Z

here it comes

admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from socket_api import WebsocketApiEndpoint
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from fastapi import WebSocket, WebSocketDisconnect
>>> from starlette.websockets import WebSocketState
>>> from pydantic import ValidationError
>>> from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)
>>> import time
>>> import asyncio
>>> from chunk_processor import ChunkProcessor
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$

2nd Machine

on my other testing machine I got it working by setting the hypervisor CPU type to host instead of default
so it can use probably all features

on this second machine the output from the command is:

user@server2:/opt/docker/sepia-stt-server$ sudo docker run --rm --name=sepia-stt2 -it sepia/stt-server:latest
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
SEPIA STT Server - Server running at: 0.0.0.0:20741
SEPIA STT Server - Speech recognition engine: dynamic
SEPIA STT Server - Models defined for engine: 3

fquirin · 2022-12-22T13:53:36Z

Can you do one more try with simply:

from engine_interface import EngineInterface, EngineNotFound
from engine_vosk import VoskProcessor
from engine_coqui import CoquiProcessor

I bet the last line is the deal breaker :-|

on my other testing machine I got it working by setting the hypervisor CPU type to host instead of default
so it can use probably all features

That is very good to know! I'm assuming that is the Proxmox VM configuration?

fquirin · 2022-12-22T13:57:36Z

Can you do one more try with simply:

and even more precise:

from text_processor import TextToNumberProcessor, DateAndTimeOptimizer
import numpy as np
from vosk import Model, SpkModel, KaldiRecognizer, SetLogLevel
from stt import Model

PlanetSmasher · 2022-12-22T14:07:44Z

from engine_interface import EngineInterface, EngineNotFound
from engine_vosk import VoskProcessor
from engine_coqui import CoquiProcessor

I bet the last line is the deal breaker :-|

yep

admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from fastapi import WebSocket, WebSocketDisconnect
>>> from starlette.websockets import WebSocketState
>>> from pydantic import ValidationError
>>>
>>> from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)
>>> import time
>>> import asyncio
>>> from engine_interface import EngineInterface, EngineNotFound
>>> from engine_vosk import VoskProcessor
>>> from engine_coqui import CoquiProcessor
Illegal instruction (core dumped)
admin@a6b00abb49f1:~/sepia-stt/server$

and even more precise:

from text_processor import TextToNumberProcessor, DateAndTimeOptimizer
import numpy as np
from vosk import Model, SpkModel, KaldiRecognizer, SetLogLevel
from stt import Model

admin@a6b00abb49f1:~/sepia-stt/server$ python3
Python 3.7.3 (default, Jan 22 2021, 20:04:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from fastapi import FastAPI, Response, WebSocket, status
>>> from fastapi.middleware.cors import CORSMiddleware
>>> from fastapi.staticfiles import StaticFiles
>>> from fastapi.responses import RedirectResponse
>>> from settings import SERVER_NAME, SERVER_VERSION
>>> from launch_setup import settings
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
>>> from http_api import HttpApiEndpoint, SettingsRequest
>>> from fastapi import WebSocket, WebSocketDisconnect
>>> from starlette.websockets import WebSocketState
>>> from pydantic import ValidationError
>>> from socket_messages import (SocketJsonInputMessage, SocketMessage, SocketPingMessage, SocketWelcomeMessage, SocketBroadcastMessage, SocketErrorMessage)
>>> import time
>>> import asyncio
>>> from text_processor import TextToNumberProcessor, DateAndTimeOptimizer
>>> import numpy as np
>>> from vosk import Model, SpkModel, KaldiRecognizer, SetLogLevel
>>> from stt import Model
Illegal instruction (core dumped)

and Yes it is a Proxmox VM

fquirin · 2022-12-22T14:14:54Z

Ok, thanks a lot for running through all of this! 😎

The silver lining is that we could theoretically kick Coqui engine and it should keep running.
In your server config (server.conf) can you change asr_engine=dynamic to asr_engine=vosk and try python3 -m launch again :-)

PlanetSmasher · 2022-12-22T16:09:07Z

that does the trick aswell

admin@a6b00abb49f1:~/sepia-stt/server$ python3 -m launch
SEPIA STT Server - Settings file used: './server.conf'
SEPIA STT Server - Settings file tag: 'Dynamic server settings'
SEPIA STT Server - Starting...
SEPIA STT Server - Server running at: 0.0.0.0:20741
SEPIA STT Server - Speech recognition engine: vosk
SEPIA STT Server - Models defined for engine: 2
INFO:     Started server process [81]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:20741 (Press CTRL+C to quit)

royrogermcfreely · 2022-12-23T08:40:35Z

sorry for the late replay.

thanks to @PlanetSmasher - after i changed my proxmox vm cpu from "default" to "host", i was able to start the stt server with the default run.sh script without changing anything.

so i didnt execute the script line by line as mentiont above.

/roy

vanseforge · 2022-12-23T18:34:19Z

Ok, thanks a lot for running through all of this! 😎

The silver lining is that we could theoretically kick Coqui engine and it should keep running. In your server config (server.conf) can you change asr_engine=dynamic to asr_engine=vosk and try python3 -m launch again :-)

That worked for me also. Thank you @fquirin and @PlanetSmasher .

@PlanetSmasher in your docker-compose example you wrote a ":" instead a "=" in the environment section. Maybe you can edit this if somebody copies your example.
environment: - SEPIA_STT_SETTINGS:/home/admin/sepia-stt/share/my.conf
insted of
environment: - SEPIA_STT_SETTINGS=/home/admin/sepia-stt/share/my.conf

I copied it and wondered why the wrong file was loaded 🫣

fquirin added bug Something isn't working Need more info Further information is requested issue with OS or platform Issues related to configuration of OS or underlying platform labels Dec 22, 2022

fquirin mentioned this issue Dec 22, 2022

Bug: Illegal instruction (core dumped) , coqui-ai/STT#2225

Closed

fquirin changed the title ~~install new stt server error~~ install new stt server error (on Celeron CPU or certain VMs) Jan 6, 2023

fquirin removed the Need more info Further information is requested label Jan 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

install new stt server error (on Celeron CPU or certain VMs) #218

install new stt server error (on Celeron CPU or certain VMs) #218

royrogermcfreely commented Dec 10, 2022

fquirin commented Dec 10, 2022

vanseforge commented Dec 20, 2022

PlanetSmasher commented Dec 21, 2022

fquirin commented Dec 21, 2022 •

edited

PlanetSmasher commented Dec 21, 2022 •

edited

fquirin commented Dec 21, 2022

PlanetSmasher commented Dec 21, 2022

fquirin commented Dec 22, 2022 •

edited

fquirin commented Dec 22, 2022

PlanetSmasher commented Dec 22, 2022 •

edited

fquirin commented Dec 22, 2022

fquirin commented Dec 22, 2022 •

edited

PlanetSmasher commented Dec 22, 2022 •

edited

fquirin commented Dec 22, 2022

fquirin commented Dec 22, 2022

PlanetSmasher commented Dec 22, 2022 •

edited

fquirin commented Dec 22, 2022

PlanetSmasher commented Dec 22, 2022

royrogermcfreely commented Dec 23, 2022

vanseforge commented Dec 23, 2022

install new stt server error (on Celeron CPU or certain VMs) #218

install new stt server error (on Celeron CPU or certain VMs) #218

Comments

royrogermcfreely commented Dec 10, 2022

fquirin commented Dec 10, 2022

vanseforge commented Dec 20, 2022

PlanetSmasher commented Dec 21, 2022

fquirin commented Dec 21, 2022 • edited

PlanetSmasher commented Dec 21, 2022 • edited

fquirin commented Dec 21, 2022

PlanetSmasher commented Dec 21, 2022

fquirin commented Dec 22, 2022 • edited

fquirin commented Dec 22, 2022

PlanetSmasher commented Dec 22, 2022 • edited

fquirin commented Dec 22, 2022

fquirin commented Dec 22, 2022 • edited

PlanetSmasher commented Dec 22, 2022 • edited

2nd Machine

fquirin commented Dec 22, 2022

fquirin commented Dec 22, 2022

PlanetSmasher commented Dec 22, 2022 • edited

fquirin commented Dec 22, 2022

PlanetSmasher commented Dec 22, 2022

royrogermcfreely commented Dec 23, 2022

vanseforge commented Dec 23, 2022

fquirin commented Dec 21, 2022 •

edited

PlanetSmasher commented Dec 21, 2022 •

edited

fquirin commented Dec 22, 2022 •

edited

PlanetSmasher commented Dec 22, 2022 •

edited

fquirin commented Dec 22, 2022 •

edited

PlanetSmasher commented Dec 22, 2022 •

edited

PlanetSmasher commented Dec 22, 2022 •

edited