unable to start default pelias docker due to "Elasticsearch service did not come up, check configuration" error

[wkulesza]

Describe the bug

Unable to start default pelias docker with default portland-metro project due to "Elasticsearch service did not come up, check configuration" error

Steps to Reproduce

1. on a clear machine, logged as a sudo user i created a sh script taken from https://github.com/pelias/docker documentation
2. git cloned my fork of repo (fork only differs from the original by adding a new region called wlkp - for later tests)
3. edited the path/to/install to /home/{myname}/pelias folder that i created manually
4. chmoded +x that sh script
5. run the ./pelias.sh script
6. during the process, i saw following errors:
   Creating pelias _elasticsearch .. done

• pelias elastic wait
  waiting for elasticsearch service to come up
  ...........................

Elasticsearch service did not come up, check configuration

• pelias elastic create
  Creating pelias_schema_run ... done

---

Create index

Elasticsearch ERROR: 2020-09-15T11:53:21Z
Error: Request error, retrying
GET http://elasticsearch:9200/ => connect ECONNREFUSED 172.19.0.2:9200
[..]

Elasticsearch WARNING: 2020-09-15T11:53:21Z
Unable to revive connection http://elasticsearch:9200/

Elasticsearch WARNING: 2020-09-15T11:53:21Z
No living connections

[...]

unsupported elasticsearch version. try: >=7.4.2

• pelias download all

...
all next log items seem to be ok apart from tests.

Expected behavior
Elasticsearch should load and website api should return correct results?

Environment (please complete the following information):
Ubuntu Ubuntu 20.04 LTS Server 64bit
Docker version 19.03.12, build 48a66213fe
docker-compose version 1.27.2, build 18f557f9

Pastebin/Screenshots
Api works and returns data, but incomplete
[http://prntscr.com/uhnaf4]

References

As requested by @missinglink in issue #180 180 adding this a separate issue.

[missinglink]

Can you please post the output of these commands:

sudo netstat -plnt

docker ps -a | grep pelias

[wkulesza]

Output of netstat:

root@vps671012:/home/wojciech# sudo netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      578/systemd-resolve
tcp        0      0 0.0.0.0:8822            0.0.0.0:*               LISTEN      1545/sshd: /usr/sbi
tcp6       0      0 :::4000                 :::*                    LISTEN      56538/docker-proxy
tcp6       0      0 :::4100                 :::*                    LISTEN      56399/docker-proxy
tcp6       0      0 :::4200                 :::*                    LISTEN      56493/docker-proxy
tcp6       0      0 :::4300                 :::*                    LISTEN      56319/docker-proxy
tcp6       0      0 :::4400                 :::*                    LISTEN      56464/docker-proxy
tcp6       0      0 :::8822                 :::*                    LISTEN      1545/sshd: /usr/sbi

Output of docker ps:

root@vps671012:/home/wojciech# docker ps -a | grep pelias
2aa69951da29        pelias/schema:master          "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_schema
c764577b5a9c        pelias/polylines:master       "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_polylines
8e15a3d4af22        pelias/libpostal-service      "./bin/wof-libpostal…"   About an hour ago   Up About an hour                0.0.0.0:4400->4400/tcp   pelias_libpostal
c9c1394b0664        pelias/pip-service:master     "./bin/start"            About an hour ago   Up About an hour                0.0.0.0:4200->4200/tcp   pelias_pip-service
433a77fb2c90        pelias/openstreetmap:master   "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_openstreetmap
743fda531b72        pelias/fuzzy-tester:master    "./bin/fuzzy-tester …"   About an hour ago   Exited (0) About an hour ago                             pelias_fuzzy_tester
3559566ff397        pelias/transit:master         "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_transit
ae747c8a14d5        pelias/interpolation:master   "./interpolate serve…"   About an hour ago   Up About an hour                0.0.0.0:4300->4300/tcp   pelias_interpolation
64b4d8352418        pelias/whosonfirst:master     "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_whosonfirst
dbb7277d9a7e        pelias/openaddresses:master   "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_openaddresses
697aed3f1cf9        pelias/csv-importer:master    "/bin/bash"              About an hour ago   Exited (0) About an hour ago                             pelias_csv_importer
e5f79125dffc        pelias/placeholder:master     "./cmd/server.sh"        About an hour ago   Up About an hour                0.0.0.0:4100->4100/tcp   pelias_placeholder
f6fd61fdcb92        pelias/api:master             "./bin/start"            About an hour ago   Up About an hour                0.0.0.0:4000->4000/tcp   pelias_api
1b2f7495e50f        pelias/elasticsearch:7.5.1    "/usr/local/bin/dock…"   2 hours ago         Restarting (1) 27 seconds ago                            pelias_elasticsearch

[orangejulius]

Hi @wkulesza,

The output of pelias compose logs elasticsearch will likely provide some hints.

The output can be quite verbose so can you please paste it all here and be sure it's formatted correctly? Your copy/paste above appears to have lost quite a few newlines making it hard to read. For example on my machine, docker ps -a looks like this:

$ docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                                            NAMES
cd45ae5ee4e5        pelias/elasticsearch:7.5.1       "/usr/local/bin/dock…"   4 days ago          Up 4 days           0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   pelias_elasticsearch
71e836aaa83e        pelias/api:master                "./bin/start"            4 days ago          Up 4 days           0.0.0.0:4000->4000/tcp                           pelias_api
21b8f752c75b        pelias/libpostal-service         "./bin/wof-libpostal…"   4 days ago          Up 4 days           0.0.0.0:4400->4400/tcp                           pelias_libpostal
853b761bbbbc        pelias/pip-service:master        "./bin/start"            4 days ago          Up 4 days           0.0.0.0:4200->4200/tcp                           pelias_pip-service
cdb2be7d7d9c        pelias/interpolation:master      "./interpolate serve…"   4 days ago          Up 4 days           0.0.0.0:4300->4300/tcp                           pelias_interpolation
e70ce6d0efbe        pelias/placeholder:master        "./cmd/server.sh"        4 days ago          Up 4 days           0.0.0.0:4100->4100/tcp                           pelias_placeholder

[missinglink]

I fixed the formatting above, the trick for markdown is to paste it between a pair of triple backticks

[wkulesza]

Thanks @missinglink for helping with formatting.

Pasting in the elasticsearch log - obviously is very long, but i think this part is repeating towards the end:

Attaching to pelias_elasticsearch
�[36melasticsearch_1  |�[0m OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
�[36melasticsearch_1  |�[0m {"type": "server", "timestamp": "2020-09-15T11:52:49,011Z", "level": "WARN", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "pelias-dev", "node.name": "1b2f7495e50f", "message": "uncaught exception in thread [main]", 
�[36melasticsearch_1  |�[0m "stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "Caused by: org.elasticsearch.ElasticsearchException: failed to bind service",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.node.Node.<init>(Node.java:607) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.node.Node.<init>(Node.java:253) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "... 6 more",
�[36melasticsearch_1  |�[0m "Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes",
�[36melasticsearch_1  |�[0m "at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]",
�[36melasticsearch_1  |�[0m "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]",
�[36melasticsearch_1  |�[0m "at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]",
�[36melasticsearch_1  |�[0m "at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]",
�[36melasticsearch_1  |�[0m "at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]",
�[36melasticsearch_1  |�[0m "at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]",
�[36melasticsearch_1  |�[0m "at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:272) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:209) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.node.Node.<init>(Node.java:273) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.node.Node.<init>(Node.java:253) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.5.1.jar:7.5.1]",
�[36melasticsearch_1  |�[0m "... 6 more"] }

[missinglink]

Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes"

This directory should be available on your host machine at ${DATA_DIR}/elasticsearch
(with the DATA_DIR being defined in the .env file, as per the docs).

Is there anything unusual about that directory? ie. permissions or network mount etc?
Who is the owner of that directory?

[missinglink]

Possibly related:

• Elasticsearch 5 won't start if data directory is owned by root, or not created #33
• Elasticsearch did not come up, check configuration concerning node #118
• https://stackoverflow.com/questions/51201699/elasticsearch-docker-image-with-data-persistence

I suspect the issue is related to user permissions.

[wkulesza]

@missinglink DATA DIR is set in portland-metro to ./data so i understand this directory is /home/wojciech/pelias/docker/projects/portland-metro/data/elasticsearch ? and not usr/share/elasticsearch ?

that elasticsearch in home/wojciech is empty
all folders are root/root, as this is how i run all projects and UID is 1001

[orangejulius]

@wkulesza the ownership of the directory is likely the issue, it can't be owned by root.

This is a common problem, and we were working on some better automatic detection/fixing of this issue in #74, maybe we will pick that up.

For now I happen to have just listed some basic troubleshooting tips in another comment, let me know if any of these work for you: #31 (comment)

[wkulesza]

@orangejulius will work on it, but for the time being, when i try to run it as normal user (without sudo) i get permission errors for creating symbolic link (which is normal, as standard user can't work on /usr/local/bin directory)

wojciech@vps671012:~$ mkdir pelias
wojciech@vps671012:~$
wojciech@vps671012:~$
wojciech@vps671012:~$ ./pelias.sh
+ cd /home/wojciech/pelias
+ git clone https://github.com/wkulesza/docker.git
Cloning into 'docker'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 1637 (delta 4), reused 11 (delta 4), pack-reused 1624
Receiving objects: 100% (1637/1637), 435.28 KiB | 1.32 MiB/s, done.
Resolving deltas: 100% (1046/1046), done.
+ cd docker
++ pwd
+ ln -s /home/wojciech/pelias/docker/pelias /usr/local/bin/pelias
ln: failed to create symbolic link '/usr/local/bin/pelias': Permission denied
+ cd projects/portland-metro
+ mkdir ./data
+ sed -i /DATA_DIR/d .env
+ echo DATA_DIR=./data
+ sed -i /DOCKER_USER/d .env
++ id -u
+ echo DOCKER_USER=1001
+ pelias compose pull
./pelias.sh: line 30: pelias: command not found
+ pelias elastic start
./pelias.sh: line 31: pelias: command not found
+ pelias elastic wait
./pelias.sh: line 32: pelias: command not found
+ pelias elastic create
./pelias.sh: line 33: pelias: command not found
+ pelias download all
./pelias.sh: line 34: pelias: command not found
+ pelias prepare all
./pelias.sh: line 35: pelias: command not found
+ pelias import all
./pelias.sh: line 36: pelias: command not found
+ pelias compose up
./pelias.sh: line 37: pelias: command not found
+ pelias test run
./pelias.sh: line 40: pelias: command not found

[orangejulius]

Right. Only the ln command should be run with sudo. I've just opened #216 to clarify that in the docs.

[wkulesza]

OK. i made that manually now, added ln command as sudo
then when i try to continue and being in the projects/portland-metro folder i get this:

wojciech@vps671012:~/pelias/docker/projects/portland-metro$ pelias compose pull
pelias: command not found

[orangejulius]

Is /usr/local/bin part of your $PATH? We tried to pick a common one but I doubt there are any 100% reliable choices. We have some troubleshooting tips in the README: https://github.com/pelias/docker/#resolving-path-issues

[wkulesza]

@orangejulius yes, /usr/local/bin is part of my PATH, i checked this.
This is really strange now... as a normal user, i can't go forward at all. this installation only runs as sudo user.

I also tried to manually set the ln first (as sudo user) and then perform the pelias.sh script with that ln line commented out.
also stops at those pelias commands not being found...

even after going sudo, i get no response from pelias command:

root@vps671012:/home/wojciech/pelias/docker# ls -l
total 48
-rw-r--r--  1 root root  1086 Sep 15 18:15 LICENSE
-rw-r--r--  1 root root 17948 Sep 15 18:15 README.md
drwxr-xr-x  2 root root  4096 Sep 15 18:15 cmd
drwxr-xr-x  3 root root  4096 Sep 15 18:15 common
drwxr-xr-x  4 root root  4096 Sep 15 18:15 images
drwxr-xr-x  2 root root  4096 Sep 15 18:15 lib
-rwxr-xr-x  1 root root   728 Sep 15 18:15 pelias
drwxr-xr-x 18 root root  4096 Sep 15 18:15 projects
root@vps671012:/home/wojciech/pelias/docker# pelias
pelias: command not found
root@vps671012:/home/wojciech/pelias/docker# which pelias
root@vps671012:/home/wojciech/pelias/docker#

[wkulesza]

Just to add some notes,
after i deleted /usr/local/bin/pelias ln, I was able to start the process as root, but obviously couldn't get past elasticsearch service coming up.

i was now trying to run pelias compose pull as normal user but got a lot of permission denied (all folders in /home/wojciech/pelias are owned by that user (wojciech)

wojciech@vps671012:~/pelias/docker/projects/wlkp$ pelias compose pull
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "docker/transport/unixconn.py", line 43, in connect
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 727, in urlopen
  File "urllib3/util/retry.py", line 403, in increment
  File "urllib3/packages/six.py", line 734, in reraise
  File "urllib3/connectionpool.py", line 677, in urlopen
  File "urllib3/connectionpool.py", line 392, in _make_request
  File "http/client.py", line 1252, in request
  File "http/client.py", line 1298, in _send_request
  File "http/client.py", line 1247, in endheaders
  File "http/client.py", line 1026, in _send_output
  File "http/client.py", line 966, in send
  File "docker/transport/unixconn.py", line 43, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker/api/client.py", line 205, in _retrieve_server_version
  File "docker/api/daemon.py", line 181, in version
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 228, in _get
  File "requests/sessions.py", line 543, in get
  File "requests/sessions.py", line 530, in request
  File "requests/sessions.py", line 643, in send
  File "requests/adapters.py", line 498, in send
requests.exceptions.ConnectionError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 3, in <module>
  File "compose/cli/main.py", line 67, in main
  File "compose/cli/main.py", line 123, in perform_command
  File "compose/cli/command.py", line 69, in project_from_options
  File "compose/cli/command.py", line 132, in get_project
  File "compose/cli/docker_client.py", line 43, in get_client
  File "compose/cli/docker_client.py", line 170, in docker_client
  File "docker/api/client.py", line 188, in __init__
  File "docker/api/client.py", line 213, in _retrieve_server_version
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))
[313307] Failed to execute script docker-compose

[wkulesza]

I don’t know if I should provide more details but as root I went in to Wlkp projects folder and was able to perform all pelias commands including pelias elastic wait - which finished with success!

But running instance doesn’t return addresses to inquiries.

[wkulesza]

From my last observatio0n @missinglink @orangejulius docker with elasticsearch is constantly restarting - once every 1 minute it's going up, then after 1-3 secs it's restarting again.
the DATA-DIR has elasticsearch and this is root:root owner with permissions set to 755.
My machine is 12GB of ram, so it should be enough and 2 cores and htop shows everytime elasticsearch docker tries to start a lot CPU usage for those processes:
http://prntscr.com/uj4cbe

[missinglink]

The directory is owned by root and 755 then no user other than root can write to it.

[wkulesza]

@missinglink Hi, thanx for pointers. I chmoded portland-metro to 777 and this allowed me to go forward (elasticsearch is up) but stumbled upon this upon pelias elastic create for openaddresses:

http://prntscr.com/ujbp43

Error: dir option must be relative to "/tmp", found "data/openaddresses".

I have not set tmp as a data folder, but projects/portland-metro/data/

Any pointers ?

[missinglink]

Agh nice catch, this is a bug introduced last week by our dependency management bot.

I've reverted that commit in pelias/openaddresses#473 which has automatically published a new docker image.

You can pull the new image with pelias compose pull and try again.

I'm going to have to go through the other repositories and check them for this same error, so let me know if you see it again.

[gabmartini]

Same problem here, not solved with latest docker images. Fedora Server 32.

[orangejulius]

Hi folks, we've just merged some improvements in #220 that should help with this sort of problem. The pelias helper tool will now require you to run it as a non-root user, which we suspect was one of the major causes of issues.

If you get a chance please try things out with the latest code. We recommend removing your data directory completely to ensure all permissions are set properly going forward.

[wkulesza]

Just to follow up, I successfully run through this updated version without having to be root (just needed to provide sudo pasword for ln command), but ended up with pelias test run that provided me with this error:
"connect ECONNREFUSED 172.19.0.4:4400"

my docker ps says everythjing is running and I haven't noticed any errors, when following the standard tutorial:

curl also says: connect econnrefused

my docker ps says:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                            NAMES
a68729566a41        pelias/interpolation:master   "./interpolate serve…"   9 minutes ago       Up 6 seconds        0.0.0.0:4300->4300/tcp                           pelias_interpolation
adab4927b931        pelias/libpostal-service      "./bin/wof-libpostal…"   9 minutes ago       Up 18 seconds       0.0.0.0:4400->4400/tcp                           pelias_libpostal
d131162e656c        pelias/pip-service:master     "./bin/start"            9 minutes ago       Up 9 minutes        0.0.0.0:4200->4200/tcp                           pelias_pip-service
bec04e8013e3        pelias/api:master             "./bin/start"            9 minutes ago       Up 9 minutes        0.0.0.0:4000->4000/tcp                           pelias_api
324017f2b963        pelias/placeholder:master     "./cmd/server.sh"        9 minutes ago       Up 9 minutes        0.0.0.0:4100->4100/tcp                           pelias_placeholder
c859312823bd        pelias/elasticsearch:7.5.1    "/usr/local/bin/dock…"   3 hours ago         Up 3 hours          0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   pelias_elasticsearch

edit1: sorry, tagging @missinglink and @orangejulius - i didn't want to open a new issue, as this is a continuation. and also, i tried with previous project (only one region in poland), just in case, 8gb of ram wouldn't be enough for "poland" project, but ended up with the very same error connect ECONNREFUSED 172.22.0.4:4400

edit2: i noticed that libpostal and interpolation do restart according to status of docker ps:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                            NAMES
2718fbe90882        pelias/placeholder:master     "./cmd/server.sh"        About an hour ago   Up About an hour    0.0.0.0:4100->4100/tcp                           pelias_placeholder
484fc4f05845        pelias/libpostal-service      "./bin/wof-libpostal…"   About an hour ago   Up 26 seconds       0.0.0.0:4400->4400/tcp                           pelias_libpostal
ac56573d669c        pelias/interpolation:master   "./interpolate serve…"   About an hour ago   Up 9 seconds        0.0.0.0:4300->4300/tcp                           pelias_interpolation
0b232c1180a2        pelias/api:master             "./bin/start"            About an hour ago   Up About an hour    0.0.0.0:4000->4000/tcp                           pelias_api
53e9776004f5        pelias/pip-service:master     "./bin/start"            About an hour ago   Up About an hour    0.0.0.0:4200->4200/tcp                           pelias_pip-service
763e13ed52a0        pelias/elasticsearch:7.5.1    "/usr/local/bin/dock…"   2 hours ago         Up 2 hours          0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   pelias_elasticsearch

edit3:
pelias compose logs shows cannot allocate memory when it comes to interpolation. Is running a single region really require more than 8gb ?

interpolation_1  | ERR   Error loading address parser module, dir=(null)
interpolation_1  |    at libpostal_setup_parser_datadir (libpostal.c:434) errno: Cannot allocate memory

[missinglink]

Hi @wkulesza libpostal alone needs ~2GB of RAM to load its models.

It's used by the libpostal-service and also included in interpolation, so in effect it's being loaded twice == 4GB+.
The models it loads are for the whole world, so there's no easy way of reducing that memory requirement if you're running a smaller geography.

The "Cannot allocate memory" error seems to indicate that your machine doesn't have adequate memory to run Pelias, our documentation seems to indicate that 8GB is sufficient but that doesn't seem to be correct.

Adding RAM would be a simple solution, otherwise you can run Pelias without the interpolation service, although 8GB will likely not yield great performance from elasticsearch which also functions much better with memory available for the kernel page cache.

There's actually a long standing PR to make interpolation talk to the libpostal-service webservice directly, but it's not been merged and I forget the reasons why: pelias/interpolation#249, if that were merged then libpostal would only be loaded once.