Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

coco-annotator docker container loops while booting up ==> [ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out. #610

Open
ovalerio opened this issue Jul 17, 2023 · 2 comments
Open

coco-annotator docker container loops while booting up ==> [ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out. #610

ovalerio opened this issue Jul 17, 2023 · 2 comments

Comments

@ovalerio
Copy link

Dear @jsbroks,

I am following the tutorial you shared in YouTube about installing & running coco-annotator using Docker. I am running in Ubuntu and I installed a fresh version of docker and docker compose.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal

$ docker -v
Docker version 24.0.4, build 3713ee1

$ docker-compose -v
Docker Compose version v2.17.2

This is what I got when I attempt to start coco-annotator:

valerio@sco3002:~/coco-annotator (master)$ docker-compose up
[+] Running 44/44
 ✔ database 12 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                 57.1s 
   ✔ 58690f9b18fc Pull complete                                                                                                             44.5s 
   ✔ b51569e7c507 Pull complete                                                                                                             44.5s 
   ✔ da8ef40b9eca Pull complete                                                                                                             44.5s 
   ✔ fb15d46c38dc Pull complete                                                                                                             44.5s 
   ✔ a0dc15b16822 Pull complete                                                                                                             44.5s 
   ✔ b7a3e92f19af Pull complete                                                                                                             44.6s 
   ✔ ed4a7b863fa1 Pull complete                                                                                                             44.6s 
   ✔ a58b030ea8e4 Pull complete                                                                                                             44.6s 
   ✔ 6aa1ba699846 Pull complete                                                                                                             44.6s 
   ✔ ebc52c729dca Pull complete                                                                                                             44.7s 
   ✔ 52e8c440d4d6 Pull complete                                                                                                             55.2s 
   ✔ 22b97876323d Pull complete                                                                                                             55.2s 
 ✔ workers 15 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                               52.5s 
   ✔ e79bb959ec00 Pull complete                                                                                                              5.1s 
   ✔ d4b7902036fe Pull complete                                                                                                              5.3s 
   ✔ 1b2a72d4e030 Pull complete                                                                                                              5.4s 
   ✔ d54db43011fd Pull complete                                                                                                              7.7s 
   ✔ 69d473365bb3 Pull complete                                                                                                             35.1s 
   ✔ 35b4aebab92b Pull complete                                                                                                             35.2s 
   ✔ 60ddbccc0359 Pull complete                                                                                                             35.6s 
   ✔ 0b3c337d0c3e Pull complete                                                                                                             35.6s 
   ✔ 2b75f9fe405b Pull complete                                                                                                             35.7s 
   ✔ 8afafd04385f Pull complete                                                                                                             35.7s 
   ✔ c2da40f3dd21 Pull complete                                                                                                             35.7s 
   ✔ 61f4d37d1625 Pull complete                                                                                                             43.7s 
   ✔ 557b33a4cdcb Pull complete                                                                                                             50.7s 
   ✔ 8e4e2338390e Pull complete                                                                                                             50.9s 
   ✔ af9511f0c320 Pull complete                                                                                                             50.9s 
 ✔ messageq 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                   54.2s 
   ✔ 9d19ee268e0d Pull complete                                                                                                             48.1s 
   ✔ 682a058fc094 Pull complete                                                                                                             48.1s 
   ✔ c6f199470454 Pull complete                                                                                                             48.1s 
   ✔ e588fdd2bc14 Pull complete                                                                                                             51.5s 
   ✔ b2df8cdb3812 Pull complete                                                                                                             51.5s 
   ✔ eec8ffe01a78 Pull complete                                                                                                             52.1s 
   ✔ ef7e4d422ae7 Pull complete                                                                                                             52.1s 
   ✔ a0337bb83298 Pull complete                                                                                                             52.1s 
   ✔ 79a8a5649e7c Pull complete                                                                                                             52.1s 
   ✔ 15c297f005d6 Pull complete                                                                                                             52.3s 
 ✔ webserver 3 layers [⣿⣿⣿]      0B/0B      Pulled                                                                                          52.5s 
   ✔ b7da3f88928c Pull complete                                                                                                             50.9s 
   ✔ 7adcd72abda2 Pull complete                                                                                                             50.9s 
   ✔ f45d8c5057fd Pull complete                                                                                                             50.9s 
[+] Running 6/4
 ✔ Network coco-annotator_cocoannotator  Created                                                                                             0.0s 
 ✔ Volume "coco-annotator_mongodb_data"  Created                                                                                             0.0s 
 ✔ Container annotator_mongodb           Created                                                                                             0.2s 
 ✔ Container annotator_message_q         Created                                                                                             0.2s 
 ✔ Container annotator_workers           Created                                                                                             0.0s 
 ✔ Container annotator_webclient         Created                                                                                             0.0s 
Attaching to annotator_message_q, annotator_mongodb, annotator_webclient, annotator_workers
annotator_workers    | fatal: Not a git repository (or any of the parent directories): .git
annotator_webclient  | [2023-07-17 08:29:26 +0000] [7] [DEBUG] Current configuration:
annotator_webclient  |   config: webserver/gunicorn_config.py
annotator_webclient  |   bind: ['0.0.0.0:5000']
annotator_webclient  |   backlog: 2048
annotator_webclient  |   workers: 1
annotator_webclient  |   worker_class: eventlet
annotator_webclient  |   threads: 1
annotator_webclient  |   worker_connections: 1000
annotator_webclient  |   max_requests: 0
annotator_webclient  |   max_requests_jitter: 0
annotator_webclient  |   timeout: 180
annotator_webclient  |   graceful_timeout: 30
annotator_webclient  |   keepalive: 2
annotator_webclient  |   limit_request_line: 4094
annotator_webclient  |   limit_request_fields: 100
annotator_webclient  |   limit_request_field_size: 8190
annotator_webclient  |   reload: False
annotator_webclient  |   reload_engine: auto
annotator_webclient  |   reload_extra_files: []
annotator_webclient  |   spew: False
annotator_webclient  |   check_config: False
annotator_webclient  |   preload_app: False
annotator_webclient  |   sendfile: False
annotator_webclient  |   reuse_port: False
annotator_webclient  |   chdir: /workspace
annotator_webclient  |   daemon: False
annotator_webclient  |   raw_env: []
annotator_webclient  |   pidfile: None
annotator_webclient  |   worker_tmp_dir: None
annotator_webclient  |   user: 0
annotator_webclient  |   group: 0
annotator_webclient  |   umask: 0
annotator_webclient  |   initgroups: False
annotator_webclient  |   tmp_upload_dir: None
annotator_webclient  |   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
annotator_webclient  |   forwarded_allow_ips: ['127.0.0.1']
annotator_webclient  |   accesslog: None
annotator_webclient  |   disable_redirect_access_to_syslog: False
annotator_webclient  |   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
annotator_webclient  |   errorlog: -
annotator_webclient  |   loglevel: debug
annotator_webclient  |   capture_output: False
annotator_webclient  |   logger_class: gunicorn.glogging.Logger
annotator_webclient  |   logconfig: None
annotator_webclient  |   logconfig_dict: {}
annotator_webclient  |   syslog_addr: udp://localhost:514
annotator_webclient  |   syslog: False
annotator_webclient  |   syslog_prefix: None
annotator_webclient  |   syslog_facility: user
annotator_webclient  |   enable_stdio_inheritance: False
annotator_webclient  |   statsd_host: None
annotator_webclient  |   statsd_prefix: 
annotator_webclient  |   proc_name: None
annotator_webclient  |   default_proc_name: webserver:app
annotator_webclient  |   pythonpath: None
annotator_webclient  |   paste: None
annotator_webclient  |   on_starting: <function OnStarting.on_starting at 0x7fab207582f0>
annotator_webclient  |   on_reload: <function OnReload.on_reload at 0x7fab20758400>
annotator_webclient  |   when_ready: <function WhenReady.when_ready at 0x7fab20758510>
annotator_webclient  |   pre_fork: <function Prefork.pre_fork at 0x7fab20758620>
annotator_webclient  |   post_fork: <function Postfork.post_fork at 0x7fab20758730>
annotator_webclient  |   post_worker_init: <function PostWorkerInit.post_worker_init at 0x7fab20758840>
annotator_webclient  |   worker_int: <function WorkerInt.worker_int at 0x7fab20758950>
annotator_webclient  |   worker_abort: <function WorkerAbort.worker_abort at 0x7fab20758a60>
annotator_webclient  |   pre_exec: <function PreExec.pre_exec at 0x7fab20758b70>
annotator_webclient  |   pre_request: <function PreRequest.pre_request at 0x7fab20758c80>
annotator_webclient  |   post_request: <function PostRequest.post_request at 0x7fab20758d08>
annotator_webclient  |   child_exit: <function ChildExit.child_exit at 0x7fab20758e18>
annotator_webclient  |   worker_exit: <function WorkerExit.worker_exit at 0x7fab20758f28>
annotator_webclient  |   nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7fab204d40d0>
annotator_webclient  |   on_exit: <function OnExit.on_exit at 0x7fab204d41e0>
annotator_webclient  |   proxy_protocol: False
annotator_webclient  |   proxy_allow_ips: ['127.0.0.1']
annotator_webclient  |   keyfile: None
annotator_webclient  |   certfile: None
annotator_webclient  |   ssl_version: 2
annotator_webclient  |   cert_reqs: 0
annotator_webclient  |   ca_certs: None
annotator_webclient  |   suppress_ragged_eofs: True
annotator_webclient  |   do_handshake_on_connect: False
annotator_webclient  |   ciphers: TLSv1
annotator_webclient  |   raw_paste_global_conf: []
annotator_webclient  | [2023-07-17 08:29:26 +0000] [7] [INFO] Starting gunicorn 19.9.0
annotator_webclient  | [2023-07-17 08:29:26 +0000] [7] [DEBUG] Arbiter booted
annotator_webclient  | [2023-07-17 08:29:26 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
annotator_webclient  | [2023-07-17 08:29:26 +0000] [7] [INFO] Using worker: eventlet
annotator_webclient  | [2023-07-17 08:29:26 +0000] [11] [INFO] Booting worker with pid: 11
annotator_webclient  | [2023-07-17 08:29:26 +0000] [7] [DEBUG] 1 workers
annotator_webclient  | [2023-07-17 08:29:27 +0000] [11] [WARNING] MaskRCNN model is disabled.
annotator_webclient  | [2023-07-17 08:29:27 +0000] [11] [WARNING] DEXTR model is disabled.
annotator_message_q  | 2023-07-17 08:29:28.378703+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
annotator_message_q  | 2023-07-17 08:29:28.380955+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
annotator_message_q  | 2023-07-17 08:29:28.381310+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE
annotator_message_q  | 2023-07-17 08:29:28.479181+00:00 [info] <0.230.0> ra: starting system quorum_queues
annotator_message_q  | 2023-07-17 08:29:28.479253+00:00 [info] <0.230.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@29539ca56422/quorum/rabbit@29539ca56422
annotator_message_q  | 2023-07-17 08:29:28.505732+00:00 [info] <0.261.0> ra system 'quorum_queues' running pre init for 0 registered servers
annotator_message_q  | 2023-07-17 08:29:28.510774+00:00 [info] <0.262.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
annotator_message_q  | 2023-07-17 08:29:28.516680+00:00 [notice] <0.267.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
annotator_message_q  | 2023-07-17 08:29:28.519773+00:00 [info] <0.230.0> ra: starting system coordination
annotator_message_q  | 2023-07-17 08:29:28.519812+00:00 [info] <0.230.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@29539ca56422/coordination/rabbit@29539ca56422
annotator_message_q  | 2023-07-17 08:29:28.520359+00:00 [info] <0.274.0> ra system 'coordination' running pre init for 0 registered servers
annotator_message_q  | 2023-07-17 08:29:28.520681+00:00 [info] <0.275.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
annotator_message_q  | 2023-07-17 08:29:28.520797+00:00 [notice] <0.280.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
annotator_message_q  | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0> 
annotator_message_q  | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0>  Starting RabbitMQ 3.12.1 on Erlang 25.3.2.3 [jit]
annotator_message_q  | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0>  Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
annotator_message_q  | 2023-07-17 08:29:28.522025+00:00 [info] <0.230.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
annotator_message_q  | 
annotator_message_q  |   ##  ##      RabbitMQ 3.12.1
annotator_message_q  |   ##  ##
annotator_message_q  |   ##########  Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
annotator_message_q  |   ######  ##
annotator_message_q  |   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
annotator_message_q  | 
annotator_message_q  |   Erlang:      25.3.2.3 [jit]
annotator_message_q  |   TLS Library: OpenSSL - OpenSSL 3.1.1 30 May 2023
annotator_message_q  |   Release series support status: supported
annotator_message_q  | 
annotator_message_q  |   Doc guides:  https://rabbitmq.com/documentation.html
annotator_message_q  |   Support:     https://rabbitmq.com/contact.html
annotator_message_q  |   Tutorials:   https://rabbitmq.com/getstarted.html
annotator_message_q  |   Monitoring:  https://rabbitmq.com/monitoring.html
annotator_message_q  | 
annotator_message_q  |   Logs: <stdout>
annotator_message_q  | 
annotator_message_q  |   Config file(s): /etc/rabbitmq/conf.d/10-defaults.conf
annotator_message_q  |                   /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf
annotator_message_q  | 
annotator_message_q  |   Starting broker...2023-07-17 08:29:28.526716+00:00 [info] <0.230.0> 
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>  node           : rabbit@29539ca56422
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>  home dir       : /var/lib/rabbitmq
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>  config file(s) : /etc/rabbitmq/conf.d/10-defaults.conf
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>                 : /etc/rabbitmq/conf.d/20-management_agent.disable_metrics_collector.conf
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>  cookie hash    : xyJUhoBkOPNciTnQJtbbwQ==
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>  log(s)         : <stdout>
annotator_message_q  | 2023-07-17 08:29:28.526716+00:00 [info] <0.230.0>  data dir       : /var/lib/rabbitmq/mnesia/rabbit@29539ca56422
annotator_message_q  | 2023-07-17 08:29:29.757980+00:00 [info] <0.230.0> Running boot step pre_boot defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.758034+00:00 [info] <0.230.0> Running boot step rabbit_global_counters defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.758219+00:00 [info] <0.230.0> Running boot step rabbit_osiris_metrics defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.758317+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.758838+00:00 [info] <0.230.0> Running boot step rabbit_alarm defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.760850+00:00 [info] <0.299.0> Memory high watermark set to 12752 MiB (13371775385 bytes) of 31880 MiB (33429438464 bytes) total
annotator_message_q  | 2023-07-17 08:29:29.762249+00:00 [info] <0.301.0> Enabling free disk space monitoring (disk free space: 133362720768, total memory: 33429438464)
annotator_message_q  | 2023-07-17 08:29:29.762303+00:00 [info] <0.301.0> Disk free limit set to 50MB
annotator_message_q  | 2023-07-17 08:29:29.763119+00:00 [info] <0.230.0> Running boot step code_server_cache defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.763181+00:00 [info] <0.230.0> Running boot step file_handle_cache defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.763313+00:00 [info] <0.304.0> Limiting to approx 1048479 file handles (943629 sockets)
annotator_message_q  | 2023-07-17 08:29:29.763395+00:00 [info] <0.305.0> FHC read buffering: OFF
annotator_message_q  | 2023-07-17 08:29:29.763435+00:00 [info] <0.305.0> FHC write buffering: ON
annotator_message_q  | 2023-07-17 08:29:29.763614+00:00 [info] <0.230.0> Running boot step worker_pool defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.763659+00:00 [info] <0.282.0> Will use 12 processes for default worker pool
annotator_message_q  | 2023-07-17 08:29:29.763694+00:00 [info] <0.282.0> Starting worker pool 'worker_pool' with 12 processes in it
annotator_message_q  | 2023-07-17 08:29:29.764046+00:00 [info] <0.230.0> Running boot step database defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.764299+00:00 [info] <0.230.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@29539ca56422 is empty. Assuming we need to join an existing cluster or initialise from scratch...
annotator_message_q  | 2023-07-17 08:29:29.764339+00:00 [info] <0.230.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
annotator_message_q  | 2023-07-17 08:29:29.764365+00:00 [info] <0.230.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
annotator_message_q  | 2023-07-17 08:29:29.764416+00:00 [info] <0.230.0> All discovered existing cluster peers:
annotator_message_q  | 2023-07-17 08:29:29.764438+00:00 [info] <0.230.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
annotator_message_q  | 2023-07-17 08:29:29.766045+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
annotator_message_q  | 2023-07-17 08:29:29.807749+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q  | 2023-07-17 08:29:29.807842+00:00 [info] <0.230.0> Successfully synced tables from a peer
annotator_message_q  | 2023-07-17 08:29:29.808164+00:00 [notice] <0.283.0> Feature flags: attempt to enable `stream_sac_coordinator_unblock_group`...
annotator_message_q  | 2023-07-17 08:29:29.829500+00:00 [notice] <0.283.0> Feature flags: `stream_sac_coordinator_unblock_group` enabled
annotator_message_q  | 2023-07-17 08:29:29.829645+00:00 [notice] <0.283.0> Feature flags: attempt to enable `restart_streams`...
annotator_message_q  | 2023-07-17 08:29:29.851801+00:00 [notice] <0.283.0> Feature flags: `restart_streams` enabled
annotator_message_q  | 2023-07-17 08:29:29.851965+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q  | 2023-07-17 08:29:29.852057+00:00 [info] <0.230.0> Successfully synced tables from a peer
annotator_message_q  | 2023-07-17 08:29:29.861128+00:00 [info] <0.230.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
annotator_message_q  | 2023-07-17 08:29:29.861263+00:00 [info] <0.230.0> Successfully synced tables from a peer
annotator_message_q  | 2023-07-17 08:29:29.861308+00:00 [info] <0.230.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
annotator_message_q  | 2023-07-17 08:29:29.861348+00:00 [info] <0.230.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
annotator_message_q  | 2023-07-17 08:29:29.861494+00:00 [info] <0.230.0> Running boot step tracking_metadata_store defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.861572+00:00 [info] <0.490.0> Setting up a table for connection tracking on this node: tracked_connection
annotator_message_q  | 2023-07-17 08:29:29.861625+00:00 [info] <0.490.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost
annotator_message_q  | 2023-07-17 08:29:29.861681+00:00 [info] <0.490.0> Setting up a table for per-user connection counting on this node: tracked_connection_per_user
annotator_message_q  | 2023-07-17 08:29:29.861724+00:00 [info] <0.490.0> Setting up a table for channel tracking on this node: tracked_channel
annotator_message_q  | 2023-07-17 08:29:29.861762+00:00 [info] <0.490.0> Setting up a table for channel tracking on this node: tracked_channel_per_user
annotator_message_q  | 2023-07-17 08:29:29.861816+00:00 [info] <0.230.0> Running boot step networking_metadata_store defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.861876+00:00 [info] <0.230.0> Running boot step feature_flags defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.861969+00:00 [info] <0.230.0> Running boot step codec_correctness_check defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862015+00:00 [info] <0.230.0> Running boot step external_infrastructure defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862047+00:00 [info] <0.230.0> Running boot step rabbit_event defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862145+00:00 [info] <0.230.0> Running boot step rabbit_registry defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862202+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862262+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862322+00:00 [info] <0.230.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862385+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862425+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862461+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862501+00:00 [info] <0.230.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862543+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862580+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862619+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862658+00:00 [info] <0.230.0> Running boot step rabbit_priority_queue defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862689+00:00 [info] <0.230.0> Priority queues enabled, real BQ is rabbit_variable_queue
annotator_message_q  | 2023-07-17 08:29:29.862729+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862769+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862813+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_random defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862882+00:00 [info] <0.230.0> Running boot step kernel_ready defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862907+00:00 [info] <0.230.0> Running boot step rabbit_sysmon_minder defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.862996+00:00 [info] <0.230.0> Running boot step rabbit_epmd_monitor defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.863375+00:00 [info] <0.498.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
annotator_message_q  | 2023-07-17 08:29:29.863457+00:00 [info] <0.230.0> Running boot step guid_generator defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.864356+00:00 [info] <0.230.0> Running boot step rabbit_node_monitor defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.864498+00:00 [info] <0.502.0> Starting rabbit_node_monitor
annotator_message_q  | 2023-07-17 08:29:29.864568+00:00 [info] <0.230.0> Running boot step delegate_sup defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.864910+00:00 [info] <0.230.0> Running boot step rabbit_memory_monitor defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865042+00:00 [info] <0.230.0> Running boot step rabbit_fifo_dlx_sup defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865100+00:00 [info] <0.230.0> Running boot step core_initialized defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865126+00:00 [info] <0.230.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865155+00:00 [info] <0.230.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865187+00:00 [info] <0.230.0> Running boot step rabbit_definitions_hashing defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865242+00:00 [info] <0.230.0> Running boot step rabbit_exchange_parameters defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865451+00:00 [info] <0.230.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865677+00:00 [info] <0.230.0> Running boot step rabbit_policies defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865913+00:00 [info] <0.230.0> Running boot step rabbit_policy defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.865980+00:00 [info] <0.230.0> Running boot step rabbit_queue_location_validator defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.866044+00:00 [info] <0.230.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.866085+00:00 [info] <0.230.0> Running boot step rabbit_stream_coordinator defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.866173+00:00 [info] <0.230.0> Running boot step rabbit_vhost_limit defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.866226+00:00 [info] <0.230.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
annotator_message_q  | 2023-07-17 08:29:29.866260+00:00 [info] <0.230.0> Management plugin: using rates mode 'basic'
annotator_message_q  | 2023-07-17 08:29:29.866419+00:00 [info] <0.230.0> Running boot step recovery defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.866906+00:00 [info] <0.230.0> Running boot step empty_db_check defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.866949+00:00 [info] <0.230.0> Will seed default virtual host and user...
annotator_message_q  | 2023-07-17 08:29:29.867010+00:00 [info] <0.230.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
annotator_message_q  | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0> Inserted a virtual host record {vhost,<<"/">>,[],
annotator_message_q  | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0>                                       #{description =>
annotator_message_q  | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0>                                             <<"Default virtual host">>,
annotator_message_q  | 2023-07-17 08:29:29.868513+00:00 [info] <0.230.0>                                         tags => []}}
annotator_message_q  | 2023-07-17 08:29:29.874012+00:00 [info] <0.547.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@29539ca56422/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
annotator_message_q  | 2023-07-17 08:29:29.874500+00:00 [info] <0.547.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
annotator_message_q  | 2023-07-17 08:29:29.876120+00:00 [info] <0.547.0> Starting message stores for vhost '/'
annotator_message_q  | 2023-07-17 08:29:29.876272+00:00 [info] <0.557.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
annotator_message_q  | 2023-07-17 08:29:29.876856+00:00 [info] <0.547.0> Started message store of type transient for vhost '/'
annotator_message_q  | 2023-07-17 08:29:29.876971+00:00 [info] <0.561.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
annotator_message_q  | 2023-07-17 08:29:29.877336+00:00 [warning] <0.561.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
annotator_message_q  | 2023-07-17 08:29:29.877804+00:00 [info] <0.547.0> Started message store of type persistent for vhost '/'
annotator_message_q  | 2023-07-17 08:29:29.877894+00:00 [info] <0.547.0> Recovering 0 queues of type rabbit_classic_queue took 3ms
annotator_message_q  | 2023-07-17 08:29:29.877940+00:00 [info] <0.547.0> Recovering 0 queues of type rabbit_quorum_queue took 0ms
annotator_message_q  | 2023-07-17 08:29:29.877974+00:00 [info] <0.547.0> Recovering 0 queues of type rabbit_stream_queue took 0ms
annotator_message_q  | 2023-07-17 08:29:29.879609+00:00 [info] <0.230.0> Created user 'user'
annotator_message_q  | 2023-07-17 08:29:29.880371+00:00 [info] <0.230.0> Successfully set user tags for user 'user' to [administrator]
annotator_message_q  | 2023-07-17 08:29:29.881166+00:00 [info] <0.230.0> Successfully set permissions for user 'user' in virtual host '/' to '.*', '.*', '.*'
annotator_message_q  | 2023-07-17 08:29:29.881209+00:00 [info] <0.230.0> Running boot step rabbit_observer_cli defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881265+00:00 [info] <0.230.0> Running boot step rabbit_looking_glass defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881294+00:00 [info] <0.230.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881385+00:00 [info] <0.230.0> Running boot step background_gc defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881487+00:00 [info] <0.230.0> Running boot step routing_ready defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881525+00:00 [info] <0.230.0> Running boot step pre_flight defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881549+00:00 [info] <0.230.0> Running boot step notify_cluster defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881575+00:00 [info] <0.230.0> Running boot step networking defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881604+00:00 [info] <0.230.0> Running boot step definition_import_worker_pool defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.881644+00:00 [info] <0.282.0> Starting worker pool 'definition_import_pool' with 12 processes in it
annotator_message_q  | 2023-07-17 08:29:29.882123+00:00 [info] <0.230.0> Running boot step cluster_name defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.882181+00:00 [info] <0.230.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-eQfBJ6nw978Ey1-d__x7KQ'
annotator_message_q  | 2023-07-17 08:29:29.882977+00:00 [info] <0.230.0> Running boot step direct_client defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.883049+00:00 [info] <0.230.0> Running boot step rabbit_maintenance_mode_state defined by app rabbit
annotator_message_q  | 2023-07-17 08:29:29.883093+00:00 [info] <0.230.0> Creating table rabbit_node_maintenance_states for maintenance mode status
annotator_message_q  | 2023-07-17 08:29:29.884551+00:00 [info] <0.607.0> Resetting node maintenance status
annotator_message_q  | 2023-07-17 08:29:29.893547+00:00 [info] <0.634.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
annotator_message_q  | 2023-07-17 08:29:29.893684+00:00 [info] <0.607.0> Ready to start client connection listeners
annotator_message_q  | 2023-07-17 08:29:29.894988+00:00 [info] <0.678.0> started TCP listener on [::]:5672
annotator_message_q  |  completed with 3 plugins.
annotator_message_q  | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0> Server startup complete; 3 plugins started.
annotator_message_q  | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0>  * rabbitmq_prometheus
annotator_message_q  | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0>  * rabbitmq_web_dispatch
annotator_message_q  | 2023-07-17 08:29:29.939302+00:00 [info] <0.607.0>  * rabbitmq_management_agent
annotator_workers    | /usr/local/lib/python3.6/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
annotator_workers    | absolutely not recommended!
annotator_workers    | 
annotator_workers    | Please specify a different user using the --uid option.
annotator_workers    | 
annotator_workers    | User information: uid=0 euid=0 gid=0 egid=0
annotator_workers    | 
annotator_workers    |   uid=uid, euid=euid, gid=gid, egid=egid,
annotator_workers    | [2023-07-17 08:29:31,570: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers    | Trying again in 2.00 seconds...
annotator_workers    | 
annotator_workers    | [2023-07-17 08:29:37,580: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers    | Trying again in 4.00 seconds...
annotator_workers    | 
annotator_workers    | [2023-07-17 08:29:45,598: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers    | Trying again in 6.00 seconds...
annotator_workers    | 
annotator_workers    | [2023-07-17 08:29:55,622: ERROR/MainProcess] consumer: Cannot connect to amqp://user:**@messageq:5672//: timed out.
annotator_workers    | Trying again in 8.00 seconds...
annotator_workers    | 
annotator_webclient  | [2023-07-17 08:29:57 +0000] [11] [ERROR] Exception in worker process
annotator_webclient  | Traceback (most recent call last):
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
annotator_webclient  |     worker.init_process()
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 102, in init_process
annotator_webclient  |     super(EventletWorker, self).init_process()
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
annotator_webclient  |     self.load_wsgi()
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
annotator_webclient  |     self.wsgi = self.app.wsgi()
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
annotator_webclient  |     self.callable = self.load()
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
annotator_webclient  |     return self.load_wsgiapp()
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
annotator_webclient  |     return util.import_app(self.app_uri)
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
annotator_webclient  |     __import__(module)
annotator_webclient  |   File "/workspace/webserver/__init__.py", line 63, in <module>
annotator_webclient  |     app = create_app()
annotator_webclient  |   File "/workspace/webserver/__init__.py", line 57, in create_app
annotator_webclient  |     ImageModel.objects.update(annotating=[])
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/mongoengine/queryset/manager.py", line 37, in __get__
annotator_webclient  |     queryset = queryset_class(owner, owner._get_collection())
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 208, in _get_collection
annotator_webclient  |     db.client.is_primary:
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 857, in is_primary
annotator_webclient  |     return self._server_property('is_writable')
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 689, in _server_property
annotator_webclient  |     writable_server_selector)
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 224, in select_server
annotator_webclient  |     address))
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 183, in select_servers
annotator_webclient  |     selector, server_timeout, address)
annotator_webclient  |   File "/usr/local/lib/python3.6/site-packages/pymongo/topology.py", line 199, in _select_servers_loop
annotator_webclient  |     self._error_message(selector))
annotator_webclient  | pymongo.errors.ServerSelectionTimeoutError: database:27017: timed out
annotator_webclient  | [2023-07-17 08:29:57 +0000] [11] [INFO] Worker exiting (pid: 11)
annotator_webclient  | [2023-07-17 08:29:58 +0000] [7] [INFO] Shutting down: Master
annotator_webclient  | [2023-07-17 08:29:58 +0000] [7] [INFO] Reason: Worker failed to boot.
annotator_webclient exited with code 0
@SixK
Copy link

SixK commented Jul 29, 2023

I see no logs for annotator_mongodb image and you have an error about mongodb access:
annotator_webclient | pymongo.errors.ServerSelectionTimeoutError: database:27017: timed out

So your problem seem's related to mongodb.

I would try to run:
docker-compose up annotator_mongodb

@geminixiang
Copy link
Contributor

docker compose up starup coco-annotartor must be like this (4 container)

  • annotator_webclient
  • annotator_workers
  • annotator_mongodb
  • annotator_message_q

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ovalerio @geminixiang @SixK