Launching Gunicorn

[TommiRTVA]

Environmenti is Red Hat Linux 8 and Podman is used instead of Docker. Startup srcips is used instead of Docker-compose.

Before 0.44, this script worked:

podman run --pod annif --name annif-gunicorn -dt -v /fs/data1/annif/annif-projects/:/annif-projects/ -u xxx:xxx quay.io/natlibfi/annif gunicorn "annif:create_app()" "--bind" "0.0.0.0:8000"

Now it fails:

gunicorn --log-level debug annif "annif:create_app()" --bind 0.0.0.0:8000
[2019-11-25 09:02:48 +0000] [76] [DEBUG] Current configuration:
config: None
bind: ['0.0.0.0:8000']
backlog: 2048
workers: 1
worker_class: sync
threads: 1
worker_connections: 1000
max_requests: 0
max_requests_jitter: 0
timeout: 30
graceful_timeout: 30
keepalive: 2
limit_request_line: 4094
limit_request_fields: 100
limit_request_field_size: 8190
reload: False
reload_engine: auto
reload_extra_files: []
spew: False
check_config: False
preload_app: False
sendfile: None
reuse_port: False
chdir: /annif-projects
daemon: False
raw_env: []
pidfile: None
worker_tmp_dir: None
user: 10006475
group: 10006475
umask: 0
initgroups: False
tmp_upload_dir: None
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
accesslog: None
disable_redirect_access_to_syslog: False
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
errorlog: -
loglevel: debug
capture_output: False
logger_class: gunicorn.glogging.Logger
logconfig: None
logconfig_dict: {}
syslog_addr: udp://localhost:514
syslog: False
syslog_prefix: None
syslog_facility: user
enable_stdio_inheritance: False
statsd_host: None
dogstatsd_tags:
statsd_prefix:
proc_name: None
default_proc_name: annif
pythonpath: None
paste: None
on_starting: <function OnStarting.on_starting at 0x7f5c534393b0>
on_reload: <function OnReload.on_reload at 0x7f5c534394d0>
when_ready: <function WhenReady.when_ready at 0x7f5c534395f0>
pre_fork: <function Prefork.pre_fork at 0x7f5c53439710>
post_fork: <function Postfork.post_fork at 0x7f5c53439830>
post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f5c53439950>
worker_int: <function WorkerInt.worker_int at 0x7f5c53439a70>
worker_abort: <function WorkerAbort.worker_abort at 0x7f5c53439b90>
pre_exec: <function PreExec.pre_exec at 0x7f5c53439cb0>
pre_request: <function PreRequest.pre_request at 0x7f5c53439dd0>
post_request: <function PostRequest.post_request at 0x7f5c53439e60>
child_exit: <function ChildExit.child_exit at 0x7f5c53439f80>
worker_exit: <function WorkerExit.worker_exit at 0x7f5c5344f0e0>
nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f5c5344f200>
on_exit: <function OnExit.on_exit at 0x7f5c5344f320>
proxy_protocol: False
proxy_allow_ips: ['127.0.0.1']
keyfile: None
certfile: None
ssl_version: 2
cert_reqs: 0
ca_certs: None
suppress_ragged_eofs: True
do_handshake_on_connect: False
ciphers: None
raw_paste_global_conf: []
[2019-11-25 09:02:48 +0000] [76] [INFO] Starting gunicorn 20.0.0
[2019-11-25 09:02:48 +0000] [76] [DEBUG] Arbiter booted
[2019-11-25 09:02:48 +0000] [76] [INFO] Listening at: http://0.0.0.0:8000 (76)
[2019-11-25 09:02:48 +0000] [76] [INFO] Using worker: sync
[2019-11-25 09:02:48 +0000] [80] [INFO] Booting worker with pid: 80
[2019-11-25 09:02:48 +0000] [76] [DEBUG] 1 workers
[2019-11-25 09:02:50 +0000] [80] [DEBUG] Exception while loading the application
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 340, in import_app
app = getattr(mod, obj)
AttributeError: module 'annif' has no attribute 'application'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 133, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 142, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 344, in import_app
raise AppImportError("Failed to find application object %r in %r" % (obj, module))
gunicorn.errors.AppImportError: Failed to find application object 'application' in 'annif'
Failed to find application object 'application' in 'annif'
[2019-11-25 09:02:50 +0000] [80] [INFO] Worker exiting (pid: 80)
[2019-11-25 09:02:51 +0000] [76] [INFO] Shutting down: Master
[2019-11-25 09:02:51 +0000] [76] [INFO] Reason: App failed to load.

Command "annif run" does work though.

podman exec -it annif-shell bash

annif run

• Environment: production
  WARNING: This is a development server. Do not use it in a production deployment.
  Use a production WSGI server instead.
• Debug mode: off
• Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

[juhoinkinen]

This is most probably due to a change in gunicorn behaviour in its version 20.0. The behaviour in gunicorn will be changed back in a future release. For the moment I'll pin the gunicorn to 19.* for Annif, and the default image in quay.io (i.e. with latest tag) should work.

[juhoinkinen]

Actually gunicorn 20.0.1 is already released and this is fixed in that.

[juhoinkinen]

Thanks @TommiRTVA for reporting, this problem was not only in case of Podman but in any usage of Gunicorn 20.0.0 started with gunicorn annif:create_app(). Both annif:latest and annif:0.44 images in quay.io have now working Gunicorn version (just rerun of the 0.44 build was enough to install gunicorn 20.0.2).