How to use argparse with gunicorn

[yarnabrina]

Hi!

I've come across multiple issues here where people faced problems to use gunicorn with their own application, which accepts command line arguments and parses them with argparse. For all of them, removing argparse was sufficient to solve the problem.

I'm facing the same issue, and I can confirm that instead of passing command line arguments and parsing with argparse, if I hardcode the configurations, my application works. But I will have to use argparse, since I need to run multiple instances of my Flask application with different configurations.

If I don't use gunicorn, I run my application as follows:

python flask_service.py -l log_configurations.json -s system_configurations.ini

Based on these arguments, I prepare a deep learning model for classification and then finally create a simple Flask app to expose the predict method. I need to run multiple models for a hierarchical classification problem. So, if I run the above command multiple times (with varying configuration file names), it works fine.

But if I try to run with gunicorn like this:

gunicorn -c gunicorn.conf.py flask_service:APP -l log_configurations.json -s system_configurations.ini

, then I get the following error:

usage: gunicorn [OPTIONS] [APP_MODULE]
gunicorn: error: unrecognized arguments: -l log_configurations.json -s system_configurations.ini

If I understand correctly, then the argparse of gunicorn trying to parse these arguments, and those are not passed to the argparse of my program.

But #2125 leads me to believe that it is actually possible to overcome this problem. I tried to follow #1988, but failed. I ran this:

gunicorn -c gunicorn.conf.py flask_service:APP -- -l log_configurations.json -s system_configurations.ini

It entered my program alright and reached the step for parsing arguments, but then it showed the following:

usage: gunicorn [-h] -l LOG -s SYSTEM
gunicorn: error: the following arguments are required: -l/--log, -s/--system

Can someone please provide a detailed code example to pass arguments?

Thanks.

[levinxo]

you can pass app's args through:

$ gunicorn 'you_app_file:app_func(foo="bar")'

[tilgovi]

You may have more success making your custom CLI start Gunicorn from the Python API.

Check the documentation on making a custom Gunicorn application: https://docs.gunicorn.org/en/latest/custom.html

[evil-shrike]

I have the same problem.
Definitely gunicorn's CLI arguments handling isn't perfect.
I see the following issues:

1. gunicorn should cut off its own arguments from args array before executing user module. So that we could use normal argparse.ArgumentParser.parse_args (currently it's not possible as it'll complain about gunicorn's arguments that are unknown to user module)
2. gunicorn should provide a way to pass arguments to user module that have same names as gunicorn's ones (--config for example). I'd expect @yarnabrina 's syntax to work (gunicorn -c gunicorn.conf.py flask_service:APP -- -l log_configurations.json) but it doens't

[benoitc]

passing user arguments is not supported. Since applications are handled in a process, it is expected to get custom settings using the environment orotjercustoms ways. Arguments are only there for gunicorn.

asimple way to pass custom environement functions from the command line is using the —env setting.

[tilgovi]

Agreed. I'll close this issue.

When gunicorn is run with its CLI the command line arguments are for gunicorn.

If there are ways we can improve the documentation around how to import gunicorn from your own application, please let us know. We'd be happy to review a PR.

[yarnabrina]

I'm no longer linked with the app I was developing while creating this issue, so don't have a quick way to check.

I'll try to create a small dummy app in this week to verify, but I think that if you use parse_known_args instead of parse_args here, it'd perhaps solve this issue indirectly.

gunicorn/gunicorn/app/base.py

Line 158 in cf55d2c

args = parser.parse_args()

@benoitc/ @tilgovi , Do you see any immediate issue in this change?

[tilgovi]

@yarnabrina that would be a fine way to handle things, but I would rather not make that change in Gunicorn. I think there's some risk that users meant to pass all the arguments to Gunicorn and somewhere there was a mistake. In that case, it might be better to fail fast than silently drop the unknown arguments.