Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support --conf-path when using mrjob spark-submit #2147

Open
mj3c opened this issue Feb 11, 2020 · 0 comments
Open

Support --conf-path when using mrjob spark-submit #2147

mj3c opened this issue Feb 11, 2020 · 0 comments

Comments

@mj3c
Copy link

mj3c commented Feb 11, 2020

Am I doing something wrong or is the --conf-path / -c option not supported when trying to run a PySpark script using mrjob spark-submit? For example, I have an EMR runner configuration specified in the custom.conf file, and I try to run the script with:

mrjob spark-submit -r emr -c custom.conf main.py

In the logs, it clearly says:

Looking for configs in /home/<my_home>/.mrjob.conf
Looking for configs in /etc/mrjob.conf
No configs found; falling back on auto-configuration

Why does it only check those two locations, and not consider the passed argument?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant