Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache directory with pre processed models is hard-coded #207

Open
cabral1888 opened this issue Sep 30, 2019 · 1 comment
Open

Cache directory with pre processed models is hard-coded #207

cabral1888 opened this issue Sep 30, 2019 · 1 comment

Comments

@cabral1888
Copy link

cabral1888 commented Sep 30, 2019

Hello. I am using the sparkdl in a Spark cluster with YARN integrated with Docker. I am having problems related to user home directory when the codes fetch the preprocessed models (like InceptronV3, XCeptron, etc) and stores it into my HOME_DIR. For advanced reasons, YARN doesn't create the user HOME_DIR, and when the library tries to write into this directory, it fails. What I need to do is to change the default behavior to store models in any directory as I want.

Would it be possible to change code behavior to define the cache directory at execution time? For instance, when I instantiate the following class:

featurizer = DeepImageFeaturizer(inputCol="image", outputCol="features", modelName="InceptionV3", cacheDir=<SOME_PATH>)

Obs.: The file with the HOME_DIR hard-coded is: src/main/scala/com/databricks/sparkdl/ModelFetcher.scala on line 40

Best regards!

@cabral1888 cabral1888 changed the title Cache directory with pre processed models hard-coded Cache directory with pre processed models is hard-coded Sep 30, 2019
@cabral1888
Copy link
Author

Someone?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant