Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make media pipeline storage more flexible #5991

Open
Gallaecio opened this issue Jul 28, 2023 · 0 comments
Open

Make media pipeline storage more flexible #5991

Gallaecio opened this issue Jul 28, 2023 · 0 comments

Comments

@Gallaecio
Copy link
Member

Looking at how people need to use subclassing or monkeypatching for some straightforward cases of media pipeline storage, such as configuring the target Google Storage Cloud project ID or configuring a new storage class for a different service, I think we need to look into making media pipeline storage configuration more flexible.

Specifically, I think we need to consider:

  • Expose FilesPipeline‘s STORE_SCHEMES as a setting (with a better name) that works also for images, rather than requiring subclassing to define new storage classes.
  • Make storage classes Scrapy components, that can define from_crawler or from_settings to configured themselves based on settings, instead of requiring users to subclass base media pipeline classes to set class-level values based on settings, which feels like monkeypatching.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant