New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feed URL has been added to the stats. #5982
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -480,6 +480,8 @@ def _handle_store_error(self, f, logmsg, spider, slot_type): | |
|
||
def _handle_store_success(self, f, logmsg, spider, slot_type): | ||
logger.info("Stored %s", logmsg, extra={"spider": spider}) | ||
feed_uri = logmsg.split(":")[-1].strip() | ||
self.crawler.stats.set_value(f"feedexport/feed_url/{feed_uri.split('/')[-1]}", feed_uri) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe this should be a Scrapy Extensions that users can opt-in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What would you prefer? @Gallaecio @wRAR There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please, use alternatives like https://docs.python.org/3/library/pathlib.html or https://docs.python.org/3/library/urllib.parse.html to operate with URIs paths There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If there are multiple feeds exported to the same top level path - For example, to the same bucket - only the last one will be keep. That behavior could be okish depending on your user case, but at least , it should be clearly documented. |
||
self.crawler.stats.inc_value(f"feedexport/success_count/{slot_type}") | ||
|
||
def _start_new_batch(self, batch_id, uri, feed_options, spider, uri_template): | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Avoid this type of text transformation, is quite possible it will break on the future if the format changes. If you need
slot.uri
you can pass it as an additional argument