Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add locking mechanism around scraping to support distributed setups #495

Open
pushrbx opened this issue Feb 4, 2024 · 0 comments
Open

Comments

@pushrbx
Copy link
Collaborator

pushrbx commented Feb 4, 2024

Currently Jikan API works the following way:
image

If you have multiple workers/instances of Jikan API running behind a load balancer, and if the requested anime/manga is not in mongodb then there can be a race condition, where two instances of Jikan API would scrape the same datasheet from MAL resulting in duplicated items in mongodb.

This behaviour has been reported several times: #262 #333

The solution I can think of is some sort of pessimistic locking through some lock ticket stored in a separate collection in mongodb, and if there is an entry for an anime/manga item there, the second worker which would try to scrape the item would not insert the scraped item. However this can cause the risk of ip ban, because while Worker A is scraping and inserting into mongodb, Worker B might get two requests for the same item and it would have to scrape twice.

In case of pessimistic locking the worker which finds a locking ticket should return 423 status code.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/423

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

1 participant