Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Resource and Cache Updater #248

Draft
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

irfan-dahir
Copy link
Contributor

@irfan-dahir irfan-dahir commented Jun 29, 2022

The crux of this PR is to improve response performance and to help improve #177

I'll be focusing on implementing the following:

  • Executing update resource/cache after serving response (so the requester does not have to wait for the server to parse MAL first)
  • Stale response will be served until the updater job completes
  • Improving code flow

The following flow is to be implemented for Resources:

graph TD;
    X[Start] --> A
    A[Query DB] --> B{Resource Exists?};
    B -->|Yes| C{Is Expired?}
    B -->|No| D{Scrape MAL}
    C -->|Yes| G[dispatch UpdateResource]
    C -->|No| Z[Jikan returns Resource]
    D -->|MAL returns error| F[Jikan returns error]
    D -->|MAL page exists and parsed data is returned| E[Insert as new resource]
    E --> A
    G--> Z

The following flow is to be implemented for Cache:

graph TD;
    X[Start] --> B{Cache Exists?}
    B -->|Yes| C{Is Expired?}
    B -->|No| D{Scrape MAL}
    C -->|Yes| G[dispatch UpdateCache]
    C -->|No| Z[Jikan returns Cache]
    D -->|MAL returns error| F[Jikan returns error]
    D -->|MAL page exists and parsed data is returned| E[Insert as new cache]
    E --> B
    G--> Z

There will be 2 Updater Jobs

  1. UpdateResource: Uses mal_id as unique ID and updates through query and scraper - both available in defined in: App\Anime, App\Manga, App\Character, and App\Person classes
  2. UpdateCache: Uses request_hash as unique ID and updates by passed table name and scraper closure

Jobs utilize Queues and queues are stored in MongoDB and utilize Redis.
Pending Jobs table: jobs
Failed Jobs table: jobs_failed

This PR also adds new migrations for these tables which can be added by
php artisan migrate --path=/database/migrations/2022_06_29_231146_create_jobs_table.php
php artisan migrate --path=/database/migrations/2022_06_29_231217_create_jobs_failed_table.php


Running Queues

php artisan queue:work --queue=high,default

Notable changes

  • The internal last modified property has been changed from modifiedAt to updated_at
  • The internal created property has been changed from createdAt to created_at

Todo

  • Create UpdateResource
  • Create UpdateCache
  • Configure delay between jobs to prevent rate limiting
  • Implement UpdateResource for Anime, Manga, Character and Person
  • Implement UpdateCache for all cache based requests

⚠️ This will require resolving merge conflicts with #227 to include the new framework

@irfan-dahir irfan-dahir added this to the REST 4.0 milestone Jun 29, 2022
@irfan-dahir irfan-dahir self-assigned this Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

None yet

2 participants