Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create DAG to scrape Littlepay Group and Product information nightly #3293

Open
5 tasks
thekaveman opened this issue Feb 27, 2024 · 0 comments
Open
5 tasks

Comments

@thekaveman
Copy link
Member

thekaveman commented Feb 27, 2024

User story / feature request

As a Cal-ITP Customer Success representative, I need to be able to monitor changes to Littlepay Group and Product configuration in order to help diagnose issues with transactions and configuration related to discounts provided by the Cal-ITP Benefits app.

Acceptance Criteria

  • A table of Littlepay Groups exists containing at least the columns id, label, participant_id, populated from the Littlepay Back Office Groups endpoint
  • A table of Littlepay Products exists containing at least the columns id, code, status, type, description, participant_id, populated from the Littlepay Back Office Products endpoint
  • A table of Littlepay Group<>Product linkages exists containing at least the columns group_id, product_id, participant_id, populated from the Littlepay Back Office Product Groups endpoint
  • The data is acquired nightly and timestamped to allow viewing changes over time
  • This data is available to Metabase users

Notes

The cal-itp/littlepay Python package implements the necessary APIs to fetch this data as Python objects and in CSV format.

A GitHub Action is set up to scrape this data into CSVs stored in the repository and can be used as a model for a new DAG.

See this Slack thread for more background / context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant