Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BFD-Insights initial PR #257

Merged
merged 29 commits into from
May 15, 2020
Merged

BFD-Insights initial PR #257

merged 29 commits into from
May 15, 2020

Conversation

RickHawesUSDS
Copy link
Contributor

@RickHawesUSDS RickHawesUSDS commented Apr 21, 2020

Why
The basic idea behind the BFD-Insights project is listed here. https://confluence.cms.gov/display/BB/Business+and+Security+Analytics+for+DASG

What
This is the initial check-in for bfd-insights. It creates the basic terraform for the project. As an example workflow, a food truck project.

See the doc folder for details.

Choices Made

  • insights top-level folder
  • Use terraform for deployment including table definitions
  • Food Truck example project

Future Work
The choice of terraform leads to thinking that GitHub actions will be used for deploys.

Rick Hawes added 4 commits April 18, 2020 15:37
- prod-lake
- groups
- common modules
- docs
- foodtruck
- new database/table/ organizatoin
- timestamp format in purchases
- Analysts permissions
- Documentation in organization
@RickHawesUSDS RickHawesUSDS marked this pull request as ready for review April 23, 2020 15:18
- Sarah Tully as an author
- Adjusted permissions for readers and authors
njdister
njdister previously approved these changes Apr 30, 2020
Copy link
Contributor

@njdister njdister left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking really good, just an informational question: now that there are separate project buckets, but also a data lake bucket, what are the delineation for their uses? Project buckets for the databases, and data lake bucket for user queries/scripts?

@RickHawesUSDS
Copy link
Contributor Author

RickHawesUSDS commented May 1, 2020

Looking really good, just an informational question: now that there are separate project buckets, but also a data lake bucket, what are the delineation for their uses? Project buckets for the databases, and data lake bucket for user queries/scripts?

@njdister Project buckets will contain the raw logs from the projects. They may also contain aggregations that are project-specific. The main data lake bucket will contain common aggregations (which there are none right now). From a security perspective, the project buckets are the entry-points for the project. The data-lake bucket is not accessible by the partners.

njdister
njdister previously approved these changes May 1, 2020
@RickHawesUSDS RickHawesUSDS requested a review from jzulim May 6, 2020 13:33
@RickHawesUSDS RickHawesUSDS changed the title BFD-Insights initial check BFD-Insights initial PR May 7, 2020
Copy link
Contributor

@njdister njdister left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed additional documentation and diagrams, great work. 🎉

@dtisza1
Copy link
Contributor

dtisza1 commented May 15, 2020

@RickHawesUSDS A good way to improve package manager level security is to utilize the pip-tools hash-checking feature. I think this should be added to this project. Maybe as a backlog item.
For example, this uses the pip-compile "--generate-hashes" option when generating the requirements.txt file.

Copy link
Contributor

@dtisza1 dtisza1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've read through the related confluence documentation and this PR several times. This makes sense to me and is well thought out. It looks like a great base implementation. I'm looking forward to helping with related BB2 tasks!

Great work on this PR and project design!

@RickHawesUSDS RickHawesUSDS merged commit 793b3ea into master May 15, 2020
@njdister njdister deleted the rick/bfd-insights branch June 11, 2020 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants