Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: multiprocessing #54

Draft
wants to merge 11 commits into
base: main
Choose a base branch
from
Draft

feat: multiprocessing #54

wants to merge 11 commits into from

Conversation

ericbutera
Copy link
Contributor

@ericbutera ericbutera commented Oct 25, 2023 •
edited

Add multiprocessing to Cloud Connector.

Branch worker-pool-v4.0 will track this work as a feature branch until AWS, Azure, & GCP are all converted. Then worker-pool-v4.0 will be merged into main.

ericbutera and others added 11 commits October 3, 2023 07:40
@ericbutera
docs: architecture
617f1c8
@ericbutera
WIP
c530451
@ericbutera
WIP
1409fa2 
- aws_endpoint_url setting
- add max proc setting
- test scan_all pool fix
@ericbutera
WIP
1a8b003 
- pool can't be pickled, need to figure out self.pool issue
- introduced AwsScanContext to manage state of outer loop in an easier way inside workers
- so many TODOs for refactoring class state into context
@ericbutera
WIP
a4cd8ce 
- scan context everywhere
@ericbutera
WIP
df0e36a 
- introduce cloud events
- emit payloads
- hacky Aurora client
- research log line showing provider info
@ericbutera
WIP:
8a94b65 
- add provider to log (aws only for now)
- change add_seed to use a list and submit_seed_payload
- change add_cloud_asset to use a map + submit_cloud_asset_payload
- rough and ready aurora client
@ericbutera
WIP: add payload source
3c15888
@ericbutera
fix: issues from rebase
838b469
@ericbutera
WIP:
dd52318 
- remove unused comment code
- add temp_sts_credential to ctx
@grace-murphy
Merge branch 'AP-2560-worker-pool' into worker-pool-v4.0
52a5285
@ericbutera ericbutera added the do not merge This PR is not ready to be merged label Oct 25, 2023
@ericbutera ericbutera self-assigned this Oct 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@ericbutera ericbutera

Labels
do not merge This PR is not ready to be merged
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ericbutera @grace-murphy