Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tie confluent cloud topic state management into dev/stage clinvar-streams deployments #17

Open
theferrit32 opened this issue Apr 5, 2021 · 2 comments
Open

Tie confluent cloud topic state management into dev/stage clinvar-streams deployments #17

theferrit32 opened this issue Apr 5, 2021 · 2 comments
Assignees
@theferrit32
Labels
clinvar Clinvar data exchange and reporting

Comments

@theferrit32
Copy link
Contributor

ccloud cli can be installed in a thin deployment startup image and have credentials passed in via secret. Either a ~/.netrc file or pass into ccloud login invocation.

These should only happen on dev/stage since topic state on prod cannot just be wiped.

  • clinvar-raw: delete/recreate clinvar-raw topic before launching deployment
  • clinvar-combiner: delete/recreate clinvar-combined topic before launching deployment
@theferrit32 theferrit32 added the clinvar Clinvar data exchange and reporting label Apr 5, 2021
@theferrit32 theferrit32 self-assigned this Apr 5, 2021
@theferrit32 theferrit32 changed the title Tie confluent cloud topic state management into dev/stage deployments Tie confluent cloud topic state management into dev/stage clinvar-streams deployments Apr 5, 2021
@theferrit32
Copy link
Contributor Author

It looks like the only way to authenticate to the ccloud CLI is through Confluent admin accounts, which are email-based. I don't see a way to authenticate to it using a Confluent cluster service account. We could potentially use the clingen devs google group account. Or create another account if that one is too broad use.

@theferrit32
Copy link
Contributor Author

The java org.kafka API for managing topics seems a little not-straightforward. The confluent kafka library in python has an admin client that can do it

https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html#confluent_kafka.admin.NewTopic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@theferrit32 theferrit32

Labels
clinvar Clinvar data exchange and reporting
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@theferrit32