Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Spark Streaming using Spark Operator #516

Merged
merged 11 commits into from
May 16, 2024

Conversation

lusoal
Copy link
Contributor

@lusoal lusoal commented May 3, 2024

What does this PR do?

🛑 Please open an issue first to discuss any significant work and flesh out details/direction - we would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

This will provision a stack for Spark streaming from Amazon MSK, doesn't need to be from MSK, could be replaced with Kafka in strimzi for example. The most value of the solution is showing examples of applications working in this stack using the operator and spark streaming.

Motivation

Customer requested PoC

More

  • Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
  • Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
  • Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
  • Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

  • E2E Test successfully complete before merge?

Additional Notes

@lusoal lusoal marked this pull request as ready for review May 6, 2024 17:56
@lusoal lusoal requested a review from vara-bonthu May 6, 2024 17:58
Copy link
Contributor

@vara-bonthu vara-bonthu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this PR @lusoal ! PR looks good to me. I presume that you have tested the blueprint end to end. Please ask some of our internal teams to run through the instructions. I would also add a warning to the website doc for any security requirements that needs to be tighten if they implement this in prod.

@lusoal
Copy link
Contributor Author

lusoal commented May 13, 2024

Nice work on this PR @lusoal ! PR looks good to me. I presume that you have tested the blueprint end to end. Please ask some of our internal teams to run through the instructions. I would also add a warning to the website doc for any security requirements that needs to be tighten if they implement this in prod.

Yes I tested indeed! Will ask to someone else to execute the steps. Thanks @vara-bonthu

@lusoal lusoal requested a review from askulkarni2 May 13, 2024 19:29
…e, added dynamic allocation and prometheus annotations to manifests
Copy link
Contributor

@vara-bonthu vara-bonthu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@vara-bonthu vara-bonthu merged commit 6fa0ca6 into awslabs:main May 16, 2024
38 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants