Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka stream 2 #3231

Open
wants to merge 98 commits into
base: sprint-1.15
Choose a base branch
from
Open

Kafka stream 2 #3231

wants to merge 98 commits into from

Conversation

dabasov
Copy link
Member

@dabasov dabasov commented Apr 7, 2024

Fixes

Changes

Need to be mentioned in CHANGELOG.md?

Tests

Tasks to complete before merging PR:

  • Ensure system tests are passing. If not Run them manually to check for any regressions 📋
  • Do any new system tests need added to test this change? do any existing system tests need updated? If so create a PR at 0chain/system_test
  • Merge your system tests PR to master AFTER merging this PR

Associated PRs (Link as appropriate):

  • blobber:
  • gosdk:
  • system_test:
  • zboxcli:
  • zwalletcli:
  • Other: ...

divyank-aggarwal and others added 30 commits February 20, 2024 21:19
@divyank-aggarwal @Roshan-Mehta
initial setup for kafka and basic use of kafka to send events data fo…
cca3b97 
…r processing to 0box
@divyank-aggarwal @Roshan-Mehta
corrected setup after testing
b3e1c2f
@divyank-aggarwal @Roshan-Mehta
refactoring kafka
8bd3a6f
@devyetii @Roshan-Mehta
resolve simple comments
e34cae0
@devyetii @Roshan-Mehta
kafka support multiple topics
acc41c2
@devyetii @Roshan-Mehta
publish events after processing them
8b3f0a0
@devyetii @Roshan-Mehta
publish events async
470b2d0
@devyetii @Roshan-Mehta
finalize event sequece number with manual testing
fae0e62
@devyetii @Roshan-Mehta
mod tidy
1818fc2
@devyetii @Roshan-Mehta
counter for round events not global
3236458
@devyetii @Roshan-Mehta
separate kafka in a separate compose file
d5343a5
@devyetii @Roshan-Mehta
clean kafka within clean.sh
1a965c5
@devyetii @Roshan-Mehta
global counter
bd42e66
@devyetii @Roshan-Mehta
revert conductor workflow files
22d5cac
@devyetii @Roshan-Mehta
revert
7a43e35
@Roshan-Mehta
Revert unwanted changes
89758a7
@Roshan-Mehta
Revert go.work.sum
e2d3e87
@devyetii
fix missing dependency
20e2e49
@devyetii
go mod tidy
8cc5be7
@devyetii
Merge branch 'sprint-1.13' into kafka_stream_2
c7d8ee6
@devyetii
fix db migrations
f4e814e
@Roshan-Mehta
Publish kafka msg from addEvents
beea784
@Roshan-Mehta
Filter events based on data field
f97cbf0
@Roshan-Mehta
Fix corner case in filter events
bc45448
@Roshan-Mehta
Add round number
16acda2
@Roshan-Mehta
Handle concurrent read and write safely
fa15506
@Roshan-Mehta
Fix bug
307781e
@Roshan-Mehta
Send event instead of event slice
8dbae12
@Roshan-Mehta
Merge branch 'sprint-1.13' into kafka_stream_2
ba9d705
@Roshan-Mehta
fix bug in event
ea6d689
peterlimg and others added 28 commits April 22, 2024 14:01
@peterlimg
Add back all config for idempotent
0cb4566
@peterlimg
Set MaxOpenRequests to fix idempotent
68dcf92
@peterlimg
Fix write timeout
77cd961
@peterlimg
Debug no idempotent
d5aef60
@peterlimg
Require return true
543e7f6
@peterlimg
Debug message push
751d1ab
@peterlimg
Avoid timeout for pushing messages
01bb2bd
@peterlimg
Add kafka push logs
3f9f483
@peterlimg
Turn off idempotent push as it is too slow
e577c00
@peterlimg
Debug enable idempotent but not wait for success
c3c6f14
@peterlimg
Fix max open requests
65098e7
@peterlimg
Test store events
4f8864e
@peterlimg
Add more test
76f8f55
@peterlimg
Log seq
905d29d
@peterlimg
Fix events is_published setting
c15977f
@peterlimg
Fix events table update
d9f8521
@peterlimg
Add debug logs
a54071f
@peterlimg
Fix get block events
f58034e
@peterlimg
Debug set return success true
6d3356e
@peterlimg
Add unpublish test
53737e3
@peterlimg
Remove test code
cad2f3f
@peterlimg
Refactor the block events entity name
1440547
@peterlimg
Enable the activator to store Events
d48be7f
@peterlimg
Merge branch 'sprint-1.14' into kafka_stream_2
7a807ef
@peterlimg
Merge branch 'kafka_stream_2_peter' into kafka_stream_2
e1e09d3 
* kafka_stream_2_peter: (30 commits)
  Enable the activator to store Events
  Refactor the block events entity name
  Remove test code
  Add unpublish test
  Debug set return success true
  Fix get block events
  Add debug logs
  Fix events table update
  Fix events is_published setting
  Log seq
  Add more test
  Test store events
  Fix max open requests
  Debug enable idempotent but not wait for success
  Turn off idempotent push as it is too slow
  Add kafka push logs
  Avoid timeout for pushing messages
  Debug message push
  Require return true
  Debug no idempotent
  ...
@peterlimg
Remove empty test file
587e584
@peterlimg
Fix missing var
3822382
@peterlimg
Minor lint fix
b985424
@Jayashsatolia403 Jayashsatolia403 changed the base branch from sprint-1.14 to sprint-1.15 May 13, 2024 06:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peterlimg peterlimg Awaiting requested review from peterlimg peterlimg is a code owner

@guruhubb guruhubb Awaiting requested review from guruhubb guruhubb is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@dabasov @divyank-aggarwal @devyetii @Roshan-Mehta @peterlimg @shahnawaz-creator @Jayashsatolia403