Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: sync/harvest speed #1168

Open
LukasKalbertodt opened this issue May 15, 2024 · 0 comments
Open

Tracking issue: sync/harvest speed #1168

LukasKalbertodt opened this issue May 15, 2024 · 0 comments
Labels
area:backend Everything backend related area:sync Syncing with an Opencast instance kind:improvement priority:low Low priority

Comments

@LukasKalbertodt
Copy link
Member

Since I looked into this briefly, let me just post some numbers here.

  • Test system: 7k events, 1k series => 3.5min
  • Bern system: 87k events, 10k series => 90min

The bern numbers are from a resync today. 80min were spent on Opencast side, 10min on Tobira side. For Tobira, storing series seems to be much slower. 500 series take roughly 14s to store. 500 events only take 1s. It's the opposite on the Opencast side: preparing a response with 500 series takes roughly 1s, but preparing one with 500 events takes 35s.

Others can use these numbers to estimate how long the sync is going to take on their system.

And for us: it would be nice to optimize these sync times still. Like, this is super slow compared to the data we are processing. This could be made faster by orders of magnitude. However, the more important question is: how much faster can we get it with only investing a little bit of development time? ;-) I still guess that a few hours invested here could bring significant wins. But yeah, not our priority.

@LukasKalbertodt LukasKalbertodt added area:backend Everything backend related kind:improvement priority:low Low priority area:sync Syncing with an Opencast instance labels May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:backend Everything backend related area:sync Syncing with an Opencast instance kind:improvement priority:low Low priority
Projects
None yet
Development

No branches or pull requests

1 participant