FEATURE: Streaming endpoint for xref matches #3520
Labels
backend
Issues related to Aleph’s backend, API, CLI etc.
feature-request
Requests for new features or enhancements of existing features
Is your feature request related to a problem? Please describe.
There are cases when it is necessary to process xref matches offline, e.g. in order to use a custom scoring method. Currently, there’s no easy and scalable way to fetch all xref results.
Describe the solution you'd like
There should be a streaming API endpoint similar that allows to stream xref matches similar to the entity streaming endpoint.
Describe alternatives you've considered
Aleph has an option to export xref matches. This doesn’t work well for large numbers of matches (100k).
The default xref API is limited to a maximum result set of 10k matches, even when using pagination.
Additional context
Based on user feedback
The text was updated successfully, but these errors were encountered: