Practical fix for securedrop-client#1025 Faster sync for all source

[kushaldas]

Status

Ready for review

Description of Changes

Fixes freedomofpress/securedrop-client#1025

Reimplements the get_sources call. On a staging system, direct calls using exposed port gave us the following numbers. Key size: 1024, all keys and fingerprints are in redis cache.

1461 sources, 50 submissions, and 20 replies for each.

This is the pure python implementation (as in this PR).

1.61 s ± 38.5 ms per loop (mean ± std. dev. of 3 runs, 50 loops each)

I also tested the same logic in a complied Rust extension just to see the difference.

Python module written in Rust on mod_wsgi-express

1.11 s ± 47.3 ms per loop (mean ± std. dev. of 3 runs, 50 loops each)

Python module written in Rust on our apache2

1.44 s ± 114 ms per loop (mean ± std. dev. of 3 runs, 50 loops each)

Testing

How should the reviewer test this PR?

• make staging
• Run the client against this server.
• CI is green

I am waiting for the CI :)

Deployment

Any special considerations for deployment? Consider both:

1. Upgrading existing production instances.
2. New installs.

Checklist

If you made changes to the server application code:

• Linting (make lint) and tests (make test) pass in the development container

If you made changes to securedrop-admin:

• Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

• Configuration tests pass

If you made non-trivial code changes:

• I have written a test plan and validated it for this PR

If you made changes to documentation:

• Doc linting (make docs-lint) passed locally

If you added or updated a code dependency:

Choose one of the following:

• I have performed a diff review and pasted the contents to the packaging wiki
• I would like someone else to do the diff review

[eloquence]

@rmol @kushaldas @creviera Is this worth revisiting, or should we close it?

[kushaldas]

If v2 API is further away time line wise, I think we should look at this one.

[cfm]

I've updated the title and description here to reflect that it addresses freedomofpress/securedrop-client#1025, not #1025 here. I think we should revisit this in the broader conversation around when and how to consider the next version of the Journalist API.