Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend /submit service #310

Open
maciejhirsz opened this issue Jan 21, 2021 · 0 comments
Open

Backend /submit service #310

maciejhirsz opened this issue Jan 21, 2021 · 0 comments
Labels

Comments

@maciejhirsz
Copy link
Contributor

To keep up with increasing traffic, we should split out a new service from the current backend that replaces the /submit endpoint. This new service should take ownership of JSON deserialization of incoming messages from the nodes, discarding messages that Telemetry does not need, resolving chain multiplexing (this will likely need some two-way communication with the main backend when a new node connects), and then forwarding those messages using a lightweight protocol (Cap'n Proto or Protocol Buffers) to the main telemetry backend.

Unlike the backend, which needs to have a single instance to keep track of all state changes, this new service should be stateless and therefore we should be able to spawn multiple instances of it behind a load balancer. This would solve the two bottlenecks we're currently having: the number of concurrent connections going to the backend, and the CPU use that comes from IO switching and JSON deserialization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant