PoC solution for orbitdb pin requests handling

[msterle]

If the pinning nodes and ipfs are decoupled, pin requests will arrive over pubsub to all pinning node instances. How to determine which pinning node(s) should handle the request?

Some starting ideas:

• Add in pubsub load balancing mechanism (producer/consumer model)
• Create a peerlist consensus mechanism to implement DHT on pinning node orbitdb peers

AC:
Working solution in docker-compose (with separate ipfs container)

[zachferland]

does this assume one ipfs node? if not it may mean separating libp2p out, could we split some points out in to a shared architecting story that is more problem focused than solution focused ?

[zachferland]

Im thinking if we want to better balance immediate needs and long term solutions, may want to reconsider solution from last sprint as one possible option still (with addition)

Feels like changing path too quickly could be a bit risky, especially as more clarity about which problems we need to solve is just starting to surface, no benchmarks to compare sols against, some path dependence, and real tradeoffs so far in the solution space. And we have really scattered requirements for what other's infra will need. With a bit more work/research feels like we could map out a more iterative approach now, and solve each problem. Plus some implementations/work may come from others with time.

Original short term fix proposed last sprint but with an addition has reasonable tradeoffs and low risk. This would re use existing infra, and one simple service add, and minimal hooks/interface from ipfs.

Basically relay pin db message internally over existing internal pubsub as said before. But also keep a shared window of seen pin db messages (pretty sure these are unique). Could just be ttl key vals in redis even, since only adds, no issue with shared writes/reads. Now the system wouldn't process pin db across multiple times across nodes. General network pubsub traffic would not have too much overhead added over what is seen already (especially few ipfs nodes running / partitions), plus client unsub once they see messages they need.

But yes this solution alone would be limiting for other service runners. It would support personal nodes, running our entire docker compose infra as is, and running one 3box/orbit node with one remote ipfs node.

Any thoughts? Maybe we could solve all this in sprint, but feel like implementation to unblock highest priorities and then room to prioritize out other requirements/solutions may be worth it.

[zachferland]

Also starting to realize my original thinking about being able to run against (or share) existing ipfs infra may be misguided. Basically as we were talking about, those clusters are only good at serving gateway to remote services. They are pretty unsuited to handle pubsub right now, some dont run it even really, especially across nodes. Thinking the only way to really run against an agnostic ipfs service, is to pull out libp2p/pusbub handling to 3box/orbits nodes, while running ipfs infra adjacent. Until then, the first providers will probably will run out ipfs wrapped together or with a single ipfs node anyways. Otherwise our layer has a lot of requirements for ipfs layer that likely dont align with existing deployments.

[oed]

I think circling back now to a solution that we know won't work in long term / for other service providers easily is not a great direction. I feel like the path suggested by Mike is more solid long term and seems like the logical separation of concern. There will likely be problems to solve with that approach, but at least we will be solving the right problems.

Also starting to realize my original thinking about being able to run against (or share) existing ipfs infra may be misguided.

Don't think this is true. Infura and others are already working on making pubsub work better with their ifrastructure and we should trust that they are able to do a good job there. I think we can also help them by communicating our needs etc. as we move in this direction of separation of concerns. Anyway I think just the ability to choose go vs js ipfs is reason enough to separate pinning-node and ipfs node.

[msterle]

I agree with @zachferland in that we are introducing significant changes to our deployment, and without a way to benchmark it aside form putting it into production, we're opening ourselves up to risk. This risk can be mitigated by doing a canary or blue-green deployment, at the cost of additional time (we don't have a setup for these yet).

In regards to bringing back #279: I'm sure there's some way around the limitations @oed has brought up, but even wrapping my head around it is difficult because we're breaking the high-level abstractions like "connections", "peers" and "pubsub" that have been implemented for us. Then, when they stop working, we're trying to reimplement them in a different way (for example, just calling our messaging layer a pubsub is misleading, and what led us to believe we could easily bootstrap a 3rd party node).

To rephrase @oed's points, by sharing an IPFS node ID, we're going to have to find a way to patch these two abstractions, that should work out of the box:

• pubsub: we're going to have to propagate pubsub messages between containers (not only pinning room, but orbitdb rooms)
• connections: we're going to require direct connections between specific nodes and 3rd party nodes which isn't easy because libp2p sees them as the same node, and they are behind a load balancer anyways.

It feels like solving these would not be insignificant in itself, and we are putting in work knowing we'll be removing it later, so I would argue that it's not incremental or iterative. If we do want to pursue this, though, we should have a clearer understanding of the motivation behind pushing this out the door rapidly.

[zachferland]

https://www.notion.so/threebox/Technical-Planning-7ffee27704b34b20be8f01d5f6567ce1?p=1f4d88ecb0f444f88679513981a4b696