Federation

[victorb]

I opened a preliminary PR (#10) for Federation but probably best to go via a issue first, to better enable discussions around it. Here is what I've been thinking so far.

Old proposal: https://gist.github.com/victorb/82ace9e6fe7adf578833527b8b94f914

New proposal:

Open-Registry Federation

Summary

Open-Registry as a crowdfunded registry won't be able to reach the same scale
of npm inc registry without raising significant amount of funds. What we can do
however, is setup a federation of registries which would significantly lower our
operating costs and also give the users the benefit of faster performance and
local resource sharing.

The model of federation proposed here will decentralize the storage and
transfer of tarballs first, as it poses an easier way of getting started with
federation for Open-Registry.

Once implemented and used, we can start focusing on research about federated
publishing as well.

Motivation

• Lower bandwidth/storage/hosting expenses
• Faster performance for participants
• Resilience
• User Control

Constraints

• Needs to handle npm namespace to be npm compatible (global + scoped packages)
• Handles propagation of package updates
• Anti-spam measures if needed
• Cheap to run (Federated version needs to be lightweight)
• Downloads metadata + tarballs on-demand
• Space aware (never cause "out of space" state by itself)
• Users should be able to benefit from federation by just changing the registry
  url (DNS/HTTP federation)
• Users can benefit further by running federation software locally
• Runs offline

Use Cases

• Individuals can find closer mirrors
• Teams can share the same mirror
• Companies can deploy on-prem mirrors
• Organizations depending on Open Source packages can help host packages
• Registry will continue to work even though main mirror is down

Security

• Malicious people might try to be a part of federation honestly, until they
  aren't honest anymore
  • Content-addressing helps address this specific issue
• Tarballs are verified when downloaded via content-addressing + popular
  clients (npm + yarn) checks the checksums before extracting, so mutating
  served tarballs is hard without client detecting it

Practical steps

Ok, so the working plan is the following:

• Write a lightweight proxy to run locally
  • Should connect to set of bootstrap nodes run by Open-Registry, and can
    find other nodes via those bootstrap nodes
  • Runs a local HTTP proxy that fetches the right package when needed
    • Support metadata route
    • Support tarball route
    • Support index route
  • Otherwise runs in the background, potentialy helping others finding
    packages

This is the small, MVP version to ensure the idea is viable in the wild.

First step towards federation is having the metadata index centralized with
Open-Registry while tarballs can be served from anywhere and anyone.

Plan is to use ipfs-lite by @hsanjuan to start a embedded libp2p node that will
expose the traditional registry interface as HTTP endpoints.

The software will connect to the central registry to find out the latest root
hash and also listen for any changes, automatically update it's local pointer
when Open-Registry's pointer changes.

The root hash can be found in multiple different ways, depending on the
environment of the software.

The software will basically be a resolver for (packageName, packageVersion) =>
IPFS hash via it's local proxy.

CLI interface

$ open-registry --federate
                --share
                --update-type=<http|dns|ipns|pubsub>
                --offline

--federate <multiaddr>   - Connect to already running instance and use it's
                           root hash.
                           Default: /dns4/npm.open-registry.dev/tcp/6736

--share                  - Enable other peers to connect to you and download
                           public packages.
                           Default: true

--update-type            - How to get the latest root hash from Open-Registry.
                           Default: http

--offline                - Don't do any connections, use last known root hash.
                           Default: false

Example usage:

$ open-registry
Connecting to npm.open-registry.dev
Getting latest hash via HTTP over TLS
Started sharing downloaded public packages with others
Started HTTP server on http://localhost:6736 # mnemonic: "open" in T9
...
Currently connected to 3 peers
Upload/Download [current/total]: 32kbps/0kbps [3mb/7.3mb]

Pointing your package manager to http://localhost:6736 should now allow
you to download and install packages on-demand, while caching them and serving
it to other users who are trying to download them too.

Federation Protocol

When the federation software gets started on the users device, it connects to
the main registry.

Once connection has been established, it asks for the latest version of the
registry (just a pointer), and saves it for future use.

Concurrently, it starts a HTTP server locally.

Now the user can point it's client to the local HTTP server

Requests will be proxied via the latest root hash the federation software knows
about, and cache fetched data

When the root hash of the main registry changes, it publishes it via the
following ways:

• txt record on npm.open-registry.dev under the format "registry-hash="
• Under property hash in response to a GET request to npm.open-registry.dev
• Send the hash via the topic npm.open-registry.dev on the used libp2p
  network
• (maybe) updates the IPNS name that the main registry uses

If the local client makes a request for a package that doesn't exists in the
local root hash, the client needs to make a request to the central registry to
download the package. After this is done, the package will be included in the
new root hash, and can therefore be downloaded by the local client without any
requests to the central registry.

Simulator

First step of the federation setup is creating a suitable testing environment
where we can run tests about how well the federation is working.

Simulator should start with running the following scenarios:

• Starting one node connected to the main registry, downloading packages
  for one project. Run two times and ensure second is faster than first
• Start two nodes. Make sure wanted packages is cached in the first one.
  Download packages without being connected to the Internet in the second one.
  Ensure second node is faster than first node as connection should now be
  local.
• Start five nodes connected to internet and download packages for one
  project. Compare to starting five nodes where only one is connected to the
  internet. Second phase should be less bandwidth intense as packages are
  only downloaded from the Internet once instead of for each node.

More elaborate schemes can be created in the future.

Bootstrap nodes

Open-Registry will run a couple of bootstrap nodes. These are responsible for
being accessible to the federation nodes and provide the data for metadata and
tarballs if the federation nodes doesn't have it locally.

Metrics

Both the bootstrap nodes and the main registry index should publish metrics in
the Prometheus format to be collected by the metrics gatherer. These metrics
will eventually be made accessible via a public dashboard.

For the federation nodes, we can offer opt-in metrics in the future, so we can
see the health of the federation.

Existing infrastructure migration

The current Open-Registry is just one instance which is the main Open-Registry
index. With federation, the architecture would change to add another component
which would be the federated instances. We have more flexibility on where to
place these but are in no rush to add them currently.

Potential Issues

• Lockfiles contains direct location-based URLs
  • hard for project to migrate without having to rewrite their lockfiles
• Efficient and fast lookup in the IPFS network
  • private networks solve this but brings it's own problems

Drawbacks

• Requiring software to be installed and run in the background for people
  wanting to take advantage of it
  • ^ could possibly be solved with HTTP/DNS routing, but initial routing will
    be centralized in that case and require internet connectivity

Alternatives

• Continue to run a centralized service
• Skip federation and start researching a architecture for fully decentralized
  registry for both tarball and metadata
  • Probably a too huge of a undertaking right now

Unresolved Problems

• Using a IPFS private VS public network
  • private network will be faster to bootstrap + finding content
  • public network gives us a bigger reach and ability to download content from
    other nodes
  • Should benchmark and see which one is faster (although private network is
    pretty much sure to be faster, would be interesting to see how much)

Future

• After implementation of the tarball federation, further research should be
  done on how metadata can be federated as well
• Research URL scheme currently used to define packages
  • Right now, entire ecosystem is in one namespace (lets call it the npm
    namespace)
  • Things are referred to as class-is directly in the package.json and
    lockfiles
  • We'd like to support multiple registries by doing something similar to
    /registry.npmjs.org/class-is instead. More verbose, but more accurate and
    flexible

[max-mapper]

I don't have time to work on this right now, but here's an old thread from a similar initiative I worked on depjs/dep#8

[victorb]

Thanks a lot @maxogden, will check that out.

[retrohacker]

Running a global network of the scale of the npm registry will be impossible to do with just being funded by the community as the costs will be too high.

Wonder how far we can get with cloudflare + cloud storage.

Will be experimenting with this in the coming weeks and will report back :-)

[victorb]

@retrohacker thanks, appreciate it, ping here once you have some results to share :)

That said, I do think that even if we find the fastest CDN, we can make it faster for people by having a federated model. But CDN in front of the metadata registry would still be a good idea.

[victorb]

I've updated the initial issue here with an updated version of the proposed federation, will also bump it on the roadmap.

Old proposal can be found here: https://gist.github.com/victorb/82ace9e6fe7adf578833527b8b94f914

[marcusnewton]

Build it using Holochain, it's exactly what you need for distributed (fully sharded) storage and cryptographic security

[retrohacker]

@victorb as promised, circling back to report on cost.

Self hosting on cloud providers turned out to be reasonable. Our GCP mirror ended up costing ~$300 to do the initial mirroring (pulling 5TB of data through cloud functions and into cloud storage).

Once the files are sitting in storage (multi-zone within the US), the cost is ~$6 a day. That includes the instance that is sitting there watching the CouchDB stream from the npm registry to keep the mirror fresh. The breakdown is $3.46 per day for storage and $2.28 for the compute instance.

Cloudflare functions (where we are doing our load balancing) costs $0.50 per million invocations.

BTW the service is up and running if you want to give it a try: https://freajs.com