Shutting down the hosted Hub infrastructure

[carsonfarmer]

What’s the Textile Hub?

Built on Libp2p, IPFS, and Filecoin, the Textile Hub provides a centralized API to three peer-to-peer stacks:

• ThreadDB: A server-less p2p database built on Libp2p
• Powergate: File storage built on Filecoin and IPFS
• Buckets: File and dynamic directory storage built on ThreadDB, Powergate, and UnixFS.

The Hub was built to on-board developers to the Textile stack who didn’t want to run their own ThreadDB peer, IPFS peer, or Powergate service. It also serves as a large ThreadDB relayer and encrypted backup service.

Since it’s release in 2019, the Hub has become… well, a true “hub” of activity:

• ~100,000 developer accounts created
• ~113,000 ThreadDB databases created
• ~73,000 IPNS keys created
• Hundreds of thousands of Buckets created on ThreadDBs, resulting in one of the biggest IPFS nodes on the Internet (7.8TB)

Why are we shutting it down?

We’re sad to let the Hub go, but it was never meant to be a long-term solution for folks to build on. It was meant to be an on-ramp for developers, and it no longer serves that purpose in a sustainable way. As it turns out, centralized on-ramps tend to be sticky!

However, we remain completely dedicated to building a database layer for the next web. That’s why last year we kicked off Tableland, a decentralized database and network built on SQLite and the EVM. The knowledge and experience we gained over the years with the Textile stack has been instrumental in the design of Tableland. At its heart is a ruthless focus on sustainability and developer experience. Unlike the Textile stack, Tableland will be an incentivized network, removing the need for costly federated relays and backup nodes in the long term like the Hub.

What’s happening with my Hub data?

Textile has been a member of the IPFS/Filecoin community since the early days, and we’re here to stay. That is why we are committed to working with folks to ensure a smooth transition off the Hub. Our intent is to shut down the Hub for good on January 9, 2023. At this time, all ThreadDB and Bucket data will no longer be available, and will subsequently be removed.

If you need more time to migrate, no problem! Just let us know by opening a support ticket on our Discord.

Here’s how it is going to play out:

• On November 28th, 2022, the Hub will stop supporting Bucket archives to Filecoin.
• On December 12th, 2022, the Hub will stop accepting new developer accounts.
• On January 9th, 2023, we will take the Hub offline and delete all ThreadDB and Bucket data.

We are here for you every step of the way, and there are multiple options for migrating data off the Hub:

1. Migrate data to Tableland
   1. It’s just SQL! Get started with the docs here.
   2. For questions, create a support ticket in our Discord.
2. Use one of these other awesome solutions
   • Filecoin is ready for prime-time and doesn't require tools like the Hub make it useful in your application.
   • NFT.Storage and Web3.Storage are crushing Filecoin data onboarding.
   • Estuary is another great solution for getting data onto Filecoin and pinning it on IPFS... the easy way.
   • Filebase, Fleek, and Pinata run excellent and widely used IPFS pinning infrastructure.
3. Deploy your own Hub
   1. See here for an example of deploying the Hub to Google Kubernetes Engine.
   2. For questions, create a support ticket in our Discord.

What’s happening with my FIL balance?

A small number of you transferred FIL into the deal making address that was associated with your Hub account. If that’s you, and you’d like for us to transfer that balance to a new wallet, let us know by opening a support ticket on our Discord, by January 9th, 2023. After January 9th we will require you to verify ownership of the account email address. Any remaining funds will be donated to future Filecoin hackathon teams.

FAQ

1. Will Hub code still be open source? YES
2. Can I run the Hub on my own infrastructure? YES
3. Can I get help deploying my own Hub? YES

[carsonfarmer]

Migrating from the Textile Hub

If you are currently storing files and/or data on Textile’s hosted Hub infrastructure, it is time to think about migrating your data elsewhere, or hosting your own Hub infrastructure. But don’t worry, we are here for you every step of the way, and there are multiple options for migrating data off the Hub:

Migrate data to Tableland

It’s just SQL! Get started with the docs here. We also have a number of command line tools to help you migrate data via SQL statements, with more tooling improvements each week. For questions, create a support ticket in our Discord. Tableland is our core focus right now, so we’re happy to onboard you to this exciting web3 option.

Migrate to one of our partners

Filecoin is ready for prime-time and doesn't require tools like the Hub make it useful in your application. To leverage all the power that Filecoin has to offer, you might want to consider NFT.Storage and Web3.Storage, both projects that are crushing Filecoin data onboarding. We also highly recommend Estuary, which is another great solution for getting data onto Filecoin and pinning it on IPFS… the easy way! Additionally, IPFS ecosystem players like Filebase, Fleek, and Pinata run excellent and widely used IPFS pinning infrastructure. And these groups all have payment plans that are comparable to what the Hub was offering.

Migrate to your own Hub

For folks that want to continue to leverage a “Hub” like experience without the hosted infrastructure provided by Textile, you may choose to self-host a Textile Hub. As with all things Textile, the Hub (and Buckets, ThreadDB, Mailbox, etc) is open source and ready for you to clone, fork, and/or deploy on your own infrastructure.

If you are only using one or two of the hosted services provided by The Hub (say, Buckets or ThreadDB), you might want to consider deploying and hosting only those services you require. For instance, it is relatively easy to host your own Buckets API, or to host a ThreadDB service and relayer. Instructions for deploying these open source libraries are available right within their READMEs:

• Buckets — https://github.com/textileio/go-buckets#getting-started
• ThreadDB — https://github.com/textileio/go-threads#getting-started

If you require a more full-featured Hub setup, then you’ll probably want to deploy your own Hub to Google Kubernetes Engine. Before going down this road, ensure that you, or someone from your team, is familiar with Kubernetes deployments (or is able to leverage Docker and other infra tooling more directly).

There is a getting started guide in the README of the Hub GitHub repo, but this GKE setup for the Hub is probably the most straightforward way to deploy your own Hub infrastructure. In theory, you can get the Hub deployed without much more than starting in that folder, and executing kubectl -n <your-k8-namespace> apply -f .. The envs in those YAML files are well configured, so it should “just work”. In practice, however, your setup may be more nuanced than that, so there are some other things to consider.

The above YAMLs were testing using both Azure managed K8 and GKE managed K8. For those interested in using EC2, if you guide your installation setup by reading the YAML files in that setup, that should provide you with the recommended setup. Everything you need to know should be in those files. However, the Textile team highly recommends GKE if that is at all possible. The cluster management and auto-scaling of cluster resources are pure gold!

There are a few items to consider when deploying. For instance, to disable the Billing API, you can simply not run the Billing Daemon. The default value for the Billing API in hubd is "not set”. Related the CustomerIO service can also be “ignored”. For example if you wanted to put The Hub behind your own API (which is likely what most folks will want), you'll just have one "account". You can confirm that account manually by setting this flag in code, and then visiting http://localhost:8006/confirm/[secret].

As always, for questions or support on any of these options, please create a support ticket in our Discord, and our team will do out best to get you going. Unfortunately, our team is not offering dedicated support for running your own Hub instance. This is an open source project, and there are many community members that are able to help, but we are not offering dedicated support services at this time.