Which "premium" version of hetzner-k3s would you be more interested in? SaaS or a more advanced CLI tool?

[vitobotta]

Hello everyone!

I am planning to build a new premium tool called the "Cluster Ninja," designed to extend the capabilities of hetzner-k3s with an array of additional features including but not limited to:

• Support for popular API based providers, not just Hetzner (like DigitalOcean, Linode, Scaleway, Vultr and more)
• A "bring your own provider" feature, allowing you to integrate servers purchased from any provider, only requiring SSH access
• The ability to create high-availability, multi-cloud clusters affordably
• Advanced customisation options for k3s and Kubernetes overall
• Streamlined installations of essential cluster components like CNI, ingress, storage, monitoring/logging, and more
• In the case of the SaaS model, an integrated marketplace for simplified installation of numerous open-source apps

While I love contributing to open-source and free tools, realistically, to expand the project and deliver a significant impact, it needs to be backed by a sustainable revenue model.

Here's where I need your input: would you be more interested in a SaaS version, providing a unified platform for managing clusters across all your providers, or would you prefer a standalone, more advanced CLI tool?

The proposed SaaS platform (akin to a managed Kubernetes service but on your infrastructure) would oversee cluster creation, upgrades, reconciliation and more, but would require storing (in encrypted and secure way) information like API keys and kubeconfig files in order to create infrastructure and manage clusters. In contrast, the CLI tool would be more autonomous, putting all operational responsibility on you while guaranteeing your cluster remains entirely private to you.

I would appreciate your insights on:

• Your preferences between the two tools and why
• Potential concerns or feedback about either version
• Desired features, not already mentioned
• Your willingness to pay and suggested pricing models
• Pain points you've experienced with existing services and what you'd like to see improved
• Your role in your company (or anyway as potential customer of this tool)

I have been a developer for over 25 years (working with Kubernetes for 5+ years), but I'm relatively new to the business-side of things. Therefore, I genuinely value your input and guidance as I wish to build a tool that is genuinely useful and widely adopted.

Thank you in advance for your time and valuable insights!

[mrsimonemms]

My preference would be for a SaaS replacement. I started some preliminary work on https://github.com/k3smanager/ (and registered k3smanager.dev) for the purpose before finding your Hetzner-k3s library. If you did go down the SaaS, I'd be very happy to donate these two things for the effort.

[vitobotta]

Thanks for chiming in! Great to hear you would be interested in the SaaS option. Are there any features you would like to see not mentioned in the original post? Also, what do you think would be a fair pricing? Thanks!

[Funzinator]

Maybe a good play to mention that there's also Gardener. Gardener supports the big (expensive) cloud providers and provides a "vanilla" K8s cluster that abstracts away the vendor-specificy. 23technologies uses Gardener to provide okeanos. Okeanos basically adds Hetzner (hcloud) support as a Gardener plugin.

So the SaaS offering should have a different focus/use-case to not compete with this solution.

I played with okeanos and ditched it in favor of hetzner-k3s because I didn't want to be dependent on a provider that could take away the fancy dashboard anytime - leaving me with a not-so-easy-to-manage cluster ;-) - also I had the feeling that okeanos was not supported/updated anymore. Didn't support ARM, didn't support newer k8s versions last time I checked. That's why I went for hetzner-k3s.

So I would prefer a standalone, more advanced CLI. If you create a SaaS product, I'd prefer if it has an option to self-host for similar reasons.

I like the idea of multiple providers. Especially the smaller ones are not served in terms of Kubernetes tooling. If hetzner-k3s is able to manage Scaleway and others, and even supports hybrid clusters spanning the two providers, that would be gold!

[vitobotta]

Hi, and thanks a lot for your feedback! I wasn't aware of Gardener so thanks for mentioning it. What confuses me about it is that the home page says "Managed Kubernetes" but there is no pricing. So is it a managed service or not?

Thanks for expressing your preference and expressing why. Feedback like this is very important for me to choose the right tool to invest my time in :)

[Funzinator]

Gardener is an open source project that can be used as a SaaS-service but is not in itself a SaaS offering. You would have to install gardener yourself (in a Kubernetes cluster - they call it "Kubeception") to use it. That's what 23tech did with okeanos. Basically installed it and added the hcloud plugin.

It might even make sense to base the Cluster Ninja on Gardner - and focus on adding the providers like scaleway/Vultr, etc. You would save yourself the hassle of writing your own dashboard/UI ;-)...

[vitobotta]

Thanks for the explanation, it's much clearer now. I would prefer building my own platform because I would have more control and also because I have my own ideas of how to implement things so I would feel limited by using someone else's project as base :)

[michaelfresco]

Hi Vito, what I personally think is still an interesting product, or extension of hetzner-k3s, would be to run hetzner-k3s from inside a kubernetes cluster, to create other clusters. Then instead of running "hetzner-k3s create --config cluster.yaml", you do "kubectl apply -f cluster.yaml", but it would have the same effect eventually. (Similar to Crossplane: https://crossplane.io, or if you will, similar to Argo CD) This opens the door to true GitOps. If you already run Argo CD this approach makes a lot of sense! (you could manage everything from Git)

[vitobotta]

Thanks Michael, like I mentioned on Twitter it's something I hadn't thought before but it's worth exploring. I wonder how many people would be OK with the additional cost of a management cluster though :)

[michaelfresco]

Maybe it could be indeed a service / open source model? Tbc. I run a tiny management cluster for ArgoCD. This is really a lot of fun. Managing apps GitOps style is totally awesome :)

[vitobotta]

It's a possibility. What kind of pricing would you envision for this?

[Taronyuu]

Coincidentally, a short while back I asked about any 'bring your own servers' Kubernetes provider on LET (where you replied too), so I feel somewhat targeted by this discussion :-)

I run my own freelancing business and I would pick the SaaS platform if I had to decide for anything related to client work. The idea of 'you' having access to my kubernetes cluster and able to manage it without me having to worry about it sounds the best. However, a CLI tool would be much more preferred for my own work where I'd rather not rely solely on a SaaS application. Additionally, especially in Europe you might run into GDPR issues with a SaaS tool.

At the end, I would ultimately vote for the CLI tool, and I would gladly pay money for a CLI tool that I know will be updated and might even include support (up to a certain level of course). This way I know that it will keep on working and I am not relying on one companying managing my servers. Maybe you can consider building it a CLI tool (paid of course) but also offer it as a managed platform. That way people can choose what they want. I somewhat assume that the hardest part won't be abstracting it in such a way that it can be a CLI tool and a SaaS platform.

Ps. I came here via your Reddit post, usually I would skip these kind of posts. However, given your awesome tool I decided to chime in!

Edit: I voted for the CLI tool, but personally speaking, I wouldn't really require a lot more tools. If I can have an easy HA K3S cluster available with the guarantee that your tool will manage it in the future, I am golden! However, storage would be appreciated as addon!

[vitobotta]

Thanks for the awesome feedback! I might implement both at some point if there seems to be demand and you are right in that some stuff can be shared between the two projects. However developing what I have in mind takes time and I have also a full time day job, so I need to pick one and start with that, hence this poll. So far it seems that the CLI option is in the lead :)

[hbitsch]

I would prefer an advanced CLI version. Giving away the cluster management to a third party in a SaaS model would pose a significant security risk in terms of supply chain attacks. What would be the advantage of running k8s on my own infrastructure when somebody else holds the keys to it?

[vitobotta]

This is exactly the problem I was worried about with the SaaS option. I actually had planned to work on the SaaS originally until my boss raised the same concern so that was food for thought and a big reason why I created this poll. I wouldn't want to invest time building something that most people might not use because of security concerns, which are totally understandable. Most people have voted for the CLI option so far, so it seems that might be the direction I should take. Let's see how it progresses. Thanks a lot for you valuable input!

[bluepuma77]

We run our SaaS on Hetzner bare metal servers (with Cloud Loadbalancers), to be in full control and have (hopefully) the most secure environment. No customer in the next VM that escapes the isolation and takes over the host.

We have been looking into k8s/k3s/RKE2 multiple times, but it just seems way more complicated than Docker Swarm, which currently satisfies 95% of our requirements to run 10+ servers. It's all created manually, again we have full control.

Personally I am a bit afraid of those management tools, because what if something breaks during an update, I would be totally lost, because I don't know the (k8s) inner workings anymore. But I guess that's just a thing in my head, everyone else does it.

In general for us it's rather CLI than SaaS. With bare metal support to reduce cost.

For me the next big thing is Confidential Computing, where even the RAM of the VM is encrypted. I am sure it will arravice at some point at Hetzner, as the other big players already support it. For me it's the final piece of the puzzle to bring even security-concerned corporate IT onto the cloud. Not sure if there is anything to do for you, maybe check Edgeless' Constellation.

[vitobotta]

Thanks for expressing your preference! Yeah k8s is definitely more complex than Swarm but way more capable if you are ok with the learning curve. I wasn't familiar with Edgeless' Constellation, sounds like a very interesting concept.

[bluepuma77]

Just learned that the first "Confidential Compute"-enabled dedicated server (EX130) have landed at Hetzner in October (link)

[NotANormalNerd]

Hey Vito,

One concern I want to raise is, that you shouldn't compete with rancher in terms of SaaS (all the features you wrote are already available from Rancher, supported by SuSE). I use k3s for my private cluster and OpenLens and Kubectl for the "Management" part.

What I really like about hetzner-k3s is that it is Hetzner specific. So in my opinion, I would rather go for the advanced CLI. If you can make a business out of that, I don't know, but you can definitely setup a Patreon (next to the Github sponsors) and probably get some people to sponsor you. Would I pay 5€ a month for this tool, absolutely. Would I pay a 15€ subscription service, probably not. (Privately) On the B2B side, you are probably either not big enough or in a area where there is really little investment into tools like this. YMMV.

In terms of features for this tool, I would probably love to have a possibility to setup Helm Charts from the get-go automatically: https://docs.k3s.io/helm

[hbitsch]

Maybe you want to look at Portainer for a comparison. They are like 3000 USD + 300 USD per node and all per year. Obviously they have their audience.

[NotANormalNerd]

@vitobotta I wrote an article for my former company about Hetzner and Rancher as we used that to setup our Clusters for our SaaS Stack: https://tech.mecodia.cloud/2021/01/12/running-a-tightly-integrated-hetzner-cloud-kubernetes-with-rancher/ and that was 3 years ago. My guess is that is has only gotten better and easier. (Things like fleet management and stuff was implemented after I left the company).
Actually thinking back: You know most of that stuff because I remember getting a lot of info from your blog back then ;-)
Here: https://vitobotta.com/2020/10/30/kubernetes-hetzner-cloud-rancher-custom-nodes/ and here: https://vitobotta.com/2020/11/10/kubernetes-hetzner-cloud-rancher-node-driver/

In regards to the pricing, 5€ per month would be my support/aknowledgement for this CLI tool as I only use it privately and then maybe once a quarter. My question is rather, where is your market? If a company can afford a DevOps Engineer to run their infra, they will most likely have money/org to pay and negotiate the "contact us" pricing of Rancher/Suse. Otherwise they will go the Managed Cluster Route and go EKS or whatever they are called. Also Rancher itself is free and open source, the "contact us" pricing is for support.

Okay, lets say startup. No DevOps Engineer, but a Engineer that does some programming and some deploying. Probably also goes the AWS/GCloud route and uses whatever is available for free and open source, like we did in my former company.

I read here about Freelancers, they are probably a good niche to go for, but my 2ct are that you wouldn't pass my 100.000€/year "worth going for" revenue bar, as there is sooo much competition in this space, from free and commercial providers.

Now, considering that this is what the open source tool already does, and that the paid version would do the same and more with more providers and with more features and without requiring a management cluster, allowing you to use these features even with very cheap providers saving money, is 5e/mo really a fair price?

Fair is always relative. Fair for using it privately, once a quarter (so 15€ per use), yes from my side. Also I don't need 100 providers, I need hetzner. But the I just need it for my private stuff and maybe some homelab stuff. That said: If you get this to run against a Proxmox Cluster, that would be something I would interested in.

No to demotivate you. Go for it, maybe you find a USP that will get you a big enough market, but cheap should never be a USP.

[ddorian]

there is sooo much competition in this space, from free and commercial providers.

I've only seen free and enterprise, never a $20/month/server option.

[NotANormalNerd]

@ddorian Do you mean 20€ for IaaS or 20€ for the software?

[ddorian]

@NotANormalNerd for software to manage kubernetes on your own servers.

[prometherion]

The CLI tool would be used by developers or operators (not the Kubernetes ones) and not keen on paying for it: Open Source has forged people in having tools for free, and maybe even helping them in their daily work for which are highly paid, and not keen in giving back such as donations, or by paying a subscription (I read something such as 5 bucks/month, and honestly, no comment).

I would suggest you offer a SaaS that would be useful for managers and companies that need to operate a fleet of k3s instances, such as having a single pane of glass of their clusters, as well as metrics, etc.: COSS could be interesting, although I would highly discourage open source since it's not a business model, rather, a go-to-market strategy. If the potential customer is not interested in having an externally managed service, such a SaaS, you could sell it on their own infrastructure.

[vitobotta]

Thanks! unfortunately I have to agree. These days people expect a bit too much, like you can have everything for free or close to nothing :( Thanks for expressing your preference :)

[NotANormalNerd]

Open Source has forged people in having tools for free, and maybe even helping them in their daily work for which are highly paid, and not keen in giving back such as donations, or by paying a subscription (I read something such as 5 bucks/month, and honestly, no comment).

@prometherion This is a two sided sword. Nobody made anybody open source this. But then complaining that people use it as intended is really low. I agree on the general point that there is no market for a CLI tool. I don't know if there are any successful paid CLI tools out there.

@vitobotta I don't know where this sentiment comes from. People are willing to pay money for something that is useful and has more convenience than something else. Piracy gave way to Spotify, Steam, Netflix... But they are a lot more convenient than piracy.

I have been a developer for over 25 years (working with Kubernetes for 5+ years), but I'm relatively new to the business-side of things. Therefore, I genuinely value your input and guidance as I wish to build a tool that is genuinely useful and widely adopted.

Who is your market? Individuals? Companies? Technical or non technical people? That all affects of what kind of software you build and who you market it to. And crucially how much money you can ask for. That's why every bank wants a Business Plan before they give you any money.
Asking for guidance and then being sour at the first person that gives you a data point (private, technical guy) is really... well. But maybe I am not your target market and just a freeloader.

[bluepuma77]

Coming from a startup perspective: decide if it should make you rich (scale and exit) or just enable income (lifestyle business).

I think I agree to what has been mentioned before: software is either free or has corporate pricing (and reputation). Just as an example, nginx and traefik cost $3000/year, it seems insane.

The challenge is to find a product market fit or even your target customer, and I don't see that beside a niche. Startups mostly use the big providers, they have managed k8s. The big corporates use the big providers or run it by themselves, but then they probably have 2+ FTEs to manage they k8s. I think there are only very few companies that are in the situation to need your product. And then you have the hobbyists, that only pay like $5/month, if anything at all.

Playing devils advocate: who should buy your product? No one knows it, you got 25 watchers, why should they buy your solution? Why should anyone with money trust your solution, seemingly a company with a single person. As mentioned in my other post, my issue is risk: who ensures me that every upgrade of my infra will work, what if an upgrade fails? I am responsible, my reputation is on the line, "no one was ever fired for buying IBM", but maybe I am, if I use a small tool that accidentally kills our production system.

Just want to prepare you for a long journey, it will be hard and tiresome. You need to find a USP and your target customer for your product. Usually startup is a lot about testing, measuring and adjusting. First thing you should measure is how many people actively use your tool. But make sure to communicate that clearly and that it can be disabled, otherwise you might loose a few friends here in the open source community.

[gimafach]

Most of cloud providers have their way to bring up a cluster. Before extending towards them, in my opinion it is better to improve what Hetzner is still missing. I am trying to use your tool to avoid any security concern from the beginning but at the same time to have an initial GitOps by managing clusters in a declarative way. At some point in time, using terraform, or iPXE or API the infrastructure shall be setup before installing k3s/k8s.
Someone is proposing a cluster API for Hetzner, and this is a way to go for GitOps cluster management for k8s. But we need a management cluster to start from, anyway.
What I am looking (and pay for):

• I create my YAML file for the clusters in git (e.g. github), and the workloads on them
• I create user data config as required by 'vitobotta-image'
• I create my project and token in Hetzner
• I create with hcloud a server with 'vitobotta-image' and with user data it requires, including tokens and git repo link where clusters YAML files are
• Auto-magically the cluster is up and running
• Any changes to git will be reflected (e.g. ArgoCD or flux way) both on clusters and workloads on them
• enable_encryption will be managed by wireguard VPN connecting nodes, not only inside CNI

The 'vitobotta-image' will be hardened by this community, and tokens/kubeconfig will be encrypted by a pass set in user data.
Rootless access to it with a user and SSH port set in user data too. k3s node included.
SSH on vitobotta-image only for debug and troubleshooting.
Idempotence if for any reason I delete the vitobotta-image

Nice to have DNS and LDAP on VPN to centrally (in git) manage server/node/svc FQDN and ACL for secured projects without any access from Internet.

[michaelfresco]

Do you think that it would be possible without a management cluster? Would it be possible to just run it all in GitHub Actions? Maybe you setup a 2 GitHub Actions pipelines? (one for cluster updates, and one with a cron, to check that the state is what you want?)

[gimafach]

I guess cluster api will not address K3S, but your code could be the base for an operator in a management cluster to deploy and maintain workload clusters in hetzner. The management cluster could be a self-managed, local kind cluster, or, better, a k3s cluster deployed with your CLI.

No way for GitHub action. GitHub is the source of truth for your cluster config and any workload cluster. But nothing else. Any git repo can be used.

This is in my opinion a nice to have feature. Hetzner has cluster API for K8s; it would be wonderful to have the same for k3s.

[lloesche]

I'm not opposed to a SaaS but the platform would need to be ISO27001 and/or SOC2 certified. It's just a supply-chain requirement that we have from our customers so for us there'd be no way of using a service that has access to our infra without being certified.

I'd also like an advanced CLI with an option for using rke2 instead of k3s and enabling the FIPS140 hardening by default.
Functions like volume encryption at rest, adding Cilium, as well as local NVMe storage support for better IOPS out of the box would be neat.

[GodAtum360]

As someone new to K3S, I'd like more in-built features. For example, logging and monitoring is very very important. And Rancher would be nice too. Also making it a 3 tier solution (with storage like Longhorn) will be awesome.

[vitobotta]

An long due update: Thanks all for the feedback. In the end I decided not to go ahead with the premium version because I was very disappointed by some feedback. A tool like this with even more capabilities makes it easy to save a lot of money compared to managed Kubernetes services with bigger and more expensive providers, yet it seems that not many are ready to pay what I consider a reasonable amount for this. I know for a fact that many people use this open source tool and many have saved a lot of money, therefore some comments I have received have been very surprising to me.

To be honest I was so disappointed that I was not sure of what do with this project moving forward and I took a break.

I decided to continue supporting this open source version as is for the time being and revisit other options later. I just would love to be able to dedicate more time on these things knowing that I can have at lest some income from the effort. I was thinking of multi cloud support, allowing to use even cheaper providers than Hetzner etc I guess this was not supposed to happen for now.

I have a week off work this coming week so I will spend some time on the project and make a new release.

[vitobotta]

I forgot to mention, that if anyone is willing to support the development of this tool, please consider becoming a sponsor if this tool helps you / saves you money :)

[kamikazechaser]

@vitobotta I won't be surprised if your tool has helped create tens of thousands of clusters on HZ. The other alternative projects in this space, which mostly use Terraform, are more complex to set up, which can be a turn-off. You definitely deserve some sort of recompense.

I say go for an Advanced CLI which is strongly licensed (AGPL-3.0) and locks certain features behind a paywall that are more useful to businesses and large enterprises.

Also explore other cloud providers since HZ will most likely bring managed k8s within 2 years.

[vitobotta]

Thanks for the nice feedback! I am enjoying working on the project this week (new major release in a few days!) and I am thinking again about a paid option.... What do you think would be a fair price for a paid CLI with advanced features and support for more providers? Thanks!

[jampy]

To bring in a different point of view:

I am currently planning to move all/most of my company infrastructure to Kubernetes at Hetzner by using hetzner-k3s and ArgiCD mainly. This would be a small cluster (10 nodes at most).

To avoid potential problems with upgrades and such everything would be designed in a way so that with major problems the whole cluster would simply be destroyed and set up again thanks to hetzner-k3s and ArgoCD in a matter of just a few minutes more or less automatically.

My main concern is if hetzner-k3s would continue to get support/updates and so I would be willing to pay something like a 4-figures a year just to "make sure" the current tool continues to get development (in the sense that it continues to work) and perhaps for some extended support.

My company is not big but in scenario I described hetznerk-3s is regarded a critical component and thus be worth it - even without "premium features".

The project will not start before winter and I would get in touch with you in time of course.

[vitobotta]

Hi! thanks a lot for the feedback. I have no plans on stop building the tool. I will release the next major version (v2) pretty soon - just need to iron out some things. Of course, if I get at least some sponsors to support the project it would be even easier to work on it because some sponsorships would help me invest more time in it, as well as spending more time on issues reported by users and offering help. So rest assured that the project will be supported for the foreseeable future. If this project ends up helping you, then I would appreciate if you would consider a sponsorship when the time is right :)