[Megathread] pockethost.io system status, health, and outage reports - Report Problems Here 🌮

[benallfree]

If you experience any issues where you believe PocketHost may be down or experiencing a service disruption, please use this thread to report and discuss.

Also, head to https://discord.com/channels/1128192380500193370/1156285045494005771 for live incident chat.

[benallfree]

Ok everyone, I think there are enough of us experiencing downtime issues that we can pull together to solve it.

Background

Several of you (including me) have noticed that PocketHost seems to intermittently disappear and then come back to life 10-20 minutes later. This has never been observed during development, so it seems to be an issue that only shows up in production and under real use conditions.

Possible causes and solutions

Something about Digital Ocean

Likelihood: extremely unlikely 🚫

Cause: It is possible that the Digital Ocean network or VPS becomes inaccessible for periods of time.

Solution: Change providers

A bug in in PocketBase

Likelihood: unlikely 😒

Cause: It is possible there is an unaddressed bug in PocketBase. I consider this to be unlikely because the outages happen regardless of which version of PocketBase a user is running.

Solution: Identify and report bugs to the PocketBase project

'Bad Neighbor' theory - something about how a particular user is using PocketBase

Likelihood: likely 🤨

Cause: We have already seen instances where a single PocketBase process will spike to max CPU usage because of a badly written or long query. This starves the other processes and has the effect of making the PocketHost server unresponsive.

Solution: Isolate user processes and use cgroups to prevent a single PocketBase process from becoming a 'bad neighbor'. This is worth implementing anyway because it would make the system more secure by disallowing system-level access to PocketBase processes.

Something in the PocketHost code

Likelihood: most likely 🙈

Cause: Humility demands that I recognize my own code is the most likely problem :) When a web request comes in, it hits a nodejs proxy that decides what to do and routes the request accordingly. Most requests are forwarded to a secondary proxy layer (called the daemon) which is responsible for launching PocketBase instances on demand. Then, the request is forwarded to the actual PocketBase instance. We have seen cases where something in this chain of proxies stops responding. It isn't clear why, but it has been observed.

Solution: There is no universal answer to this. The code is not particularly deep or complicated, but it is possible we have a leak somewhere and that the process eventually runs out of some kind of resource.

Current action items

While I think the PocketHost code itself is the most likely culprit, I think that adding support to isolate PocketBase processes and limit resource usage per instance is a good way to create more visibility. If a PocketBase processes runs out of control due to a bad query or really for any reason at all, the nodejs code might not handle that situation well, especially if incoming requests are piling up. So I think I'll focus my energy on implementing process isolation and then see how stability changes.

[BasToTheMax]

I can try, but I can't guarantee anything.

[benallfree]

@BasToTheMax You could begin by figuring out a Docker solution that:

1. Runs PocketBase inside Docker on a specific architecture that matches the host OS
2. Limits the CPU and memory across all running Docker instances. For example, we might want to limit the GROUP of docker processes to 500mb memory and 80% CPU. I don't know if Docker is capable of this type of resource pooling, but that is essentially what cgroup does.

Thank you if you find out those answers!

[cheskoxd]

If you are already deploying the instances with docker containers, what if we set some constraints for the resources?

Like this (docker-compose.yml)

version: '3.8'
services:
  app:
    image:pocketbase:latest
    ports:
      - "8080:80"
    environment:
      - ENV_VAR_1=something
    deploy:
      resources:
        limits:
          cpus: '0.5' # Maximum 50% of a CPU core per instance
          memory: '256M' # Maximum 256MB of memory per instance
        reservations:
          cpus: '0.1' # Minimum 10% of a CPU core per instance
          memory: '128M' # Minimum 128MB of memory per instance

Not sure if this helps or if it's out of topic, but as you said, restricting each instance to a max amount of memory and CPU will be better, and I think this could work?...

At the end docker uses cgroups to limit resources on instances

[cheskoxd]

Also If you are using docker swarm you can specify a limit of resources with

docker service update --limit-cpu=80m --limit-memory=10G your_service_name

[benallfree]

Started separate discussion #269

[nicosql]

@benallfree it seems like long term maybe the solution is to isolate the environments, even if it is your code - your "bad neighbor" scenario never goes away and becomes worse as your repo (and pocketbase) grow in popularity.

Have you considered building on top of something like railway.io?

They use a Docker implementation of Pocketbase and while it is fairly simple to use that is subjective and would still require some Docker knowledge.

I can still see a platform like Pockethost still being very useful to the community given the roadmap and features you already provide.

There is a generous free tier and it would inherently isolate each instance and you could impose some hard capped limit more easily so the experience is stable.

I think what you're doing is great but the product manager in me sees this as non-viable for a single person long term and I want pockethose and pocketbase to go on! That's my input, I will be watching with a learning eye. :)

[BasToTheMax]

Railway doesn't have persistent storage

[nicosql]

@benallfree haha yes, railway.app is what I meant and yes I really like that idea - for people like me (I'm about 8 months into my Javascript / web dev journey coming from years of deep database admin type roles) - platforms like this are a dream.

Yes the services make it easy enough once you're familiar but having to jack with all the minutia (.env, CI, domains, dockerfiles etc..) can really rob that learner's joy.

I'm not sure how else I can help though, got a donation link or something? lol

I do like fly.io though, I've used them for some LiveView stuff and it was pretty slick; I don't think you'd be going wrong with either. Both appear to be doing well financially too and should be around for a good while.

P.S. - @BasToTheMax it is on their roadmap but frankly I do not see this as a problem for two reasons:

1. To me, PocketHost is a launch pad not a landing pad - at most maybe you're throwing up some long running MVP's but not a production app.

2. The beauty of SQLite is it is SO DAMN EASY to maintain / backup / restore - with something like Litestream backing your SQLite implementation you can take it anywhere and not have to worry about such things. All you'd have to do is restore on deployment but like I said, I don't see this as a final resting place - Pocketbase hasn't even 1.0'd, let's not forget that and overcomplicate things.

[benallfree]

@pookiepats That's a nice perspective, thanks for sharing it.

I personally plan to run prod projects on PocketHost some day! :) I think it is already a great solution for small and medium sized apps that don't need to scale to millions. Yes, we have a few stability issues but so would anyone hosting it themselves. These just feel more annoying because we often aren't the cause of the downtime we are experiencing so it feels random.

[nicosql]

@benallfree yes good point, the perception definitely matters.

I meant no offense, in your docs it says it allows for an "eject" of sorts from the platform so I thought the intention was a launch pad.

So let's say I want to help, in the long term; to help you grow pockethost alongside pocketbase - what technical areas would you suggest I focus on if I wanted to compliment your skill set ?

[benallfree]

@pookiepats I didn't take offense, I think it's a great launch pad too :)

If you'd like to contribute, here are some areas where we need help:

• Docs - There are lots of PocketHost features that have sparse or zero documentation.
• Frontend/UX - This is an easy approach to working with the repo and scoring some PRs. It doesn't require much local setup, and just requires knowledge of Svelte
• Backend - For the truly brave, there are many many backend features and ideas being talked about. This requires some pretty deep systems-level expertise as well as deep knowledge of PocketBase, nodejs, Typescript, Linux, and more

Thanks for asking!

[renatocaliari]

Thank you for structuring the possible causes and possible initial solutions @benallfree .

Would it help to try running some controlled PocketBase instances by mirroring the infrastructure where PocketHost is, and let them simulate production website patterns (such as queries per minute, execution time, and query complexity) for a few hours and observe how CPU, bandwidth, disk i/o, memory, etc. are maintained and which errors appear in the logs? Maybe this could give more insights?

[benallfree]

@renatocaliari I'm sure we might be able to get some information from that. Do you want to give it a try?

[renatocaliari]

Unfortunately, I don't have the technical knowledge or skills to help with that task at the moment. 😔

--

*I appreciate all the hard work you have put into developing PocketHost, as well as your willingness to engage with the community by answering questions and trying to resolve issues.

[benallfree]

Testing some new features, there may be some instability for a few hours!

[benallfree]

Testing has been completed. We will be monitoring for the coming days/weeks to see how stability has improved with v0.8.0!

[benallfree]

#234 Reports of expired cert.

@NatumanyaGuy @RobbeDeClerck

Investigating

[benallfree]

[benallfree]

@renatocaliari

[renatocaliari]

Great! Fixed. The platform is working again.

[RobbeDeClerck]

Thank you! problem fixed aswell for me here 😄

[NatumanyaGuy]

Thank you. Works Now!

[benallfree]

PocketHost will be down for scheduled maintenance window while we roll out v0.8.1.

[benallfree]

Back up 🎸

[HEAVYPOLY]

Are you all running production apps with PH/PB yet? I love the dx so far but worried about longer term stability. If I do go with PB/PH is it relatively easy to migrate out to planetscale, supabase etc if needed?

[benallfree]

These sound like questions for the PocketBase discussion board since you're asking about PocketBase stability and ease of migration to another platform. PocketHost doesn't get in the way of any of that, but certainly PocketHost wouldn't be a good decision if PocketBase itself isn't a good decision for your projects. Am I understanding your question correctly?

[HEAVYPOLY]

Pretty much, wondering if you'd recommend hosting a production app on PH/PB or stick to more hobby / personal projects for now.

[benallfree]

If you would run your production app on PocketBase yourself, you'd probably do fine running it on PocketHost as well.

[HEAVYPOLY]

Sounds good, thanks

[znowfox]

It's down buddy !

[GroomingMetro]

looks like it is down

[benallfree]

Looking into it.

[benallfree]

We are back up, now investigating!

[benallfree]

Just to recap, it looks like we had our first outage in a month. This is actually really impressive, because I made a bunch of stability improvements to PocketHost that appear to be very durable. We had a problem with the logging, and I'm not sure if I will be able to determine the cause of this most recent incident, but I did fix the logging issue.

For reference, here's what an outage looks like:

As you can see, bandwidth dropped to nothing and then shortly after that, CPU usage died off as well. To cause this, something would have needed to cause the main server.ts daemon to become unresponsive or die. That will remain under investigation. It took a month for it to occur, so I'll wait patiently for the conditions to occur again where I can get more insight as to the cause.

Thanks for the reports @MOBO-Timmy @GroomingMetro

[benallfree]

Getting reports of 403 errors from #253 and #254 and #255. Investigating.

[MalikWhitten67]

Thanks that was fast lol

[benallfree]

Ok, we are back up again. It looks like the v0.16.10 migrations took a LONG time and that caused an outage. Combing through logs now for more details.

[benallfree]

#259 reports an outage. Investigating...

[benallfree]

We are back up. Looking into the cause now.

[cheskoxd]

Hi!

Seems that is currently down...

https://pockethost.io/ site not showing the Login / Dashboard buttons
and if you go to https://pockethost.io/dashboard, it redirects you to the homepage

Thanks

[HEAVYPOLY]

Same here, seeing login button for a split second then it disappears.

Unexpected response {"url":"","status":0,"data":{},"isAbort":false,"originalError":{"cause":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":8090}},"name":"ClientResponseError 0"} from mothership

[benallfree]

@HEAVYPOLY @cheskoxd Back up, thanks for the report.

Investigating the cause now. Usually it ends up being some corner case that wasn't properly handled in code. Every time we go down, the system gets stronger! Thank you again!

[HEAVYPOLY]

Great, thank yoU!

[benallfree]

FYI possible dns outages today as we move to cloudflare.

[niku2676]

is every api call which is made in the meantime deleted?

[benallfree]

I’m not sure what you mean by deleted. Some API calls may fail to connect during this period.

[niku2676]

yeah thats why I mean. that is not good

[benallfree]

any backend outage for an application means API calls will not be successful. Yes, outages are not good . 🚫🙈

do you have a special case I am not understanding?

[renatocaliari]

About 10 minutes ago, I noticed that my website has been experiencing errors due to the connection with PocketBase timing out repeatedly.

[benallfree]

Yes please, that way I can check specifically for errors on your instance.

[renatocaliari]

chubby-zoo

[benallfree]

I see these two errors in the log:

[2023-09-03T11:50:33.569Z] [ERROR] [server.ts] [InstanceService] [chubby-zoo::~0.16.5] [pid:4184389] Error: Timeout waiting for pid to die

It seems possible that your instance became unresponsive. Are you running any long queries or dealing with complex data?

[renatocaliari]

They are simple queries. The database is very small. And it started giving an error without me changing anything.

[benallfree]

Ok it's unlikely you caused it. I will keep investigating. I am tracking a few intermittent errors that cause overall instability, so it's possible something else was going on and you just happened to be the one to notice it first.

[FrenchFryInteractive]

My instance seems to be down. Site can't be reached.

[benallfree]

How about now?

[FrenchFryInteractive]

Seems fixed. What was wrong?

[benallfree]

@FrenchFryInteractive Still investigating. The outages generally have to do with race conditions that appear only in production under very specific conditions, but I'll let you know what I find :)

[RA9]

My instance is giving me the below error

[benallfree]

Should be back up, I'm still looking into the underlying cause.

[DiceBreakers]

I am (along with HarithOmari) experiencing problems similar to RA9. TImeout obtaining instance API errors, and occasionally get into the admin dashboard and then "Something went wrong while processing your request" if I take any action.

Instance is starting and stopping shortly after. Tried creating a test database and it's experiencing the same. Original DB is version ~0.17.7, new DB is ~0.18.6.

2023-09-19T15:57:01.806Z info Shutting down instance
2023-09-19T15:56:56.854Z info
2023-09-19T15:56:56.827Z info └─ Admin UI: http://0.0.0.0:8090/_/
2023-09-19T15:56:56.766Z info ├─ REST API: http://0.0.0.0:8090/api/
2023-09-19T15:56:56.739Z info 2023/09/19 15:56:56 Server started at http://0.0.0.0:8090
2023-09-19T15:56:55.189Z info Starting instance
2023-09-19T15:55:18.872Z info Shutting down instance
2023-09-19T15:55:12.864Z info
2023-09-19T15:55:12.579Z info └─ Admin UI: http://0.0.0.0:8090/_/
2023-09-19T15:55:12.446Z info ├─ REST API: http://0.0.0.0:8090/api/
2023-09-19T15:55:12.417Z info 2023/09/19 15:55:12 Server started at http://0.0.0.0:8090
2023-09-19T15:55:10.459Z info Starting instance

[burraksumer]

I have almost the same issue where I can't access the admin panel for some reason and I get timeouts or 403s to api requests.

[Brisolo32]

I've been having the same issue where it goes to idle, i also don't have any hooks on PB

[benallfree]

@DiceBreakers @burraksumer @Brisolo32 PocketHost instances going idle is normal and is not related to any outage. PocketHost hibernates instances when there are no active HTTP connections.

[HEAVYPOLY]

Getting this on dashboard: Unexpected response {"url":"","status":0,"data":{},"isAbort":false,"originalError":{"cause":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":8090}},"name":"ClientResponseError 0"} from mothership

[twiddlecoding]

I get this Unexpected response {"url":"","status":0,"data":{},"isAbort":false,"originalError":{"cause":{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"127.0.0.1","port":8090}},"name":"ClientResponseError 0"} from mothership

And my service is down. Please fix this immediately.

[twiddlecoding]

It seems to be working now, what happened?

[benallfree]

@twiddlecoding @HEAVYPOLY @burraksumer @DiceBreakers Please try now. As per usual, investigating the cause takes some time but restarting the system is easy and effective :)

I'll update this thread more when I find out what is failing. I soft-released 0.9.0 which Dockerizes things, so there could be some instability with that. Typically the issues revolve around processing failing and the main thread giving up on serving more. Always looking for ways to continue improving this.

[DiceBreakers]

Thank you very much! I'm back up and running.

I appreciate you running the service and managing all the moving parts.

[benallfree]

FYI there may be some instability this morning as I roll out some enhancements. Should't be too noticeable though.

[benallfree]

@tenderassist Yes, I think that makes sense as the service grows and people begin relying upon it for production needs.

There are a lot of things I can do to optimize for uptime. Based on feedback and knowing how quickly people notice when the service is down, it sounds like this should become a point of focus :)

[benallfree]

PS - one thing I can do is keep a staging server so I understand the complexity of the upgrade. For example, today's upgrade is doing a database migration to the millions of invocation records stored in the central server. It seems quite long.

[benallfree]

Ok, we should be back to stability now. Please let me know if anything seems amiss.

[twiddlecoding]

It seems like your service is down at the moment

[benallfree]

@twiddlecoding Thank you, it looks like at least some services were running but I restarted everything just to be sure. We seem to have an issue that occurs every 2-4 days, still investigating :)

[twiddlecoding]

Your app is down

[123remodeling]

@benallfree Auth is working but Databases and Dashboard are down.

[benallfree]

We are back up. If anyone has time to send PRs are help troubleshoot, we are over in https://discord.com/channels/1128192380500193370/1159322063539023922 working on it.

[gideongrinberg]

The new update appears to have caused an outage.

[benallfree]

Definitely instability with the changeover to Docker at 0.9.0. Expected and necessary, but very annoying because things were pretty smooth before. We have all the logs and will continue investigating to track down the root causes of these outages.

[benallfree]

Everyone, please join us on Discord https://discord.gg/nVTxCMEcGT

This thread has moved to https://discord.com/channels/1128192380500193370/1179852349011939439

Discord provides a much better forum for troubleshooting issues in real time.