docs: add troubleshooting steps for prebuilt workspaces

[SasSwart]

This PR adds troubleshooting steps to guide Coder operators when they suspect that prebuilds might have overwhelmed their deployments.

Closes #19490

[ssncferreira]

Nice work 🚀 Should we also mention that users can tune the CODER_PREBUILDS_RECONCILIATION_INTERVAL to manage how frequently the prebuild reconciliation loop runs? That might help reduce the load from frequent reconciliations. Wdyt?

[ssncferreira]

I think the issue here is actually a combination of these two factors: there aren’t enough resources to handle the high demand from prebuild-related provisioner jobs. This problem can be further amplified when those jobs take a long time to complete.

Additionally, might be worth explanation an additional scenario when a user creates a new template version (a user-initiated job), once this is processed and the prebuild reconciliation loop runs, it adds even more load by scheduling new prebuild-related jobs. This means the queue could now include jobs for both template version 1 and version 2.

[SasSwart]

I'll adjust the wording to indicate that these issues aren't mutually exclusive, but I don't quite understand what else to change in response to this feedback.

Can you perhaps elaborate or rephrase to help me understand exactly what you need to me to consider changing?

[ssncferreira]

The root cause is that prebuild-related jobs are being scheduled faster than provisioner daemons can process them. Multiple scenarios can lead to this (too few provisioners, templates that request many prebuilds, or bursts from publishing new template versions). It might be clearer to frame it that way and then help readers identify when this is happening. Here’s a suggestion that rephrases the paragraph accordingly and adds some guidance for detection:

Prebuilt workspaces can overwhelm a Coder deployment when the number of prebuild‑related jobs exceeds the capacity of the provisioner daemons. This often occurs when one or more templates are published or configured to require more prebuilds than the system can sustain. The impact is amplified when prebuild jobs take a long time to complete, which keeps capacity occupied and increases queue depth. This can cause significant delays when users create new workspaces or when template administrators publish templates.

To identify if this is happening:
- Large or growing queue of prebuild‑related jobs
- User workspace creation is slow
- Publishing a new template version is not reflected in the UI because the associated template import job has not yet finished

[ssncferreira]

I’m not sure this sentence is entirely accurate. Since human-initiated jobs already have priority over prebuild-related jobs, the next available provisioner will automatically be assigned a human-initiated job if there is one. The purpose of this behavior is to help clear the queue and prevent situations where all provisioner daemons are occupied with prebuild-related jobs, which could delay human-initiated ones.

[SasSwart]

I've updated it.

[ssncferreira]

AFAIU, this command won’t actually print the list of jobs — it will pipe them directly into jq. I think it would be useful to show the list of jobs first, so users can review them before deciding to cancel. That way, they could choose to cancel only a subset of prebuilds if needed.

Wouldn’t it make more sense for coder provisioner jobs cancel to accept a list of job IDs?
Right now, we don’t support cancelling multiple jobs simultaneously (either through the CLI or the dashboard), so adding that capability would be a nice improvement.

[SasSwart]

Wouldn’t it make more sense for coder provisioner jobs cancel to accept a list of job IDs?

I would love to do this, but its beyond scope for this issue. We can definitely implement it and then come back to revise the documentation.

Right now, we don’t support cancelling multiple jobs simultaneously

This is also something we can add in the future.

I think it would be useful to show the list of jobs first

This command before this one in the documentation shows the list. In the context of prebuilds specifically, there should be no reason to cancel only a subset. These jobs are transient and will be replaced once coder prebuilds resume is executed.

[ssncferreira]

I would love to do this, but its beyond scope for this issue. We can definitely implement it and then come back to revise the documentation.

Yes, this is out of scope for this PR, but would definitely be a nice-to-have 👍

This command before this one in the documentation shows the list. In the context of prebuilds specifically, there should be no reason to cancel only a subset. These jobs are transient and will be replaced once coder prebuilds resume is executed.

A possible reason I see for canceling a subset is if you want to just cancel in-progress prebuilds from a specific template and version...Another idea would be to have the list command with --template and --version flags. But this is also out of scope of this PR, we can update this documentation if we change the jobs list and jobs cancel commands 👍

[github-actions]

🚫 [linkspector] _{reported by reviewdog 🐶}
Cannot reach https://developer.hashicorp.com/terraform/language/v1.13.x/meta-arguments#lifecycle Status: 429

[ssncferreira]

Why was this link changed? 🤔 The previous one was version-agnostic, which might be preferable so we don’t have to update it when Terraform releases new versions.

[SasSwart]

The previous link pointed to a heading that doesn't exist.
Terraform have in the last few versions also changed the content of the page such that the link wouldn't have made sense either way. I pinned it to a specific version so that we have confidence that what we link to doesn't change without us realizing it.

The trade-off is between currency and accuracy. If you'd prefer to keep the old link, I can do that.

[ssncferreira]

LGTM 👍