Releases have disappeared 😱

[beastawakens]

Running a rack on 3.5.7, went to update the node_type param but the update failed as Terraform could not download the files from GitHub as that release has disappeared.
I can't even ascertain what is the latest release to update to before going to the next minor as lots of release information appears to have been deleted/vanished.

[beastawakens]

@Twsouza I also cannot step through minor version updates of this anyway, as moving to 3.6.x is not possible as that release has disappeared. What is the upgrade path for such a Rack?

[nightfury1204]

@beastawakens apologizing for the confusion
since aws stops support of the eks below k8s 1.22. team decided to remove those release from the github, so user doesn't break their rack updating to a unsupported k8s version.

you can update your rack to the 3.8.x or more latest
if you have the latest convox cli, then it has a --force flag that will allow you to skip minor version

convox rack update 3.8 --force

[beastawakens]

@nightfury1204 Thanks for the reply! That really wasn't communicated well :/ Will have to trust the upgrade from 3.5.x to 3.8.x is seamless then!

[slaskis]

what about racks not on aws?

[beastawakens]

@beastawakens apologizing for the confusion since aws stops support of the eks below k8s 1.22. team decided to remove those release from the github, so user doesn't break their rack updating to a unsupported k8s version.

you can update your rack to the 3.8.x or more latest if you have the latest convox cli, then it has a --force flag that will allow you to skip minor version

convox rack update 3.8 --force

@nightfury1204 Just tried to force update the Rack from 3.5.7 to 3.8.6 but it failed due to:

Error: error updating EKS Cluster (prod-us) version: InvalidParameterException: Unsupported Kubernetes minor version update from 1.20 to 1.22
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "ed420ec0-cf58-4b8a-b55d-3b822f6619dc"
  },
  ClusterName: "prod-us",
  Message_: "Unsupported Kubernetes minor version update from 1.20 to 1.22"
}

  on .terraform/modules/system/terraform/cluster/aws/main.tf line 39, in resource "aws_eks_cluster" "cluster":
  39: resource "aws_eks_cluster" "cluster" {

Please restore the last k8s 1.21 release so that we can perform an intermediate update. Thank you

[ntner]

Hey @beastawakens I'm not sure if this is new to AWS or a region specific thing, but I recently ran into this while updating a 3.5.3 rack to 3.8.6 on Gov-Cloud and assumed it was specific to its on peculiarity.

In the past after some threshold of depreciation AWS would manually update the k8s version for EKS. It wouldn't effect the actual nodes as kubectl get nodes would still show k8s on the expected version based on the rack version.

This may no longer be the case if you're now seeing this issue as well. The problem with opening up a 1.21 k8s rack release (3.6.x, 3.7.x) is that you will get a different error stating this is a depreciated k8s version and you will not be allowed to spin up the new nodes. This is one of the reasons why we got rid of the releases.

The workaround I was able to make work with surprisingly no issues was to simply manually click the "update now" button in the AWS EKS Console GUI and select 1.21 (It will still let you do this for these older clusters). After it updates you will need to go in and update each node group in the GUI similarly. If you do one then the next two node groups, I found I didn't even lose rack api access. After all the node groups and EKS cluster are registering as 1.21 in EKS you can run the force command and it should complete with no issues. I did this outside the Console with terraform.

[beastawakens]

@ntner Uurgh, this is ugly... 😞 I think it will be neater to spin up a new Rack on the latest version and migrate the apps 😒

[ndbroadbent]

This is very scary, I think it's a really bad idea to delete older releases. I'm still on version 3.6.0 and still need access to my racks. I recently wiped and reinstalled my MacBook and now I can't set up the terraform racks, so I'm locked out and forced to upgrade

[ndbroadbent]

I attempted to update to 3.8.6, and had about 5 minutes of downtime while upgrading k8s from 1.21 to 1.22. I really want to figure out how to avoid any downtime in the future. It looks like everything goes offline for a while when updating the node groups.

terraform apply is also crashing with a few different errors.

│ Error: clusterroles.rbac.authorization.k8s.io "eks-console-dashboard-full-access-clusterrole" already exists
│
│   with module.system.module.cluster.kubernetes_cluster_role.console,
│   on .terraform/modules/system/terraform/cluster/aws/console.tf line 1, in resource "kubernetes_cluster_role" "console":
│    1: resource "kubernetes_cluster_role" "console" {
│
╵
╷
│ Error: clusterrolebindings.rbac.authorization.k8s.io "eks-console-dashboard-full-access-binding" already exists
│
│   with module.system.module.cluster.kubernetes_cluster_role_binding.console,
│   on .terraform/modules/system/terraform/cluster/aws/console.tf line 76, in resource "kubernetes_cluster_role_binding" "console":
│   76: resource "kubernetes_cluster_role_binding" "console" {
│
╵

I tried re-running it a few times but couldn't get this to work, so I might upgrade to a later version (3.9.4) and see if that works.

[beastawakens]

This has caused another issue when trying to delete an older Rack... You can't!

Uninstalling Rack...
Upgrading modules...
Downloading github.com/convox/convox?ref=3.5.7 for system...

Error: Failed to download module

Could not download module "system" (main.tf:2) source code from
"github.com/convox/convox?ref=3.5.7": error downloading
'https://github.com/convox/convox.git?ref=3.5.7': /usr/bin/git exited with 1:
error: pathspec '3.5.7' did not match any file(s) known to git.



Error: Failed to download module

Could not download module "system" (main.tf:2) source code from
"github.com/convox/convox?ref=3.5.7": error downloading
'https://github.com/convox/convox.git?ref=3.5.7': /usr/bin/git exited with 1:
error: pathspec '3.5.7' did not match any file(s) known to git.


ERROR: exit status 1
ERROR: we have been notified about a system error: (f9cb6baf5f0b42eda67884d3f993da18)