Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

Open
tsde opened this issue Dec 27, 2021 · 35 comments
Open

Can't install Kubernetes >=1.22 on Flatcar Linux due to missing SELinux custom policies #2788

tsde opened this issue Dec 27, 2021 · 35 comments
Labels
team/area2

Comments

@tsde
Copy link

tsde commented Dec 27, 2021

RKE version:

1.3.3 (using terraform RKE provider v1.3.0)

Docker version: (docker version,docker info preferred)

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 17
  Running: 9
  Paused: 0
  Stopped: 8
 Images: 13
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: cde01e96ed658bc5050abe1bb601b4b4510ba7a2
 runc version: e4bccdbd64361ac5ea8ba90bb8845add78f957a6
 init version: 
 Security Options:
  seccomp
   Profile: default
  selinux
  cgroupns
 Kernel Version: 5.10.84-flatcar
 Operating System: Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.807GiB
 Name: worker-01
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3033.2.0
VERSION_ID=3033.2.0
BUILD_ID=2021-12-10-1820
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3033.2.0 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar-linux.org/"
BUG_REPORT_URL="https://issues.flatcar-linux.org"
FLATCAR_BOARD="amd64-usr"

5.10.84-flatcar

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Master/Worker nodes provisionned by terraform using the RKE provider v1.3.0. Nodes are vSphere virtual machines based on the Flatcar OVA.

cluster.yml file:
As I'm using the terraform provider, here's the tf rke_cluster declaration

resource "rke_cluster" "main" {
  kubernetes_version = "v1.22.4-rancher1-1"
  cluster_name       = "test-cluster"
  authentication {
    strategy = "x509"
    sans     = "<...redacted...>"
  }
  dynamic "nodes" {
    for_each = flatten([local.rke_cluster_master_nodes, local.rke_cluster_worker_nodes])
    content {
      address           = nodes.value["address"]
      ssh_key           = nodes.value["id_rsa"]
      labels            = nodes.value["labels"]
      role              = nodes.value["roles"]
      hostname_override = nodes.value["name"]
      user              = nodes.value["user"]
    }
  }
  dns {
    provider = "coredns"
  }
  ingress {
    provider     = "none"
  }
  network {
    plugin  = "calico"
    options = {
        "calico_cloud_provider" : "none",
        "calico_flex_volume_plugin_dir" : "/var/lib/kubelet/volumeplugins"
    }
  }
  services {
    kube_api {
      audit_log {
        enabled = true
      }
      secrets_encryption_config {
        enabled = true
      }
    }
  }
  upgrade_strategy {
    drain                        = false
    max_unavailable_worker       = 1
    max_unavailable_controlplane = 1
  }
}

Steps to Reproduce:

Try to update a kubernetes cluster from 1.21 (or possibly earlier versions) to 1.22 when using Flatcar OS 3033.2.0. I imagine that a fresh 1.22 installation would lead to the same result.

Results:

The following error occurs:

Failed running cluster err:[[selinux] Host [10.130.0.241] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]

As shown in docker info above, SELinux is enabled on dockerd, triggering this specific step from RKE. Starting from 1.22, a dedicated custom SELinux policy must be installed on SELinux-enabled nodes. As I'm using Flatcar Linux, it's not possible to deploy this RPM as-is.

I'm quite a newbie when it comes to SELinux and I don't see how I can easily work around this as disabling SELinux on the docker daemon is not an option for me. Is there any plan on RKE side to better integrate this with Flatcar Linux ? I may be missing a simple way to circumvent this so don't hesitate to tell me ^^

Thanks
@xeor
Copy link

xeor commented Jan 10, 2022

Did you find a solution to this? I think rke is installing this package automatic on rpm distroes, but flatcar is not one of those.
How is this done on other non-rpm distroes?

Should this be lifted to https://github.com/rancher/rke2-selinux, and a request for adding flatcar? Anyone know how this works together?

@tsde
Copy link
Author

tsde commented Jan 11, 2022 •
edited

@xeor i didn't find a solution yet. I naively tried to register the SE module extracted from the rpm directly in a flatcar instance and, obviously, it failed as /usr is read-only on Flatcar.

I'm also considering opening an issue on Flatcar side and see what's their opinion about this and what would be the best approach. Maybe opening a request for a new package to be included in Flatcar, but don't know if it meets Flatcar requirements for new packages ?

In the meantime, it would be interesting to have a feedback from the rancher/rke team about this ^^

And yes, I imagine the same situation occurs with rke2-linux rpm package.

@tsde tsde mentioned this issue Jan 18, 2022
Open
@stale
Copy link

stale bot commented Mar 13, 2022

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the status/stale label Mar 13, 2022
@bitfisher
Copy link

Any progress so far?

@stale stale bot removed the status/stale label Mar 31, 2022
@tsde
Copy link
Author

tsde commented Apr 4, 2022

@bitfisher I opened an issue on Flatcar side too (see flatcar/Flatcar#598). The current workaround is to have selinux disabled for the docker service. This is of course not ideal but work seems currently on-going on Flatcar side to smooth out SELinux related stuff flatcar/Flatcar#673

@mohsenmottaghi
Copy link

We have the same issue in our production cluster, and we are stuck in v1.21.X. but I tried to disable SELinux in a test cluster by adding a systemd dropin and it works without issue.

systemd:
  units:
    - name: "docker.service"
      enabled: true
      dropins:
        - name: "01-selinux.conf"
          contents: |
            [Service]
            Environment=DOCKER_SELINUX=--selinux-enabled=false

But I don't want to disable SELinux in my production clusters.

@bitfisher
Copy link

Any updates?
Disabling SELinux in production clusters isn't really an option!

@bitfisher
Copy link

Is there any other option than disabling SELinux?

@github-actions
Copy link
Contributor

github-actions bot commented Oct 5, 2022

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@bitfisher
Copy link

Unstale please

@bitfisher
Copy link

Any Updates?

Is there any other option than disabling SELinux?

@github-actions
Copy link
Contributor

github-actions bot commented Jan 3, 2023

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@bitfisher
Copy link

This is still an issue, so unstale please!

A response from anyone at Rancher would be highly appreciated!

Anyone managed updating k8s to >= 1.22 without disabling SELinux?

Any guiadance from Rancher regarding this issue?

@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2023

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@bitfisher
Copy link

Unfortunately this is still an issue, so unstale please!

A response from anyone at Rancher would be highly appreciated!

Anyone managed updating k8s to >= 1.22 without disabling SELinux?

Any guidance from Rancher regarding this issue?

@mikekuzak
Copy link

The thing is that Rancher/SUSE will pull out their support matrix and say Flatcar is not supported officially .. and they don't provide so far any support for any other immutable container OS. (sic!)

@bitfisher
Copy link

That's bad news :(

I haven't seen any comment from Rancher/SUSE that Flatcar-Support will be dropped. Even in their docs it's still mentioned as supported. (weird)

So we will either have to switch to another OS or drop RKE :(

Any other options you see?

@lazyfrosch
Copy link

As @mikekuzak is not associated with SUSE or Rancher, this statement is not helping in any way...

Flatcar has never been officially supported, and I guess they are busy with other issues. So Flatcar will never be a priority.

Remember, this is open source, means you could contribute a SELinux solution. Though I would do that for RKE2.

But this is also not an official statement, give them some slack, Rancher is a nice project.

@framctr
Copy link

framctr commented Mar 22, 2023 •
edited

Same issue with Fedora CoreOS 37 on OpenStack:

Failed running cluster err:[[selinux] Host [192.168.3.104] does not recognize SELinux label [label=type:rke_container_t]. This is required for Kubernetes version [>=1.22.0-rancher0]. Please install rancher-selinux RPM package and try again]

Installing rancher-selinux from GitHub does not solve the issue.

The only solution for now is to use a 1.21 kubernetes-rancher version (e.g., v1.21.14-rancher1-1).

@framctr
Copy link

framctr commented Mar 22, 2023 •
edited

I found a similar issue #2792 . It was closed because it was just stale.....

Another which may help is #1966 .

@github-actions
Copy link
Contributor

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@mikekuzak
Copy link

mikekuzak commented May 22, 2023 via email

@mikekuzak
Copy link

unstale

@abuisine
Copy link

Same issue here while moving from unsupported RancherOS to Flatcar with recent Kubernetes versions (1.24+)

@bitfisher
Copy link

There are no mentions about specific SELinux requirements here or here.

There are no statements that Flatcar-Linux isn't supported.

There's still not any comment or advice from any of the maintainers or team members since Dec 27, 2021!

@lazyfrosch I know it's open source and i could contribute, but that's not so easy for a little team that is busy with a lot of other stuff :(
I think most of rke's users have chosen it because of it's ease of use, good feature set and good maintenance.
And in my point of view this is totally broken now.
Even worse ... users left alone and stuck ... no official comments/advises or any help so far :(

Has anyone figured out why this new SELinux requirement was introduced?

Has anyone figured out how to upgrade to k8s >= 1.22 without disabling SELinux?

Was anyone successful, switching from Flatcar-Linux to OpenSuse MicroOS?

Has anyone successfully migrated from rke to k0sctl?

@bitfisher
Copy link

bitfisher commented Jul 31, 2023 •
edited

It seems that #2541 is the root cause of the problem.
#2541 was reverted with #2624 in release/v1.4 but not in release/v1.3!

@superseb @cmurphy @kinarashah @StrongMonkey @nickgerace It seems to me that #2750 needs to be reverted also in release/v1.3 or maybe some opt-out for flatcar can be used in v1.4?
The extra SELinux-Label is still used in rke v1.4 here and here

@github-actions
Copy link
Contributor

github-actions bot commented Oct 1, 2023

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@mikekuzak
Copy link

unstale

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 1, 2023

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@tailtwo
Copy link

tailtwo commented Dec 4, 2023 •
edited

Unstale again please. There still isn't a way to upgrade past 1.21 on Fedora CoreOS.

@mikekuzak
Copy link

mikekuzak commented Dec 4, 2023 via email

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 4, 2024

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@mohsenmottaghi
Copy link

Unstale again please

@github-actions GitHub Actions
Copy link
Contributor

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.

@mikekuzak
Copy link

mikekuzak commented May 27, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
team/area2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@lazyfrosch @xeor @abuisine @jiaqiluo @tsde @tailtwo @mikekuzak @mohsenmottaghi @framctr @bitfisher