microk8s Off-line installation fails on RHEL 9.3

[hpanneti]

Summary

I followed the links https://microk8s.io/docs/install-offline to install microk8s on a VM with restricted internet access using core20 + microk8s 1.27

But I have the following result

[glass@UFRPARESB012 Install-microk8s]$ sudo snap ack core20.assert && sudo snap install ./core20.snap
[sudo] password for glass:
core20 20240227 from Canonical✓ installed

[glass@UFRPARESB012 Install-microk8s]$ sudo snap ack microk8s.assert && sudo snap install ./microk8s.snap --classic
error: cannot perform the following tasks:

• Ensure prerequisites for "microk8s" are available (cannot install system snap "snapd": Post https://api.snapcraft.io/v2/snaps/refresh: read tcp 10.29.208.22:37140->185.125.188.54:443: read: connection reset by peer)

It seems that there is still Internet access attempts even in an off-line installation.

I had similar failure with core18 + microk8s 1.26.

This instance is behind a proxy.
So I configured the proxy as required in the link https://microk8s.io/docs/install-proxy.
But this didn't allow to workaround the issue.

Thanks for your help.

Best regards

Hervé Pannetier

[hpanneti]

Hello,
I finally succeeded to install microk8s on the VM by running the following commands :

sudo snap set system proxy.http=http://proxy:port/

sudo snap set system proxy.https=http://proxy:port/

However I’m facing a new issue;
Pods keep restarting with the following error :

Error from server: Get https://ufrparesb012:10250/containerLogs/default/lbs-appli-7b6c65bfd7-fgqmr/lbs-appli?follow=true: Tunnel or SSL Forbidden

Could you help me on this issue ? Thanks a lot

[aungmoewin]

How do you create snap proxy server?

[hpanneti]

Hello,
I did not create any snap proxy server. I 'm behind the customer one.
I just run the commands :

sudo snap set system proxy.http=http://proxy:port/
sudo snap set system proxy.https=http://proxy:port/

to be able to install microk8s offline with "sudo snap ack microk8s.assert && sudo snap install ./microk8s.snap --classic".

So the offline installation doesn't seem really offline.

For the other error reported in that ticket : "Pods keep restarting" , it appears that the partition /var is full leading to error such as "The node was low on resource: ephemeral-storage" and pods eviction.

I hope that the error "Tunnel or SSL Forbidden" is also a side-effect of this /var partition full.

The /var partition should be sized up today to overcome this issue.

Best regards

[hpanneti]

Hello
The resizing of the /var partition allows to stabilize all pods.
I'm now facing a DNS issue.
Within a pod the DNS resolution fails to resolve any svc url.

I did setup /etc/environment and /var/snap/microk8s/current/args/containerd-env to specify HTTP_PROXY, HTTPS_PROXY and NO_PROXY but this seems not sufficient to get coredns working properly.

Have you any suggestion?
Thanks a lot!

BTW , I use microk8s 1.28.8

[hpanneti]

Hello,
Would you please confirm that the file /etc/environment and /var/snap/microk8s/current/args/containerd-env must be set as below when microk8s is deployed behind a proxy.
The part .svc,.local is not in the official documentation but found googling.
Some other use also .svc,.svc.cluster.local in NO_PROXY

NO_PROXY=,,127.0.0.1,<nodes internal ip(s)>/24,,.svc,.local

Anyway coredns is failing to resolv any name whatever the containerd-env configuration.

Thanks for your help

[hpanneti]

Sorry , bad copy/paste. The no_proxy setting is as below. Is that correct?

NO_PROXY=<cluster cidr>,<service cluster ip range>,127.0.0.1,<nodes internal ip(s)>/24,<cluster hostname>,.svc.cluster.local,.cluster.local
no_proxy: <cluster cidr>,<service cluster ip range>,127.0.0.1,<nodes internal ip(s)>/24,<cluster hostname>,.svc.cluster.local,.cluster.local

[hpanneti]

Hello
We finally understood that microk8s coredns stop working after reloading the firewalld rules.
This action deleted 2 iptables rules generated for microk8s pods.

Before “firewall-cmd --reload”

# sudo iptables  -L
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
ACCEPT     all  --  10.1.0.0/16          anywhere             /* generated for MicroK8s pods */
ACCEPT     all  --  anywhere             10.1.0.0/16          /* generated for MicroK8s pods */

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# sudo firewall-cmd  --permanent --add-port=587/tcp
success

After “firewall-cmd --reload”

# sudo firewall-cmd --reload
success
# sudo iptables  -L
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination