Custom install question

[vsoch]

I'm helping @kalle1024 with this issue and we were wondering how a user could go about installing custom software? Would it come down to rebuilding some base image (customized) and then using the interactive.json example to issue the command? I'm totally new to this tool, so please feel free to talk to me like I have no idea what I'm doing :) Thanks!

[kalle1024]

I'll try to explan my thoughts
What I see infront of me (i.e my user case) is that the different developer teams at my company creates/generates Singularity images with their respectively sub-functionality. Then they upload these sif's to a local sregistry, and then we can pull these to a production Kubernetes cluster where pods (based on multiple, different sif's) can be orchestrated based on demand.

It's basically the "Docker way" of doing Kubernetes, but instead I want to do it Locally, and with Singularity.

Cheers,
Kalle Larsson

[sashayakovtseva]

Hello @kalle1024,

I need to understand your issue better. Do you mean that instead of pulling image from a remote registry (docker hub, sylabs cloud, etc) you want to build images locally and use them to launch pods via K8s? Is that correct?

[kalle1024]

Yes, thats correct.
But my developer teams works independently, and will not have access to run stuff on the Kubernetes-master.
The Singularity images must be pushed to the local sregistry, and k8s will need to pull them from there and then it's up to k8s to do the pod stuff (through the definition of a yaml file such as darkflow.yaml in the singularity-cri examples, and applied by a command such as: "kubectl apply -f darkflow.yaml")

Cheers,
Kalle Larsson

[sashayakovtseva]

Kubernetes doesn't pull any images itself, but instead delegates this jib to CRI implementation (Singularity-CRI in your case). In other words, your developers don't need to have access to k8s master to start running pods. All they need to do is apply pod/deployment/whatever specification with pullable image to k8s. For SIF images the easiest way is to reference image from cloud.sylabs.io.

Singularity-CRI now supports two options: building from docker public images or pulling from an arbitrary sylabs library. There is also another option which allows you to hack Singularity-CRI (described here), but I don't recommend this approach.

@vsoch Does sregistry implements sylabs library API?

[vsoch]

Actually, the current pull request here does include just the library pull endpoint. However, with a recent versions of Singularity the variable wasn't passed on, introducing a bug, and it should be resolved with singularity 3.3.0 release.

For the other library endpoints / features, I am planning on adding them as they are needed. That particular PR is deployed at https://containers.page so you should be able to test pulling with the library API if using version 3.2.1. For example (here is 3.2.1 in docker):

$ docker run --entrypoint sh -it singularityware/singularity:3.1-slim 
> singularity pull --library https://containers.page collection/container:shadoop

Of course this makes more sense running on your host so you can actually interact with the image :) But the take away is that the pull endpoint for the library is in place. See: https://containers.page/api/docs/#v1-images-read

So if you are pulling from an arbitrary Sylabs library, this would work! The current PR is being tested until the end of the week, at which point I'll finish it up and merge into master. I'd be happy to work with you @kalle1024 if you want to get this running and start testing it out for your use case.

[vsoch]

And @sashayakovtseva is there a good getting started guide for Singularity Cri, one that doesn't require an enterprise cluster that I could run locally? I'd like to get my feet wet at some point and give it a go!

[sashayakovtseva]

@vsoch Sure, there is user docs on sylabs.io/docs and to test things locally you can use Sykube (described there as well). Or you can use vagrant vms (can be found in sylabs/wlm-operator/vagrant).
Will be happy to help with any of that!

[kalle1024]

$ docker run --entrypoint sh -it singularityware/singularity:3.1-slim 
> singularity pull --library https://containers.page collection/container:shadoop

OK, so I can pull an image without problems with this command from containers.page.

I can also browse my own/local sregistry fron a webbrowser, and also pull/push sif images to my own/local sregistry with the "sregistry" command from GlobalClient.

But I cannot do a "singularity pull" from my own/local registry.
2 questions appears in my mind:

• Is https mandatory?
• What about login with credentials when doing a "singularity pull --library xxxx"? sregistry dosen't support unauthorized pulls if I understand correctly, but still https://containers.page seems to allow just that?

Then (and I might missunderstand things here, so please bear with me) a "Singularity pull --library" is not applicable in a Kubernetes Pod definition yaml - right? So what kind of "pull" does the Singularity-CRI do under the hood?

@vsoch I have a VERY crude and quick install doc for 3-node kubernetes cluster on CentOS7 VM's including adding Singularity & Singularity-CRI support. I'll post it if you want :-)

Thanks for all your effort!

Cheers,
Kalle Larsson

[vsoch]

But I cannot do a "singularity pull" from my own/local registry.

The version deployed at containers.page isn't merged into master, so you don't have those features. When it's merged (and if you update your registry) this should work.

Is https mandatory?

You always should use https. No questions asked - the information sent to and fro your registry must be encrypted.

What about login with credentials when doing a "singularity pull --library xxxx"?

The only library endpoint that is supported is pull, I haven't taken a look at the authentication. You have to remember that Sylabs has no public documentation for their API, I've figured it out on my own.

sregistry dosen't support unauthorized pulls if I understand correctly, but still https://containers.page seems to allow just that?

All of the containers.page containers are considered public (so anyone can pull).

Then (and I might missunderstand things here, so please bear with me) a "Singularity pull --library" is not applicable in a Kubernetes Pod definition yaml - right? So what kind of "pull" does the Singularity-CRI do under the hood?

I'm not totally clear on how the logic from above leads to this question (the Singularity-CRI pull requires authentication you are saying?) so possibly @sashayakovtseva can help here.

@vsoch I have a VERY crude and quick install doc for 3-node kubernetes cluster on CentOS7 VM's including adding Singularity & Singularity-CRI support. I'll post it if you want :-)

I would love this! It will actually be very helpful - I'm a little backed up with TODOs but this is on my list (maybe if I catch up over the holiday weekend!) and I think after that I'll be able to help hugely more than I can now.

[sashayakovtseva]

You have to remember that Sylabs has no public documentation for their API, I've figured it out on my own.

@vsoch @kalle1024 We have scs-library-client that can be used to interact with sylabs library. And exact same client is used by Singularity-CRI under the hood when pulling library images.

[vsoch]

Sorry I misspoke - there isn't documentation for the library API implementation (e.g., a URL like https://library.sylabs.io/v1/images/busybox:latest) The clients / singularity to interact with it are great!

[kalle1024]

CRUDE installation doc for test kubernetes cluster on CentOS7

Based on:
https://www.howtoforge.com/tutorial/centos-kubernetes-docker-cluster/

All these steps should be executed on all 3 machines unless noted.

Install 3 CentOS minimal

vi /etc/hostname (on all 3 machines) and change respectively to:
k8s-master
node01
node02

vi /etc/hosts and add all 3 hosts with IP addesses on all machines:

yum update

yum install epel-release --enablerepo=extras
yum groupinstall 'Development Tools'
yum install git openssl-devel libuuid-devel libseccomp-devel wget squashfs-tools rpm-build golang inotify-tools socat yum-utils device-mapper-persistent-data lvm2

setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux

systemctl stop firewalld
systemctl disable firewalld

modprobe br_netfilter
echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables

swapoff -a
vi /etc/fstab (remove swap)

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

yum install -y docker-ce kubelet kubeadm kubectl

systemctl start docker && systemctl enable docker
systemctl start kubelet && systemctl enable kubelet

mkdir /etc/docker

cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF

systemctl restart docker

ONLY on k8s-master

echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables (don't know why we need to do this again :-)
kubeadm init --apiserver-advertise-address=<IP_of_k8s-master> --pod-network-cidr=10.244.0.0/16

Take note of the line: "kubeadm join......."

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl get nodes kubectl get pods --all-namespaces

ONLY on node01 & node02

kubeadm join ..... (from command above on k8s-master)

ONLY on k8s-master

Check again:
kubectl get nodes
kubectl get pods --all-namespaces

Optional test of kubernetes cluster

kubectl create deployment nginx --image=nginx
kubectl describe deployment nginx
kubectl create service nodeport nginx --tcp=80:80

[root@k8s-master ~]# kubectl get nodes
[root@k8s-master ~]# kubectl get pods
[root@k8s-master ~]# kubectl get svc

Compile & install Singularity Runtime & Singularity-CRI (on all 3 machines)

mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cd ~/rpmbuild/SOURCES/
wget https://github.com/sylabs/singularity/releases/download/v3.2.1/singularity-3.2.1.tar.gz

rpmbuild -tb singularity-3.2.1.tar.gz
cd ../RPMS/x86_64/
yum localinstall singularity-3.2.1-1.el7.x86_64.rpm

cd
git clone https://github.com/sylabs/singularity-cri.git
cd singularity-cri
make
make install

Singularity Registry

https://singularityhub.github.io/sregistry/install-host

yum install docker-compose.noarch <-- Note, this installs python36
yum install python36-pip
pip3 install --upgrade pip

cd
git clone https://github.com/singularityhub/sregistry
cd sregistry

cp shub/settings/dummy_secrets.py shub/settings/secrets.py
vi shub/settings/secrets.py <-- Fix SECRET

vi Dockerfile

vi shub/settings/config.py
'pam_auth',

vi docker-compose.yml
- /etc/passwd:/etc/passwd
- /etc/shadow:/etc/shadow

groupadd --system nginx
useradd --system -m --home-dir /var/cache/nginx -g nginx nginx

NOTE: Seems that "RUN pip install django-pam" dosent get executed in uwsgi container, log in to that and execute manually.
Verify with python, help("modules") and see that django_pam is there.

[vsoch]

Woohoo thank you! For django_pam since it's an optional plugin, note that you'd have to uncomment this line https://github.com/singularityhub/sregistry/blob/master/Dockerfile#L57 and build the main image, so in your instructions above

vi Dockerfile
# uncomment RUN pip install django-pam
docker build -t vanessa/sregistry .

[kalle1024]

Hello again :-)
@sashayakovtseva - I'm by no mean a programmer, but it I (wildly) guess, I can replace
const defaultBaseURL = "https://library.sylabs.io"
In the file "client/client.go" Row 48 in scs-library-client with my local URL, and then recompile Singularity-CRI to force Kubernetes Pod yalm definition files to actually pull images from a local sregistry when specifying something like this:

containers:
  - name: hello-kubernetes
    image: collection/container:image
      ports:
      - containerPort: 8080

and then execute "kubectl apply -f mypod.yaml"

Or am I totally wrong? :-)

Also: @vsoch @sashayakovtseva I really, really want to thank you both for taking time on this. There is no hurry whatsoever from my point, and actually I will go "off grid" a few weeks now for much needed vacation and travel. I will be back and I'm really looking forward to help creating a system where Singularity-based Kubernetes-pods in a cluster, pulled as native sif images from a local sregistry is a better alternative than todays Docker stuff :-)
My guess is that this can be a game-changer in the container world :-)

Cheers,
Kalle

[sashayakovtseva]

Hey @kalle1024,
please, don’t change code base, everything can be configured without recompilation. I will post detailed instructions here in a couple hours.

[sashayakovtseva]

Hello @kalle1024 ,
well, it looks like there is no easy way to make singularity-cri work with private library (I tried to hack it with this guide, but that is docker-centric, no way to pass library token now, and also there are internal checks in kubelet for that). I have asked k8s community about that and currently waiting for response.

What I can think of now is to override defaultBaseURL in your /etc/hosts file to make it point to your local registry. But that should be performed on each host, which is not great. Another option is to pull images manually to the specific location where singularity-cri expects images to be, but that is even more complex.

Btw in your pod spec you have a mistake. If you specify image without cloud.sylabs.io prefix, singularity-cri will treat it as docker image. So in your case you should use cloud.sylabs.io/collection/container:image . However, I am not sure how that will work with sregistry and whether that will work at all.

[vsoch]

@kalle1024 just to confirm - there isn't any way to test this without having three machines handy?

[sashayakovtseva]

@vsoch @kalle1024 Pulling private images from cloud.sylabs.io is supported now, refer to docs for details.

What else I can help with regarding this issue?

[kalle1024]

Hello :-) I have currently involved a collegue/python programmer who is looking into the code to see if we can integrate Kubernetes/Singularity-cri with sregistry so we can fetch containers from a local registry through a pod definition. He'll need some start-up time to get into the problem, but we'll keep you updated and in the loop ofc :-) (still trying to get up to pace from my vacation :-) Cheers, Kalle Larsson

…

-----Original Message----- From: "Sasha Yakovtseva" <notifications@github.com> Sent: Thursday, 15 August, 2019 16:05 To: "sylabs/singularity-cri" <singularity-cri@noreply.github.com> Cc: "kalle1024" <kalle@hemmabruket.net>, "Mention" <mention@noreply.github.com> Subject: Re: [sylabs/singularity-cri] Custom install question (#321) @vsoch @kalle1024 Pulling private images from cloud.sylabs.io is supported now, refer to [docs](https://sylabs.io/guides/cri/1.0/user-guide/constraints.html#using-image-from-private-registry) for details. What else I can help with regarding this issue?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #321 (comment)

[cjsmall717]

Hi kalle1024,
I'm looking at a similar requirement - to import a SIF file into a kubernetes environment. I've been banging my head on the sregistry thing for a couple of days but this is overkill for what I require in the immediate term. I have a singularity container that functions, as required, in hte context of a standalone container. I have a kubes implementation that is running Singularity-cri, and it can pull docker images. I have an sregistry instance that appears to work - it's authenticating against an external LDAP server.

My test environment is virtualised - qemu running on top of slackware host. 4 x archlinux guest VMs running a kubes cluster. The test environment is internet connected, the target system is not, so I have a preference to build containers, test, on hte test system and then import the SIF files into the target environment for deployment.

The sregistry thing works in as much as I can import a SIF file into the system and then pull it out again using curl. The fact that it is using http and kubes/singularity want https is causing me some consternation. I have a certificate authority defined, an intermediate certificate, etc. I just don't know enough about the sregistry/nginx/wsgi side of things to get up and running with HTTPS.

If I render down my requirements, and yours, there seems to be a lot of overlap. I'd certainly be interested to know if you've solved the local repo thing, especially if there is a means to directly import a SIF image.
Thanks.
Cheers, Chris

[vsoch]

https for sregistry comes down to creating your certificates, moving them to the right spot, and then switching up the docker-compose.yml to use the one with https (port 443). There are instructions for how I typically generate the certificates here and that references the https equivalents of the files you need here. Let me know if anything in the documentation is not clear - it's fairly straight forward for the server to have https.

[cjsmall717]

Awesome, thanks Vanessa!
Cheers, Chris.