Deploy on European Open Science Cloud (EOSC)

This tutorial provides an overview of the steps involved in setting up a CRE on the European Open Science Cloud (EOSC) using command-line.

Note: Please follow Starting a PhenoMeNal CRE on a public or private cloud provider for the general prerequisites for a deployment on a private of public cloud provider.

EOSC specific prerequisites

First, if you not already is a member, then you need join a (EGI)-Virtual Organization that have resources on the EGI-FedCloud. - For examplel Elixir research Infrastructure (https://www.elixir-europe.org/intranet)

Then when you are a member of the Elixir-VO you can go on creating an openstack configuration directory (Currently only EOSC openstack clouds are supported):

kn --preset phenomenal init openstack my-vre-config-dir
cd my-vre-config-dir

Create a dir for voms-proxy-configuration

mkdir voms-config

Copy your voms-proxy-certificate created at https://elixir-cilogon-mp.grid.cesnet.cz/vo-portal/

cp your-cert voms-config/voms-proxy-cert

Init your voms config with the cloud provider you want

kn voms-config "vo.elixir-europe.org" "https://sbgcloud.in2p3.fr:5000/v2.0"

Export variables as described in output from previous 'kn voms-config' command

e.g.

 export OS_AUTH.....
 export OS_X509.....

Configuration

All of the commands in this documentation are meant to be run in the config directory created above.

Inside this configuration directory you will need to edit the file config.tfvars where you will need to set:

Boot Image (At the end of the config file you need to edit this)

• boot_image: this need to be changed into "Image for KubeNow [Ubuntu/16.04/QEMU-KVM]"

Cluster

• cluster_prefix: every resource in your tenancy will be named with this prefix
• external_network_uuid: the uuid of the external network in the OpenStack tenancy - usually this is called ext-net or public, to find out you either look in your openstack web interface or via terminal command kn openstack network list. Here is the link to a detailed detailed guide of how to list networks in the KubeNow documentation
• floating_ip_pool: a floating IP pool name (this is usually the name of the external network, see parameter above)
• dns_nameservers: (optional, only needed if you want to use other dns-servers than default 8.8.8.8 and 8.8.4.4)

Master configuration

• master_flavor: an instance flavor for the master - Here is the link to the detailed guide of how to list flavors in the KubeNow documentation of how to list and find available flavors in your OpenStack teenancy.
• master_as_edge: master is acting as gateway for accessing services

Node configuration

• node_count: number of Kubernetes nodes to be created (no floating IP is needed for these nodes)
• node_flavor: an instance flavor name for the Kubernetes nodes

Gluster configuration - See: KubeNow Gluster documentation.

• glusternode_count: number of egde nodes to be created (1 - 3 depending on preferred replication factor)
• glusternode_flavor: an instance flavor for the glusternodes
• glusternode_extra_disk_size: disk size of the fileserver disk in GB (depending on the size of your dataset)

Edge configuration (optional)

• edge_count: number of egde nodes to be created
• edge_flavor: an instance flavor for the edge nodes

Cloudflare (optional) - See: KubeNow Cloudflare documentation.

• use_cloudflare: wether you want to use cloudflare as dns provider
• cloudflare_email: the mail that you used to register your Cloudflare account
• cloudflare_token: an authentication token that you can generate from the Cloudflare web interface
• cloudflare_domain: a zone that you created in your Cloudflare account. This typically matches your domain name (e.g. somedomain.com)
• cloudflare_subdomain: "a subdomain for this deployment"

Cloudflare proxy (optional) - See: KubeNow Cloudflare proxy documentation.

• cloudflare_proxied:: to proxy or not, e.g. true
• cloudflare_record_texts: name on services to be proxied

In the provision sub-section of the config.tfvars config file you can edit the following parameters: (Also see KubeNow Provisioning documentation.)

Services

• password_all_services: password for all your services (e.g. Galaxy, Jupyter etc.)
• username_all_services: username for all your services (e.g. Galaxy, Jupyter etc.)

Galaxy

• galaxy_include: should service be deployed in cluster (true/false)
• galaxy_admin_email: the local galaxy admin (you?)

Jupyter

• jupyter_include: should service be deployed in cluster (true/false)

Luigi

• luigi_include: should service be deployed in cluster (true/false)

Kubernetes dashboard

• dashboard_include: should service be deployed in cluster (true/false)

Logging and monitoring services - See: Logging and monitoring wiki

• logmon_include: should service be deployed in cluster (true/false)

Pachyderm + Minio (optional) - See: Pachyderm tutorial with MTBLS data

• pachyderm_release_name: a release name for the Pachyderm service
• pachyderm_etcd_pvc_size: storage dedicated for etcd (In GB)
• minio_release_name: release name for the Minio service
• minio_pvc_size: storage dedicated for the Minio service (In GB)
• minio_accesskey: access key for the S3 endpoint
• minio_secretkey: secret key for the S3 endpoint
• minio_replicas: number of replicas of the Minio service

Once you are done with your settings you are ready to deploy the cluster:

kn apply

when deployment is finished then you should be able to reach the services at:

Galaxy         = http://galaxy.<your-prefix>.<yourdomain>
Jupyter        = http://notebook.<your-prefix>.<yourdomain>
Luigi          = http://luigi.<your-prefix>.<yourdomain>
Kube-dashboard = http://dashboard.<your-prefix>.<yourdomain>
Pachyderm      = ssh into the master node and use pachctl. Pachyderm tutorial: https://github.com/phnmnl/MTBLS233-Pachyderm

and if you want to ssh into the master node:

kn ssh

and to destroy use:

kn destroy

PhenoMeNal help and support

For feedback and help