Skip to content

OCDX/OCDX-JupyterHub

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

JupyterHub - Autodeployment for Cluster/Cloud Notebooks

JupyterHub, a multi-user server, manages and proxies multiple instances of the single-user IPython Jupyter notebook server. This extension allows a JupyterHub front-end to spawn the JupyterNotebooks into cloud computing infrastructure.

AWS Auto Deploy Setup Steps

Change into the autoaws folder

cd autoaws

0. Optional : Create AWS Access Keys

  • Open Amazon AWS "Your Security Credentials"
  • Click "Create New Access Key"
  • Store Credentials in a home directory, ~/.aws/credentials

[default]
aws_access_key_id = <your_access_key_id>
aws_secret_access_key = <your_secret_access_key>

1. Create the Kubernetes Cluster

autoaws$ python3 run.py <cluster_name>
  • NOTE: Wait patiently!

OUTPUT

Success! Created cluster.yaml

Next steps:
1. (Optional) Edit cluster.yaml to parameterize the cluster.
2. Use the "kube-aws render" command to render the stack template.
Success! Stack rendered to stack-template.json.

Next steps:
1. (Optional) Validate your changes to cluster.yaml with "kube-aws validate"
2. (Optional) Further customize the cluster by modifying stack-template.json or files in ./userdata.
3. Start the cluster with "kube-aws up".
Creating AWS resources. This should take around 5 minutes.
Success! Your AWS resources have been created:
Cluster Name:	<cluster_name>
Controller IP:	52.70.194.118

The containers that power your cluster are now being dowloaded.

You should be able to access the Kubernetes API once the containers finish downloading.
Cluster Name:	<cluster_name>
Controller IP:	52.70.194.118

NOTE: You can see your running instances using the AWS Dashboard, EC2 instances, Running Instances


2. Add an entry to /etc/hosts

autoaws$ sudo sed -i '$s/$/\n52.70.194.118 <cluster_name>.omgwtf.in/' /etc/hosts

3. Kubectl "install":

For Linux:

wget https://storage.googleapis.com/kubernetes-release/release/v1.2.4/bin/linux/amd64/kubectl

For OS X:

wget https://storage.googleapis.com/kubernetes-release/release/v1.2.4/bin/darwin/amd64/kubectl

Copy kubectl to your path

autoaws$ chmod +x kubectl
autoaws$ sudo mv kubectl /usr/local/bin/

4. Kubectl to Standup Cluster:

autoaws$ kubectl --kubeconfig=<cluster_name>/kubeconfig get nodes
NAME                        LABELS                                                                                                                                                                                             STATUS
ip-10-0-0-50.ec2.internal   kubernetes.io/hostname=ip-10-0-0-50.ec2.internal                                                                                                                                                   Ready,SchedulingDisabled
ip-10-0-0-64.ec2.internal   beta.kubernetes.io/instance-type=m3.medium,failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta.kubernetes.io/zone=us-east-1c,kubernetes.io/hostname=ip-10-0-0-64.ec2.internal   Ready

JupyterHub Setup in AWS

1. Create the Hub Install into the EC2 instances

hub$ kubectl --kubeconfig=../autoaws/mudsa/kubeconfig create -f hub.yaml
hub$ kubectl --kubeconfig=../autoaws/mudsa/kubeconfig create -f config.yaml

2. Download Pods

hub$ kubectl --kubeconfig=../autoaws/mudsa/kubeconfig --namespace=jupyter get pods 
NAME        READY     STATUS    RESTARTS   AGE
hub-vi9xy   1/1       Running   0          1m

2. View Services

hub$ kubectl --kubeconfig=../autoaws/mudsa/kubeconfig --namespace=jupyter get svc
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
jupyterhub   10.3.0.40                  8000/TCP   13m
  • NOTE: Due to bug in AWS(?) the EXTERNAL-IP is not provided.

  • To find the public endpoint of the elastic load balancer:

    • In AWS Dashboard > EC2 Dashboard > Load Balancers
    • Click on Load Balancer and view description
DNS name: 
a3c48b0dc211311e685a612e85b7665b-1960005297.us-east-1.elb.amazonaws.com (A Record)
  • Copy the name and append the port in the Browser to access JupyterHub
http://a3c48b0dc211311e685a612e85b7665b-1960005297.us-east-1.elb.amazonaws.com:8000/hub/login

TO DESTROY AWS CLUSTER

  1. Log into AWS Dashboard > EC2 Dashboard > Load Balancers
  2. Remove the load balancer
  3. Use the Kubernetes to destroy the AWS cluster
autoaws$ cd <cluster_name>	
autoaws/<cluster_name>$ ../kube-aws destroy
CloudFormation stack is being destroyed. This will take several minutes

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%