Add demo from O'Reilly London 2019

demo/README.md

@@ -0,0 +1,17 @@
+Ray Live Demo
+=============
+
+This demo can be run live while presenting to show Ray's capabilities.
+
+## Instructions
+1. Launch a Ray cluster on AWS with `ray up cluster_config.yaml`
+    - To run locally, install the requirements with `pip install -r requirements.txt`
+2. Connect to the head node
+    - I recommend using SSH with port forwarding in order to use Jupyter, Ray Dashboard, and Tensorboard without compromising security
+    - For example, `ssh -L 9999:127.0.0.1:8889 -L 9998:127.0.0.1:8080 -L 9997:127.0.0.1:6006 ubuntu@12.123.123.123` should map Jupyter to `127.0.0.1:9999`, Ray Dashboard to `127.0.0.1:9998` and Tensorboard to `127.0.0.1:9997`
+2. Open the jupyter notebooks on the cluster and set the `CLUSTER_ADDRESS` parameter in `ray_api_demo.ipynb` and `rllib_demo.ipynb`
+3. Also set links for Ray Dashboard and Tensorboard
+4. Run the live-coding presentation with [rise](https://rise.readthedocs.io/)
+    - Start a presentation with `Alt-r` or by pressing the button in the top right of the toolbar
+    - Use `SpaceBar` to navigate to the next slide and `Shift-SpaceBar` to navigate to the previous slide
+    - Use `Shift-Enter` to run the code in a cell

demo/cluster_config.yaml

@@ -0,0 +1,114 @@
+# An unique identifier for the head node and workers of this cluster.
+cluster_name: ray-demo
+
+# The minimum number of workers nodes to launch in addition to the head
+# node. This number should be >= 0.
+min_workers: 2
+
+# The maximum number of workers nodes to launch in addition to the head
+# node. This takes precedence over min_workers.
+max_workers: 5
+
+# The initial number of worker nodes to launch in addition to the head
+# node. When the cluster is first brought up (or when it is refreshed with a
+# subsequent `ray up`) this number of nodes will be started.
+initial_workers: 2
+
+# Whether or not to autoscale aggressively. If this is enabled, if at any point
+#   we would start more workers, we start at least enough to bring us to
+#   initial_workers.
+autoscaling_mode: default
+
+# The autoscaler will scale up the cluster to this target fraction of resource
+# usage. For example, if a cluster of 10 nodes is 100% busy and
+# target_utilization is 0.8, it would resize the cluster to 13. This fraction
+# can be decreased to increase the aggressiveness of upscaling.
+# This value must be less than 1.0 for scaling to happen.
+target_utilization_fraction: 0.8
+
+# If a node is idle for this many minutes, it will be removed.
+idle_timeout_minutes: 5
+
+# Cloud-provider specific configuration.
+provider:
+    type: aws
+    region: us-west-2
+    # Availability zone(s), comma-separated, that nodes may be launched in.
+    # Nodes are currently spread between zones by a round-robin approach,
+    # however this implementation detail should not be relied upon.
+    availability_zone: us-west-2a,us-west-2b
+
+# How Ray will authenticate with newly launched nodes.
+auth:
+    ssh_user: ubuntu
+# By default Ray creates a new private keypair, but you can also use your own.
+# If you do so, make sure to also set "KeyName" in the head and worker node
+# configurations below.
+#    ssh_private_key: /path/to/your/key.pem
+
+# Provider-specific config for the head node, e.g. instance type. By default
+# Ray will auto-configure unspecified fields such as SubnetId and KeyName.
+# For more documentation on available fields, see:
+# http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
+head_node:
+    InstanceType: m5.large
+    ImageId: ami-02c253ecf7eaba73e  # Deep Learning AMI (Ubuntu 16.04) Version 24.2
+
+    # You can provision additional disk space with a conf as follows
+    BlockDeviceMappings:
+        - DeviceName: /dev/sda1
+          Ebs:
+              VolumeSize: 100
+
+    # Additional options in the boto docs.
+
+# Provider-specific config for worker nodes, e.g. instance type. By default
+# Ray will auto-configure unspecified fields such as SubnetId and KeyName.
+# For more documentation on available fields, see:
+# http://boto3.readthedocs.io/en/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances
+worker_nodes:
+    InstanceType: m5.large
+    ImageId: ami-0b294f219d14e6a82 # Deep Learning AMI (Ubuntu) Version 21.0
+
+    # Run workers on spot by default. Comment this out to use on-demand.
+    InstanceMarketOptions:
+        MarketType: spot
+        # Additional options can be found in the boto docs, e.g.
+        #   SpotOptions:
+        #       MaxPrice: MAX_HOURLY_PRICE
+
+    # Additional options in the boto docs.
+
+# Files or directories to copy to the head and worker nodes. The format is a
+# dictionary from REMOTE_PATH: LOCAL_PATH, e.g.
+file_mounts: {
+    "/home/ubuntu/ray_api_demo.ipynb": "./ray_api_demo.ipynb",
+    "/home/ubuntu/rllib_demo.ipynb": "./rllib_demo.ipynb",
+    "/home/ubuntu/utils.py": "./utils.py",
+}
+
+# List of shell commands to run to set up nodes.
+setup_commands:
+    - echo 'export PATH="$HOME/anaconda3/envs/tensorflow_p36/bin:$PATH"' >> ~/.bashrc
+    - pip install -U ray ray[rllib] ray[tune] ray[debug]
+    - pip install aiohttp psutil gym opencv-python
+    - pip install lz4
+
+# Custom commands that will be run on the head node after common setup.
+head_setup_commands:
+    - pip install RISE
+    - pip install boto3==1.4.8  # 1.4.8 adds InstanceMarketOptions
+    - pip install -U tensorflow
+
+# Custom commands that will be run on worker nodes after common setup.
+worker_setup_commands: []
+
+# Command to start ray on the head node. You don't need to change this.
+head_start_ray_commands:
+    - ray stop
+    - ulimit -n 65536; ray start --head --redis-port=6379 --object-manager-port=8076 --autoscaling-config=~/ray_bootstrap_config.yaml --include-webui
+
+# Command to start ray on worker nodes. You don't need to change this.
+worker_start_ray_commands:
+    - ray stop
+    - ulimit -n 65536; ray start --redis-address=$RAY_HEAD_IP:6379 --object-manager-port=8076