tensorflow-serving-openfaas

Example of packaging TensorFlow Serving with OpenFaaS to be deployed and managed through OpenFaaS with auto-scaling, scale-from-zero and a sane configuration for Kubernetes.

This example was adapted from: https://www.tensorflow.org/serving

Pre-reqs

• OpenFaaS
• OpenFaaS CLI
• Docker

Instructions

• Clone the repo

$ mkdir -p ~/dev/
$ cd ~/dev/
$ git clone https://github.com/alexellis/tensorflow-serving-openfaas

• Clone the sample model and copy it to the function's build context

$ cd ~/dev/tensorflow-serving-openfaas

$ git clone https://github.com/tensorflow/serving

$ cp -r serving/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_cpu ./ts-serve/saved_model_half_plus_two_cpu

• Edit the Docker Hub username

You need to edit the stack.yml file and replace alexellis2 with your Docker Hub account.

• Build the function image

$  faas-cli build

You should now have a Docker image in your local library which you can deploy to a cluster with faas-cli up

• Test the function locally

All OpenFaaS images can be run stand-alone without OpenFaaS installed, let's do a quick test, but replace alexellis2 with your own name.

$ docker run -p 8081:8080 -ti alexellis2/ts-serve:latest

Now in another terminal:

$ curl -d '{"instances": [1.0, 2.0, 5.0]}' \
   -X POST http://127.0.0.1:8081/v1/models/half_plus_two:predict

{
    "predictions": [2.5, 3.0, 4.5
    ]
}

From here you can run faas-cli up and then invoke your function from the OpenFaaS UI, CLI or REST API.

$ export OPENFAAS_URL=http://127.0.0.1:8080

$ curl -d '{"instances": [1.0, 2.0, 5.0]}' $OPENFAAS_URL/function/ts-serve/v1/models/half_plus_two:predict

{
    "predictions": [2.5, 3.0, 4.5
    ]
}

Find out more information about your function's endpoints:

$ faas-cli describe ts-serve
Name:                ts-serve
Status:              Ready
Replicas:            1
Available replicas:  1
Invocations:         5
Image:               alexellis2/ts-serve:latest
Function process:    
URL:                 http://127.0.0.1:8080/function/ts-serve
Async URL:           http://127.0.0.1:8080/async-function/ts-serve
Labels:              com.openfaas.function : ts-serve
                     function : true

• Try your own model

Now you can try your own model by editing ts-serve/Dockerfile.

Let me know what you think via Twitter