Serving a trained model

[reza79sh]

I have a ktrain text classifier based on BERT. What would be the right way to go about saving the model for serving?

[kinoute]

I'm about to be in a same situation and thought I will basically save the predictor after training, and load it in a API Container to get my predictions.

[amaiya]

Yes, you could serve the model by taking a Predictor object (or the underlying Keras model itself) and wrap it in a Flask app like this.

[Bidek56]

I have tried the Flask app approach but I get: RuntimeError: The Session graph is empty error.
Has anyone tried the TensorFlow Serving instead?

[amaiya]

For those of you who are trying to serve a ktrain model with Flask:

It looks like this is an issue with Flask/TensorFlow, not ktrain. The latest version of Flask causes a Session graph is empty error when trying to serve a TensorFlow model on TensorFlow 1.14. See this Keras issue for more information.

It apparently works in TensorFlow 2.0. However, when using a pre-v0.8 version of ktrain on TensorFlow 2, ktrain still runs in TensorFlow 1.x mode in order to support both TF 1.14 and 2.0 right now. This is why you see this error on both TF 1.14 and TF 2.0 when using ktrain.

This will no longer be a problem in ktrain v0.8 (which has not yet been released) because this version of ktrain will only support TensorFlow 2 (not TensorFlow 1.14).

For right now, the workaround is to downgrade Flask with: pip3 install flask==0.12.2. After doing this, you should be able to use Flask to serve a Keras model or ktrain predictor: For instance, I've verified the following toy example works:

# file name:   my_server.py
import flask
import ktrain
app = flask.Flask(__name__)
predictor = None
def load_predictor():
    global predictor
    predictor = ktrain.load_predictor('/tmp/mypred')
    if hasattr(predictor.model, '_make_predict_function'):
        predictor.model._make_predict_function()

@app.route('/predict', methods=['GET'])
def predict():
    data = {"success": False}
    if flask.request.method in ["GET"]:
        text = flask.request.args.get('text')
        if text is None: return flask.jsonify(data)
        prediction = predictor.predict(text)
        data['prediction'] = prediction
        data["success"] = True
    return flask.jsonify(data)

if __name__ == "__main__":
    load_predictor()
    port = 8888
    app.run(host='0.0.0.0', port=port)
    app.run()

After starting the server with python3 my_server.py, you can issue a prediction request to the server by opening your browser and typing: http://0.0.0.0:8888/predict?text=great%20movie

If the model was trained on IMDB, this should display the following in the browser:

prediction: "pos"
success: true