Skip to content

Latest commit

 

History

History
91 lines (61 loc) · 3.62 KB

host.md

File metadata and controls

91 lines (61 loc) · 3.62 KB
Raw

Landing OpenVINO™ Model Server on Bare Metal Hosts and Virtual Machines

Requirements

OpenVINO™ model server installation is fully tested on Ubuntu16.04, however there are no anticipated issues on other Linux distributions like CentoOS*, RedHat*, ClearLinux* or SUSE Linux*.

Download the installer from: OpenVINO™ toolkit

Installation of Dependencies

Generally, you need to repeat the steps listed in the Dockerfile

The steps include installation of Linux packages of cpio, make, wget, python3, python setup tools and virtualenv.

Later, OpenVINO™ toolkit need to be installed - at least the inference engine is required.

It is important to set correct environment variables PYTHONPATH and LD_LIBRARY_PATH pointing to Python modules of OpenVINO™ and compiled libraries.

Model Server Installation

It is recommended to run OpenVINO™ model server from Python virtual environment. This will avoid all conflicts between Python dependencies from other applications running on the same host.

Note All python dependencies are included in requirements.txt file.

OpenVINO™ Model server can be installed using commands:

make install

or if you don't want to create a virtual environment:

pip install .

or if you want to modify the source code and easily test changes:

pip install -e .

Starting the Serving Service

Process of starting the model server and preparation of the models is similar to docker containers.

When working inside the activated virtual environment or with all Python dependencies installed, the server can be started using ie_serving command:

ie_serving --help
usage: ie_serving [-h] {config,model} ...

positional arguments:
  {config,model}  sub-command help
    config        Allows you to share multiple models using a configuration file
    model         Allows you to share one type of model

optional arguments:
  -h, --help      show this help message and exit

The server can be started in interactive mode, as a background process or a daemon initiated by systemctl/initd depending on the Linux distribution and specific hosting requirements.

Refer to docker_container.md to get more details.

Using Neural Compute Sticks

OpenVINO Model Server can employ AI accelerators Intel® Neural Compute Stick and Intel® Neural Compute Stick 2.

To use Movidus Neural Compute Sticks with OpenVINO Model Server you need to have OpenVINO Toolkit with Movidius VPU support installed. In order to do that follow OpenVINO installation instruction. Don't forget about additional steps for NCS.

Before starting server, you need to specify that you want to load model on Neural Compute Stick for inference execution. You can do that by setting environment variable DEVICE to MYRIAD. If it's not specified, OpenVINO will try to load model on CPU.

Example:

export DEVICE=MYRIAD

ie_serving model --model_path /opt/model --model_name my_model --port 9001

You can also run it in Docker container

Note: Currently Neural Computing Sticks support only FP16 target precision. Make sure you have proper model. If not, take a look at OpenVINO Model Optimizer and convert your model to desired format.