xgboost models will not load

[bdod6]

Hello,

I've trying to use Clipper to deploy an xgboost model. However, the model container instantly exits after I run the python deployer. This is the traceback I'm getting from the logs:

Starting PythonContainer container
Connecting to Clipper with default port: 7000
Initializing Python function container
/opt/conda/lib/python2.7/site-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
Traceback (most recent call last):
  File "/container/python_closure_container.py", line 91, in <module>
    model = PythonContainer(model_path, input_type)
  File "/container/python_closure_container.py", line 25, in __init__
    self.predict_func = load_predict_func(predict_path)
  File "/container/python_closure_container.py", line 14, in load_predict_func
    return cloudpickle.load(serialized_func_file)
  File "/opt/conda/lib/python2.7/pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "/opt/conda/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/opt/conda/lib/python2.7/pickle.py", line 1096, in load_global
    klass = self.find_class(module, name)
  File "/opt/conda/lib/python2.7/pickle.py", line 1130, in find_class
    __import__(module)
  File "/model/modules/xgboost/__init__.py", line 11, in <module>
    from .core import DMatrix, Booster
  File "/model/modules/xgboost/core.py", line 112, in <module>
    _LIB = _load_lib()
  File "/model/modules/xgboost/core.py", line 106, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/opt/conda/lib/python2.7/ctypes/__init__.py", line 440, in LoadLibrary
    return self._dlltype(name)
  File "/opt/conda/lib/python2.7/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /model/modules/xgboost/./lib/libxgboost.so: invalid ELF header
Encountered error not related to missing packages. Please refer to the container log to diagnose.

I'm not sure if this is an issue with Clipper or with the compiler within the docker container. I don't know why the model container would exit immediately on deployment like this?

[withsmilo]

Default PyClosureContainerDockerfile and python_container_conda_deps.txt don't have the xgboost module installed. You must create own DockerFile to install xgboost module according to the this reference.

[bdod6]

Yep, I did install xgboost using a separate Dockerfile and called it as the build image. I confirmed this after running docker exec pip list and xgboost is shown as installed. So I don’t believe this is an import error.

[withsmilo]

Your error log shows that Clipper failed to call LoadLibrary() in your own container. I think that xgboost module needs to reinstall, maybe to rebuild. (refer to this link: dmlc/xgboost#463)

[bdod6]

Ok I figured it out. Docker requires the right gcc compiler in order for xgboost to work. However, even after installing gcc, it still doesn't compile correctly. That's because the default Clipper conda environ has an outdated version of /bin/../lib/libgomp.so.1. You have to also run conda install libgcc to update the file.

This is my Dockerfile that I got to run:

FROM clipper/python-closure-container:0.2

RUN apt-get update; \
    apt-get install -y \
      libgomp1 \
      python python-pip \
      build-essential \
      python-dev \
      python-setuptools \
      python-matplotlib \
      libatlas-dev \
      curl \
      libatlas3gf-base && \
      apt-get clean

# Install pip
RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

RUN pip install numpy
RUN pip install scipy
RUN pip install -U scikit-learn
RUN pip install --pre xgboost
RUN conda install libgcc

RUN update-alternatives --set libblas.so.3 \
      /usr/lib/atlas-base/atlas/libblas.so.3; \
    update-alternatives --set liblapack.so.3 \
     /usr/lib/atlas-base/atlas/liblapack.so.3