HiGHS in Docker

[iblancoi]

Hi there,

I am trying to build and run HiGHS from a Docker container (basically I want to run a python script that generates an MPS file and run this very same MPS file from the command line using HiGHS. This solution works well locally, the problem comes when installing HiGHS in Docker.

Basically, the image builds fine but I get all sorts of errors when running it. I think all these errors could be related to the Linux amd64 configuration of the Docker and that HiGHS does not support Linux but Windows or Mac.

Has anyone ever run HiGHS from Linux or installed it in a Docker image? Here is the Dockerfile:

FROM python:3.9.13

#ENV PYTHONPATH ${PYTHONPATH}:/src/"
WORKDIR /.

# ADD data .
ADD requirements.txt .

# install packages
RUN pip install -r requirements.txt

# copy project
COPY ./src ./

RUN set -ex;                       \
    apt-get update;             \
    apt-get install -qq cmake;  \
    git clone https://github.com/ERGO-Code/HiGHS.git; \
    cd HiGHS; \
    mkdir build;\
    cd build;\
    cmake ..;\
    cmake --build . ;\
    cmake --install . 

CMD [ "python", "main.py" ]

Thanks in advance :)

[sschnug]

Off the top of my head i don't see any issues related to Highs in regards to container-based usage.

On the contrary. Due to it's system-language based implementation (without many dependencies), it should be very easy to achieve something (opposed to less statically-compiled things like your python code).

Can you elaborate on your issues? Without some "mock-like" steps, we cannot really reproduce what you are really doing.

Just some remarks:

• "that HiGHS does not support Linux but Windows or Mac."
  • That's not true: I would argue, that Linux is the most important dev-target of Highs!
• If i have to guess, your problems are probably due to filesystem / permission things
  • Working with files within container is imho no fun! Care has to be taken!
    • I don't really know what you are doing, but before i would write a MPS to "disk" and run a binary on it, i would use Pythons StringIO or similar to "write into memory" and pass that as "pipe" to some binary
      • I never did that (as i'm usually focusing on lib-based aka shared-memory usage instead of crossing process-boundaries) but i guess it's much less trouble than filesystems / permissions
        • Hmm... i'm not sure if Highs supports stdin-based input right now
• I wonder why you should be able to build Highs with a base-image like "FROM python:3.9.13"
  • Why would that base-image contain a C++ compiler?
    • That's a strange python-based base-image
• I wonder if "FROM python:3.9.13" is a good base-image
• In general, i would highly recommend multi-stage builds
  • Single-stage builds like that will result in ~ 500 MB extra you don't really need (your final runtime-container does not need all the sources, a C++ compiler and sometimes not even a shell)

Edit: Minimal Demo

This is quickly hacked together. I still don't like workflows like these (filesystem!).

Dockerfile

    # Build-Container Stage
    # ---------------------

    FROM ubuntu:20.04 as builder

    ENV DEBIAN_FRONTEND=noninteractive
    RUN apt-get update && apt-get install -y curl build-essential cmake

    WORKDIR /WORKING

    RUN curl -L -o highs.tar.gz https://github.com/ERGO-Code/HiGHS/archive/refs/tags/v1.2.2.tar.gz

    RUN tar -xf highs.tar.gz --strip-components=1

    RUN mkdir build && \
        cd build && \
        cmake .. -DSHARED=Off && \
        make

    # Runtime-Container Stage
    # -----------------------

    FROM python:3.9.13

    # we only use the (hopefully "static enough") compiled binary from the first stage!
    COPY --from=builder /WORKING/build/bin/highs /usr/local/bin/highs

    WORKDIR /usr/src/app

    COPY . .

    CMD [ "python", "write_mps_and_call_highs.py" ]

write_mps_and_call_highs.py

    mps_problem = """
    NAME          TESTPROB
    ROWS
    N  COST
    L  LIM1
    G  LIM2
    E  MYEQN
    COLUMNS
        XONE      COST                 1   LIM1                 1
        XONE      LIM2                 1
        YTWO      COST                 4   LIM1                 1
        YTWO      MYEQN               -1
        ZTHREE    COST                 9   LIM2                 1
        ZTHREE    MYEQN                1
    RHS
        RHS1      LIM1                 5   LIM2                10
        RHS1      MYEQN                7
    BOUNDS
    UP BND1      XONE                 4
    LO BND1      YTWO                -1
    UP BND1      YTWO                 1
    ENDATA"""

    import os
    import tempfile
    import subprocess

    # - /tmp which will be used should be relatively pain-free in regards to user-permissions
    # - highs does want .mps extension for identification
    file, filepath = tempfile.mkstemp(suffix='.mps')

    with open(file, 'w') as fileTemp:
      fileTemp.write(mps_problem)

    print(subprocess.run(["highs {}".format(filepath)], shell=True, capture_output=True)) 

    os.remove(filepath)

Test it

docker build -t highs_experiment .

docker run --rm highs_experiment

x@y:~/Tmp/highs_docker$ docker run --rm highs_exp

CompletedProcess(
    args=['highs /tmp/tmpd8e1l8py.mps'],
    returncode=0,
    stdout=b'Running HiGHS 1.2.1 [date: 2022-09-15, git hash: n/a]\n
    Copyright (c) 2022 ERGO-Code under MIT licence terms\n
    LP   tmpd8e1l8py has 3 rows; 3 cols; 6 nonzeros\n
    Presolving model\n
    1 rows, 2 cols, 2 nonzeros\n
    0 rows, 0 cols, 0 nonzeros\n
    Presolve : Reductions: rows 0(-3); columns 0(-3); elements 0(-6) - Reduced to empty\n
    Solving the original LP from the solution after postsolve\n
    Model   status      : Optimal\n
    Objective value     :  5.4000000000e+01\n
    HiGHS run time      :          0.00\n
    ', stderr=b'')

(linebreaks manually added for readability)

[ignacio-blanco-maersk]

Hey @sschnug, thanks a lot for the super elaborated answer :)

You are absolutely right, once I found this page (https://github.com/JuliaBinaryWrappers/HiGHS_jll.jl/releases), it was actually very easy to install HiGHS in the container. Just running the tar.gz was enough to make HiGHS work.

Thanks for the tip on the MPS file, this solution is very similar to what I am currently doing.

However, I have also another question which you may have a better solution for what I am doing. Currently, once HiGHS solves, in order to extract the variables, I export the solution to a text file and read this solution from python to extract all this data. Do you know a better way to read the solution output?

Thanks again for your help and good work.

[sschnug]

It seems you commited to the CLI-based usage with in- and output files. Then it's only natural, that you somehow need to parse the output files.
(In this case i would also recommend adapting above from mkstemp to mkdtemp such that you can point Highs to the temp-directory)

As i'm not familiar with this workflow, i don't know if there is a well-established solution-format which works across solver-boundaries such that there would be chances of software already being available helping here. I somewhat doubt, that there is something helpful.

[mckib2]

@iblancoi Is it important that that problem is written as an MPS file? If you're using Python already, you can use scipy.optimize.linprog or scipy.optimize.milp

[DDAK]

@iblancoi Can you check if this helps: https://github.com/DDAK/HiGHS ?