Consider dockerizing spec tests for reproducible builds

[michaelneuder]

I spent a good chunk of time just battling different python3, pip, and venv verioning issues to try to get the spec tests running locally. Eventually I had to run #3304 to deal with a Py03 module error.

This made me wonder if dockerizing the spec tests could make them easier to interact with. Each version could be a container that you pull and run locally, which would have a specific python version and the respective packages that we know work. Just an idea, but if there is value here I would be happy to explore :-)

[wenceslas-sanchez]

Hi @michaelneuder and @hwwhww,

I also faced issues to run the project, and I did the following:

• a Dockerfile to build an image that store the project + all dependencies.
• scripts to replace make command.

The Dockerfile is stored at consensus-specs/docker:

FROM python:3.11.0-slim-bullseye as base

RUN mkdir /consensus-specs
WORKDIR /consensus-specs

FROM base as builder
ENV DEBIAN_FRONTEND=noninteractive
RUN apt update && apt install -y git
RUN python -m pip install --upgrade pip

COPY . .

RUN pip install -r requirements-dev.txt -r requirements_preinstallation.txt
RUN python setup.py pyspecdev

A requirements-dev.txt is stored at the root:

pytest==7.4.0
pytest-cov==4.1.0
pytest-xdist==3.3.1
lru-dict==1.2.0
remerkleable==0.1.27
py_ecc==6.0.0
py_arkworks_bls12381==0.3.4
milagro_bls_binding==1.9.0
trie==2.0.2
eth-hash==0.5.2
pycryptodome==3.15.0

A script named run_tests.sh is stored in consensus-specs/scripts to replace make test:

# /bin/sh

# constants
ALL_EXECUTABLE_SPECS=("phase0" "altair" "bellatrix" "capella" "deneb" "eip6110" "whisk")
TEST_PRESET_TYPE=minimal
WORKDIR="//consensus-specs//tests//core//pyspec"

COV_HTML_OUT=.htmlcov

# default flag value
version=latest
number_of_core=4

# generate coverage scope
name="eth2spec"
coverage_scope=()
for spec in "${ALL_EXECUTABLE_SPECS[@]}"
do
    coverage_scope+=("--cov=${name}.${spec}.${TEST_PRESET_TYPE}")
done

# Parse flags
while getopts v:n: flag
do
    case "${flag}" in
        v) version=${OPTARG};;
        n) number_of_core=${OPTARG};;
        \?) echo "not valid -$OPTARG:${OPTARG}";;
    esac
done

# Get IDs of containers that run the image `consensus-specs:$version`.
get_container_name() {
  echo $(docker ps -a -q --filter ancestor="consensus-specs:$version" --format="{{.ID}}")
}

# Stop and remove all container that use the `consensus-specs:$version` image
cleanup() {
  echo "Stop and remove current container"
  docker rm $(docker stop $(get_container_name))
}

copy_coverage_report() {
  docker cp $(get_container_name):$WORKDIR/$COV_HTML_OUT ./$COV_HTML_OUT
}

# Equivalent to `make test`
docker run -w $WORKDIR consensus-specs:$version \
    pytest -n $number_of_core --disable-bls $coverage_scope --cov-report="html:${COV_HTML_OUT}" --cov-branch eth2spec

# Get coverage report form container instance
$(copy_coverage_report)

# Stop and remove all containers that use the `consensus-specs:$version` image when exiting
#  the script. It helps user to limit container instances.
trap cleanup EXIT

# for testing purpose
$SHELL

Then we can run the following at the root of the repo:

1. docker build -f ./docker/Dockerfile -t consensus-specs:0.0.0 .
2. run_tests.sh -v 0.0.0 -n 4 if you are located to consensus-specs/scripts or if you are at the root of the project pushd && cd ./scripts && run_tests.sh -v 0.0.0 -n 4 && popd

The coverage report is available at the root.

[hwwhww]

Hey @wenceslas-sanchez

I’m not a docker master, but it looks good.

It would be nice to add a CircleCI job to verify if the Dockerfile is correct, but no need to run all the tests in CI. Perhaps just have a dockerfile_test.sh that runs linter or so.

[wenceslas-sanchez]

Hi @hwwhww

All right, I will make a PR soon 👍