Merge branch 'one2'

int-brain-lab · Oct 13, 2021 · c3025dc · c3025dc
2 parents ed5aa56 + adcd23d
commit c3025dc
Show file tree

Hide file tree

Showing 13 changed files with 112 additions and 135 deletions.
diff --git a/README.md b/README.md
@@ -25,10 +25,10 @@ output = dlc("Path/to/file.mp4")
 from iblvideo import run_session
 run_session("db156b70-8ef8-4479-a519-ba6f8c4a73ee")
 ```
-### Running 10 sessions from the queue
+### Running the queue
 ```python
 from iblvideo import run_queue
-run_queue(n_sessions=10)
+run_queue(machine='mymachine')
 ```
 ## Accessing results
 
@@ -40,88 +40,71 @@ DLC results are stored on the Flatrion server, with the `dataset_type` being `ca
 
 Install local server as per [this instruction](https://docs.google.com/document/d/1NYVlVD8OkwRYUaPeHo3ZFPuwpv_E5zgUVjLsV0V5Ko4).
 
-Install CUDA 10.0 libraries as documented [here](https://docs.google.com/document/d/1UyXUOx21mwrpBtCcS51avnikmyCPCzXEtTRaTetH-Mo) to match the TensorFlow version 1.13.1 required for DLC.
+Install CUDA 11.2 libraries as documented [here](https://docs.google.com/document/d/1UyXUOx21mwrpBtCcS51avnikmyCPCzXEtTRaTetH-Mo/edit#heading=h.39mk45fhbn1l)
 
 Install cuDNN, an extension of the Cuda Toolkit for deep neural networks: Download cuDNN from FlatIron as shown below, or find it online.
 
 ```bash
-wget --user iblmember --password check_your_one_settings http://ibl.flatironinstitute.org/resources/cudnn-10.0-linux-x64-v7.6.5.32.tgz  
-tar -xvf cudnn-10.0-linux-x64-v7.6.5.32.tgz  
-sudo cp cuda/include/cudnn.h /usr/local/cuda-10.0/include  
-sudo cp cuda/lib64/libcudnn* /usr/local/cuda-10.0/lib64  
-sudo chmod a+r /usr/local/cuda-10.0/include/cudnn.h /usr/local/cuda-10.0/lib64/libcudnn*  
+wget --user iblmember --password check_your_one_settings http://ibl.flatironinstitute.org/resources/cudnn-11.2-linux-x64-v8.1.1.33.tgz 
+tar -xvf cudnn-11.2-linux-x64-v8.1.1.33.tgz 
+sudo cp cuda/include/cudnn.h /usr/local/cuda-11.2/include  
+sudo cp cuda/lib64/libcudnn* /usr/local/cuda-11.2/lib64  
+sudo chmod a+r /usr/local/cuda-11.2/include/cudnn.h /usr/local/cuda-11.2/lib64/libcudnn*  
 ```
 
-(optional): check CUDNN installation or for troubleshooting only)
-Download and unzip https://ibl.flatironinstitute.org/resources/cudnn_samples_v7.zip
-If necessary, setup your CUDA environment variables with the version you want to test
-
-```
-cd cudnn_samples_v7/mnistCUDNN/
-make clean && make
-./mnistCUDNN
-```
-
-Should print a message that finishes with Test passed !
-
 ### Create a Python environment with TensorFlow and DLC
 
-Install a few things system wide and then python3.7
+Install python3.8 (if necessary)
 
 ```bash
 sudo apt update  
 sudo apt install software-properties-common  
 sudo add-apt-repository ppa:deadsnakes/ppa
-sudo apt-get install python3.7-tk  
-sudo apt install python3.7 python3.7-dev  
+sudo apt-get install python3.8-tk  
+sudo apt install python3.8 python3.8-dev  
 ```
 
-Create and activate a Python 3.7 environment called e.g. dlcenv
+Create and activate an environment called e.g. dlcenv
 
 ```bash
 mkdir -p ~/Documents/PYTHON/envs
 cd ~/Documents/PYTHON/envs
-virtualenv dlcenv --python=python3.7
+virtualenv dlcenv --python=python3.8
 source ~/Documents/PYTHON/envs/dlcenv/bin/activate
 ```
 
-Install packages (please observe order of those commands and Ubuntu version, you may need to change the wxpython link!)
+Install packages
 
 ```bash
 pip install -U setuptools
-pip install git+https://github.com/int-brain-lab/ibllib.git@1.11.0
-pip install https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/wxPython-4.0.7-cp37-cp37m-linux_x86_64.whl  
-pip install tensorflow-gpu==1.13.1  
-pip install deeplabcut==2.1.10  
+pip install git+https://github.com/int-brain-lab/ibllib.git
+pip install tensorflow
+pip install deeplabcut
 ```
 
-### Test if tensorflow and deeplabcut installation was successful
-
-Export environment variable to avoid tensorflow issue and point to CUDA libraries
+Export environment variables 
+(the second command might already be set up with your cuda install)
 ```bash
 export TF_FORCE_GPU_ALLOW_GROWTH='true'
-export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/extras/CUPTI/lib64:/lib/nccl/cuda-10:$LD_LIBRARY_PATH  
+export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH  
 ```
 
-Try importing tensorflow and deeplabcut
+### Test if tensorflow and deeplabcut installation was successful
+
 ```
 python -c 'import deeplabcut, tensorflow'
 ```
 
-(If you get a pandas incompatibility error, try this:
-```
-pip uninstall pandas
-pip install pandas==1.1.5
-```
-)
+If you get errors make sure the cuda libraries are correctly added to your paths in .bashrc as explained [here](https://docs.google.com/document/d/1UyXUOx21mwrpBtCcS51avnikmyCPCzXEtTRaTetH-Mo/edit#heading=h.ryxfckh2bbpl).
 
-If this is successful (no errors) you can set up an alias in your .bashrc file to easily enter the iblvideo environment:
+Once the import goes through without errors (it is ok to get the warning that you cannot use the GUI), you can set up an alias in your .bashrc file to easily enter the iblvideo environment:
 ```bash
 nano ~/.bashrc
 ```
 Enter this line under the other aliases:
 ```bash
-alias dlcenv="export TF_FORCE_GPU_ALLOW_GROWTH='true'; export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:/usr/local/cuda-10.0/extras/CUPTI/lib64:$LD_LIBRARY_PATH; source ~/Documents/PYTHON/envs/dlcenv/bin/activate"
+alias dlcenv="export TF_FORCE_GPU_ALLOW_GROWTH='true'; export LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH; source ~/Documents/PYTHON/envs/dlcenv/bin/activate"
+
 ```
 After opening a new terminal you should be able to type `dlcenv` and end up in an environment in which you can import tensorflow and deeplabcut like above.
 
@@ -160,10 +143,11 @@ The version of DLC weights and DLC test data are synchronized with the MAJOR.MIN
 
 If you update any of the DLC weights, you also need to update the MINOR version of the code and the DLC test data, and vice versa.
 1. For the weights, create a new directory called `weights_v{MAJOR}.{MINOR}` and copy the new weights, plus any unchanged weights into it.
-2. Make a new `dlc_test_data_v{MAJOR}.{MINOR}` directory, with subdirectories `input` and `output`.
+2. Make a new `dlc_test_data_v{MAJOR}.{MINOR}` directory.
 3. Copy the three videos from the `input` folder of the previous version dlc_test_data to the new one.
-4. Create the three parquet files to go in `output` by running iblvideo.dlc() with the new weights folder as `path_dlc`, and each of the videos in the new `input` folder as `file_mp4`.
-5. Zip and upload the new weights and test data folders to FlatIron :
+4. Create the three parquet files to go in by running iblvideo.dlc() with the new weights folder as `path_dlc`, and each of the videos in the new `input` folder as `file_mp4`.
+5. Rename the newly created folder `alf` inside the dlc_test_data folder into `output`.
+6. Zip and upload the new weights and test data folders to FlatIron :
 ```
 /resources/dlc/weights_v{MAJOR}.{MINOR}.zip
 /integration/dlc/test_data/dlc_test_data_v{MAJOR}.{MINOR}.zip

diff --git a/docker/Dockerfile b/docker/Dockerfile
diff --git a/docker/Dockerfile.base b/docker/Dockerfile.base
@@ -1,5 +1,5 @@
 #docker build -t internationalbrainlab/dlc:base -f Dockerfile.base
-FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
+FROM nvidia/cuda:11.2.2-cudnn8-devel-ubuntu18.04
 
 # link the cuda libraries
 ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
@@ -8,22 +8,21 @@ ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
 ENV TZ=Europe/Paris
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
-# Install python3.7
+# Install python3.8 and some other packages
 RUN apt-get update
 RUN apt-get install -y software-properties-common
 RUN add-apt-repository ppa:deadsnakes/ppa
 RUN apt-get update
-RUN apt-get install -y python3.7 python3.7-dev python3.7-tk python3-pip python3.7-venv git ffmpeg libgtk-3-dev
+RUN apt-get install -y python3.8 python3.8-dev python3.8-tk python3-pip python3.8-venv git ffmpeg libgtk-3-dev
 
 # Install Python dependencies
-ARG PYTHON=python3.7
+ARG PYTHON=python3.8
 ENV LANG C.UTF-8
 RUN ln -sf /usr/bin/${PYTHON} /usr/local/bin/python3
 RUN python3 -m pip install --upgrade pip
-RUN python3 -m pip install "dask[complete]"
-RUN python3 -m pip install ibllib>=1.9.0
-RUN python3 -m pip install https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-18.04/wxPython-4.0.7-cp37-cp37m-linux_x86_64.whl
-RUN python3 -m pip install tensorflow-gpu==1.13.1
+RUN python3 -m pip install -U setuptools
+RUN python3 -m pip install git+https://github.com/int-brain-lab/ibllib.git
+RUN python3 -m pip install tensorflow
 RUN python3 -m pip install deeplabcut
 
-ENV TF_FORCE_GPU_ALLOW_GROWTH 'true'
+ENV TF_FORCE_GPU_ALLOW_GROWTH 'true'
diff --git a/docker/Dockerfile.v2.0 b/docker/Dockerfile.v2.0
@@ -0,0 +1,11 @@
+#docker build -t internationalbrainlab/dlc:v2.0 -f Dockerfile.v2.0 --no-cache .
+FROM internationalbrainlab/dlc:base
+
+# copy the weights and test data
+COPY weights_v2.0 weights_v2.0
+COPY me_test_data me_test_data
+COPY dlc_test_data_v2.0 dlc_test_data_v2.0
+
+# clone the code
+RUN git clone --depth 1 --single-branch --branch one2 https://github.com/int-brain-lab/iblvideo.git
+ENV PYTHONPATH=/iblvideo
diff --git a/docker/README_DOCKER.md b/docker/README_DOCKER.md
@@ -104,15 +104,15 @@ Update the image version in `docker-compose.yaml` test and queue configurations.
 cd ~/Documents/PYTHON
 git clone https://github.com/int-brain-lab/iblvideo
 cd iblvideo/docker
-docker build -t internationalbrainlab/dlc:base -f Dockerfile.base  # this one will take a long time
+docker build -t internationalbrainlab/dlc:base -f Dockerfile.base --no-cache .  # this one will take a long time
 ```
 
 Test the image by accessing a shell inside of the container:
 
 ```shell
 docker run -it --rm --gpus all -u $(id -u):$(id -g) -v /mnt/s0/Data/FlatIron:/mnt/s0/Data/FlatIron -v ~/Documents/PYTHON/iblvideo:/root internationalbrainlab/dlc:base
 python3
-from import deeplabcut
+import deeplabcut
 ```
 
 Eventually send the image to dockerhub

diff --git a/docker/docker-compose.yaml b/docker/docker-compose.yaml
@@ -17,14 +17,14 @@ services:
       - ~/Documents/PYTHON/iblvideo:/iblvideo
       - ~:/root
   queue:
-    image: internationalbrainlab/dlc:v1.2
-    command: python3 -c "from iblvideo import run_queue; run_queue()"
+    image: internationalbrainlab/dlc:v2.0
+    command: python3 -c "from iblvideo import run_queue; run_queue(target_versions=('1.2.0', '2.0.0'))"
     runtime: nvidia
     volumes:
       - ~:/root
       - /mnt/s0/Data/FlatIron:/mnt/s0/Data/FlatIron
   test:
-    image: internationalbrainlab/dlc:v1.2
+    image: internationalbrainlab/dlc:v2.0
     command: bash -c "nvcc --version; pytest -s /iblvideo/iblvideo/tests"
     runtime: nvidia
     volumes:

diff --git a/iblvideo/__init__.py b/iblvideo/__init__.py
@@ -1,5 +1,5 @@
 
-__version__ = '1.2.0'  # This is the only place where the version is hard coded, only adapt here
+__version__ = '2.0.0'  # This is the only place where the version is hard coded, only adapt here
 
 import deeplabcut
 

diff --git a/iblvideo/motion_energy.py b/iblvideo/motion_energy.py
@@ -14,10 +14,8 @@
 import cv2
 import logging
 
-from oneibl.one import ONE
 from ibllib.io.video import get_video_frames_preload, label_from_path
 from ibllib.io.extractors.camera import get_video_length
-from oneibl.stream import VideoStreamer
 
 _log = logging.getLogger('ibllib')