Skip to content

Latest commit

 

History

History
234 lines (179 loc) · 9.25 KB

INSTALL.md

File metadata and controls

234 lines (179 loc) · 9.25 KB
Raw

Installation

Table of Contents
  1. Ubuntu 20.04/18.04 Preliminaries
  2. Mac OSX Preliminaries
  3. CentOS 7 Preliminaries
  4. Amazon Linux 2
  5. Windows Server 10 Preliminaries
  6. Pypi setup
  7. Preinstalled docker
  8. Common installation errors

fastdup is currently supported on Ubuntu 20.04 or 18.04 OS, CentOS 7.9, Amazon Linux 2, RedHat 4.8, Mac OS 10.X Intel chip, Mac OS 11.X M1 chip, Mac OS M1 pro chip 12.X 13.X, Windows 10 Server (via WSL).

Ubuntu 20.04/18.04 LTS Machine Setup

Required machine setup

sudo apt update
sudo apt -y install software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt -y install python3.8
sudo apt -y install python3-pip
sudo apt -y install libgl1-mesa-glx
python3.8 -m pip install --upgrade pip

Mac OS setup 

brew install ffmpeg@4

Note: when using conda on Mac M1 or Mac M1 Pro make sure you install conda for arm (and not conda for intel) otherwise it will not be possible to install.

Note: when using python installed via pyenv, python3.7 does not work since it is too old, use python3.8

CentOS 7 Setup / Amazon Linux 2 / RedHat 4.8 

sudo yum -y install epel-release
sudo yum -y update
sudo yum -y groupinstall "Development Tools"
sudo yum -y install openssl-devel bzip2-devel libffi-devel xz-devel
sudo yum -y install wget
sudo yum install redhat-lsb-core # for lsb_release
sudo yum install -y ffmpeg ffmpeg-devel # for video support

Download and istall CentOS 7 whl image from our release page.

python3.7 -m pip install <path of the downloaded whl>

Amazon Linux 2 Setup 

sudo yum install mesa-libGL -y

Download and isntall CentOS 7 whl image from our release page.

python3.7 -m pip install <path of the downloaded whl>

Windows 10 Server Setup

Setting up WSL. The below instructions are for Windows Server 10. More detailed instructions are here.

For Windows 10+11 follow the instructions here.

  • Enable WSL on your machine (Search -> powershell-> right click -> run as administrator)
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
  • Reboot your machine
  • Check that wsl is enabled using the command:
Get-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
  • Download ubuntu 18.04 from https://aka.ms/wslubuntu1804
  • After the download, run powershell as admin, goto your download folder (for example c:\users/danny_bickson/downloads/)
cd c:\\users\danny_bickson\Download # change to your download folders
Expand-Archive .\Ubuntu_1804.2019.522.0_x64.zip .\Ubuntu_1804
cd .\Ubuntu_1804\
 .\ubuntu1804.exe

This will take a few minutes, you will see an output of the kind:

Please create a default UNIX user account. The username does not need to match your Windows username.
For more information visit: https://aka.ms/wslusers
Enter new UNIX username: danny_bickson # (chnage to your username)
Enter new UNIX password: *******
Retype new UNIX password: *******
passwd: password updated successfully
Installation successful!
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Optional: Update WSL to version 2 (recommended, significant performance wins!)

Once WSL and Ubuntu 18.04 are set up, continue with the below instructions

  • Inside the Ubuntu shell, run the following installers (you will be asked for password you entered before).
sudo apt update
sudo apt -y install software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt -y install python3.8
sudo apt -y install python3-pip
sudo apt -y install libgl1-mesa-glx
pip3 install --upgrade pip
python3.8 -m pip install fastdup

In case python fails to find fastdup, do the following:

Pip Package setup

Using pypi

This is the recommended installation method for all Mac, Ubuntu 18-20, Debian 10, Windows Server 10. Will not work for Centos 7.9, RedHat 4.8.

python3.8 -m pip install -U pip
python3.8 -m pip install fastdup

Note: you may need to upgrade your pip, using the command python3.8 -m pip install -U pip.

Using stable release.

This is mandatory for CentOS 7.9 / RedHat 4.8 and similiar OS.

  • download the latest wheel for your system from our release page. Assuming the wheel file is found in your working folder, run:
python3.8 -m pip install *.whl

Note: you may need to upgrade your pip, using the command python3.8 -m pip install -U pip.

Debian package install

  • download the latest deb for your system from our release page. Assuming the wheel file is found in your working folder, run:
sudo dpkg -i fastdup-<VERSION>-ubuntu-20.04.deb

Application name is fastdup.

Docker

##Pull from docker hub the latest ubuntu

docker pull karpadoni/fastdup-ubuntu-20.04

Build your own docker

docker build -f Dockerfile -t fastdup-ubuntu .

Currently supported software/hardware

Operating system

  • Ubuntu 20.04 LTS
  • Ubuntu 18.04 LTS
  • Mac OSX M1 Chip (tested on Big Sur)
  • Mac OSX M2 Chip (tested on Ventura)
  • Mac Intel Chip (tested on Mojave)
  • CentOS 7.9 (via stable release)
  • Amazon Linux 2 (via stable release)
  • Windows 10 Server (via WSL)

Software versions

  • Python 3.7, 3.8, 3.9 (via pip)
  • Note: conda is not recommended
  • pyenv installed python is not working on Mac, install python using brew install python@3.8

Hardware support

  • CPU (GPU not needed!)

Common installation issues and their solution

ERROR: fastdup-0.39-cp38-cp38-manylinux_2_31_x86_64.whl is not a supported wheel on this platform.

  • Check that you are on ubuntu 20.04 or 18.04 (via the command lsb_release -r). Alternatively on Mac M1 Big Sur or Mac Intel Mojave (use the command sw_vers)
  • Check that you are using the right python version (python3.8 and not python)
  • Make sure pip is up to date using python3.8 -m pip install -U pip).
  • Make sure you install using python3.8 -m pip install.. and not just pip install....
  • If that does not work, please open an issue with the otuput of python3.8 -m pip debug --verbose or join our slack channel.
  • If you are on mac using conda, make sure conda is installed for the right platform intel/ arm.
  • If you are on mac using pyenv, make sure to use python3.8 and not python3.7

ERROR on Ubuntu: libGL.so.1: cannot open shared object file: No such file or directory

  • Need to install depedency: sudo apt -y install libgl1-mesa-glx

Error on Mac+conda: OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/ zsh: abort python3

  • Solution from StackOverflow
  • You should install all packages without MKL support:
conda install nomkl
conda install numpy scipy pandas tensorflow
conda remove mkl mkl-service # may fail, don't worry

Error on Mac M1: AttributeError: partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline' (most likely due to a circular import) Solution: downgrade your cv2 version to 4.5.5.64 using the command python.XX -m pip install -U opencv-python==4.5.5.64 where XX is your python version.

Error on Mac M1: library not loaded when trying to import cv2 ImportError: dlopen(/Users/mikasnoopy/homebrew/lib/python3.9/site-packages/cv2/python-3.9/cv2.cpython-39-darwin.so, 2): Library not loaded: /Users/mikasnoopy/homebrew/opt/dav1d/lib/libdav1d.5.dylib Referenced from: /Users/mikasnoopy/homebrew/opt/ffmpeg@4/lib/libavcodec.58.dylib Reason: image not found or any similar error. Solution: Downgrade your ffmpg using brew remove ffmpeg; brew install ffmpeg@4