failed to install on CentOS 7.2

[wangyugui]

Hi.

failed to install on CentOS 7.2 by the guide https://github.com/PacificBiosciences/FALCON-integrate/wiki/Installation

What is wrong?

[root@T620 FALCON-integrate]# git checkout 0.4.2
HEAD is now at 3547dfc... Fix tiny pypeflow restart bug
[root@T620 FALCON-integrate]# make init
git submodule update --init --depth=1 virtualenv
usage: git submodule [--quiet] add [-b <branch>] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]
make: *** [init] Error 1

[wangyugui]

There is no "--depth=1" option on git 1.8.3.1 of CentOS 7.2
The Error of "git submodule update --init --depth=1 virtualenv"

[pb-cdunn]

True. If you cannot update git, then manually run

git submodule init
git submodule update

[wangyugui]

with the fix of git command, the command of "make init" become OK, but the next command make virtualenv still failed.

[root@T620 FALCON-integrate]# git diff
diff --git a/makefile b/makefile
index 36218c7..6134503 100644
--- a/makefile
+++ b/makefile
@@ -15,8 +15,10 @@ install:
 # These can be helpful for setup.
 pre: init
 init:
-       git submodule update --init --depth=1 virtualenv
-       git submodule update --init
+       git submodule init virtualenv
+       git submodule update virtualenv
+       git submodule init
+       git submodule update
 virtualenv:
        mkdir -p ${FALCON_PREFIX}
        python2.7 virtualenv/virtualenv.py --no-site-packages  --always-copy ${FALCON_PREFIX}

[root@T620 FALCON-integrate]# make virtualenv
mkdir -p /bio/git/pacbio/FALCON-integrate/fc_env
python2.7 virtualenv/virtualenv.py --no-site-packages  --always-copy /bio/git/pacbio/FALCON-integrate/fc_env
New python executable in /bio/git/pacbio/FALCON-integrate/fc_env/bin/python2.7
Not overwriting existing python script /bio/git/pacbio/FALCON-integrate/fc_env/bin/python (you must use /bio/git/pacbio/FALCON-integrate/fc_env/bin/python2.7)
Installing setuptools, pip, wheel...
  Complete output from command /bio/git/pacbio/FALC...fc_env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/bio/git/pacbio/FALCON-integrate/virtualenv/virtualenv_support/pip-7.0.3-py2.py3-none-any.whl/pip/__init__.py", line 4, in <module>
  File "/usr/lib64/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "virtualenv/virtualenv.py", line 2363, in <module>
    main()
  File "virtualenv/virtualenv.py", line 832, in main
    symlink=options.symlink)
  File "virtualenv/virtualenv.py", line 1004, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "virtualenv/virtualenv.py", line 969, in install_wheel
    'PIP_NO_INDEX': '1'
  File "virtualenv/virtualenv.py", line 910, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /bio/git/pacbio/FALC...fc_env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with error code 1
make: *** [virtualenv] Error 1

[wangyugui]

virtualenv is buggy on CentOS? or virtualenv is not stable?

[root@T620 ~]# virtualenv --no-site-packages  --always-copy fc_env
Cannot find file lib (bad symlink)
New python executable in fc_env/bin/python
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 821, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 956, in create_environment
    site_packages=site_packages, clear=clear, symlink=symlink))
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1377, in install_python
    shutil.copyfile(py_executable_base, full_pth)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: 'python'
[root@T620 ~]# which virtualenv
/usr/bin/virtualenv
[root@T620 ~]# rpm -qf /usr/bin/virtualenv
python-virtualenv-1.10.1-2.el7.noarch

[wangyugui]

python-virtualenv-12.0.7 seems have this problem too.
Can we use FALCON without python-virtualenv?

[pb-cdunn]

FALCON-integrate is for convenience. It's not necessary. You are free to install all dependencies however you like.

If I were you, I would not give up on make virtualenv so quickly. Your message says ImportError: No module named time, and you are on Centos7. Here is your error:

• Breakage in --always-copy - fails with: ImportError: No module named time pypa/virtualenv#565

You can modify your FALCON-integrate/virtual repo source code and maybe get it working.

[pb-cdunn]

First, try this:

cd FALCON-integrate/virtualenv
git fetch origin
git checkout -b develop origin/develop # This will take a minute.

Then try again. I've updated to the latest virtualenv. Maybe that will solve the problem.

[wangyugui]

'make install' of FALCON-integrate/FALCON-make seems OK on my CentOS 7.2.
I will use that firstly.

[wangyugui]

By the way, 'virtualenv --no-site-packages --no-setuptools --no-pip --no-wheel --always-copy fc_env' seems OK, but I am not sure whether setuptools/pip/wheel is need.

[pb-cdunn]

'make install' of FALCON-integrate/FALCON-make seems OK on my CentOS 7.2.

With our updated virtualenv repo? Please try our updated repo. Then, try to alter that version of virtualenv according to the pypa link I posted.

We use pip. But look: If you want to install your own way, just skip FALCON-make. There are lots of ways to install Python. If you have the dependencies fully integrated, you can use FALCON and even FALCON-examples independent of your integration system.

Another idea is this:

• https://github.com/PacificBiosciences/pitchfork

You can use that to get a working Python virtualenv, which we think will work on Centos7. Then use FALCON-integrate to install everything into that.

Will you be using an RS-II machine, or the newer Sequel machine?

[wangyugui]

I will use newer Sequel machine.

the test result(failed again):

#rpm -qa python-setuptools python-pip python-wheel
python-pip-7.1.0-1.el7.noarch
python-wheel-0.24.0-2.el7.noarch
python-setuptools-0.9.8-4.el7.noarch
#git clone https://github.com/PacificBiosciences/FALCON-integrate.git
#cd FALCON-integrate
#git submodule update --init

#cd virtualenv
#git fetch origin

# git checkout -b develop origin/develop
fatal: A branch named 'develop' already exists.
# git checkout develop
Previous HEAD position was 8e9cb4a... Merge branch 'master' into develop
Switched to branch 'develop'
# git branch
* develop

# cd ..

# make virtualenv
mkdir -p /bio/git/pacbio/FALCON-integrate/fc_env
python2.7 virtualenv/virtualenv.py --no-site-packages  --always-copy /bio/git/pacbio/FALCON-integrate/fc_env
Cannot find file lib (bad symlink)
New python executable in /bio/git/pacbio/FALCON-integrate/fc_env/bin/python2.7
Also creating executable in /bio/git/pacbio/FALCON-integrate/fc_env/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /bio/git/pacbio/FALC...fc_env/bin/python2.7 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/usr/lib64/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/usr/lib64/python2.7/io.py", line 51, in <module>
    import _io
ImportError: No module named _io
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "virtualenv/virtualenv.py", line 2329, in <module>
    main()
  File "virtualenv/virtualenv.py", line 708, in main
    symlink=options.symlink)
  File "virtualenv/virtualenv.py", line 941, in create_environment
    download=download,
  File "virtualenv/virtualenv.py", line 897, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "virtualenv/virtualenv.py", line 792, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /bio/git/pacbio/FALC...fc_env/bin/python2.7 - setuptools pip wheel failed with error code 1
make: *** [virtualenv] Error 1
#

[pb-cdunn]

Centos7 virtualenv problems will be tough to solve remotely. Either try pitchfork, or use a complete smrtlink tarball from PacBio.

[wangyugui]

Neither pitchfork nor smrtlink seems too complex.
virtualenv seems good, but it is too hard for CentOS 7.2.

With the guide of
https://github.com/PacificBiosciences/FALCON/wiki/Setup%3A-Installation-and-Environment
https://github.com/PacificBiosciences/FALCON/wiki/Setup%3A-Running
I succeeded to run fc_run.py with the E.coli samples.
1)skip the virtualenv
2)some info(param) of the guide is out of date.

[pb-cdunn]

Neither pitchfork nor smrtlink seems too complex.

Could you re-phrase that? Did pitchfork work? I am thinking of copying the virtualenv script from pitchfork, but only if it actually solves the problems.

[wangyugui]

I am building pitchfork now.

I found no document about how to build FALCON-integrate only(make pbfalcon?make falcon_kit?), so I used 'make world'. A lot of componments is building now.

It seems that the design of virtualenv / pitchfork is very like autoconfig/automake, if it is OK, it looks simple, but if there is some problem , that will be very difficult to fix it for most of people?

Most of people of BIO need a simple solution ? and Even if there is some problem, we hope that the problem is just in the componment that we used, so we can fix it because that source is not big.

it is very difficult to fix a problem of base componment such as virtualenv / python because that source is very big.

[wangyugui]

the build failed.

Collecting pycrypto==2.6.1
  Downloading pycrypto-2.6.1.tar.gz (446kB)
    100% |████████████████████████████████| 450kB 536kB/s
Building wheels for collected packages: pycrypto
  Running setup.py bdist_wheel for pycrypto ... error
  Complete output from command /bio/git/pacbio/pitchfork/deployment/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0r0yGW/pycrypto/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpuVETR4pip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/Crypto
  copying lib/Crypto/__init__.py -> build/lib.linux-x86_64-2.7/Crypto
  copying lib/Crypto/pct_warnings.py -> build/lib.linux-x86_64-2.7/Crypto
  creating build/lib.linux-x86_64-2.7/Crypto/Hash
  copying lib/Crypto/Hash/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Hash
  ...
  creating build/lib.linux-x86_64-2.7/Crypto/Cipher
  copying lib/Crypto/Cipher/PKCS1_OAEP.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher
  ...
  copying lib/Crypto/Signature/PKCS1_PSS.py -> build/lib.linux-x86_64-2.7/Crypto/Signature
  running build_ext
  running build_configure
  configure: error: cannot find sources (src/pycrypto_compat.h) in /usr/hpc-bio/RepeatMasker or ..
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-0r0yGW/pycrypto/setup.py", line 456, in <module>
      core.setup(**kw)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/core.py", line 151, in setup
      dist.run_commands()
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/dist.py", line 953, in run_commands
      self.run_command(cmd)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/site-packages/wheel/bdist_wheel.py", line 179, in run
      self.run_command('build')
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/command/build.py", line 127, in run
      self.run_command(cmd_name)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-0r0yGW/pycrypto/setup.py", line 251, in run
      self.run_command(cmd_name)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/cmd.py", line 326, in run_command
      self.distribution.run_command(command)
    File "/bio/git/pacbio/pitchfork/deployment/lib/python2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-0r0yGW/pycrypto/setup.py", line 278, in run
      raise RuntimeError("autoconf error")
  RuntimeError: autoconf error

  ----------------------------------------
  Failed building wheel for pycrypto
  Running setup.py clean for pycrypto
Failed to build pycrypto
    ----------------------------------------
Command "/bio/git/pacbio/pitchfork/deployment/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0r0yGW/pycrypto/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-J9wI9S-record/install-record.txt --single-version-externally-managed --root /bio/git/pacbio/pitchfork/staging/pycrypto-2.6.1 --compile" failed with error code 1 in /tmp/pip-build-0r0yGW/pycrypto/
make[1]: *** [/bio/git/pacbio/pitchfork/deployment/var/pkg/pycrypto-2.6.1] Error 1
make[1]: Leaving directory `/bio/git/pacbio/pitchfork/ports/python/pycrypto'
make: *** [pycrypto] Error 2

[pb-cdunn]

make pbfalcon?make falcon_kit?

Yes, use make falcon_kit.

it is very difficult to fix a problem of base componment such as virtualenv / python because that source is very big.

True. No-one has a good solution for that. To me, Python is the problem. But we could try installing into something like ~/.python-site-packages as PYTHONNOUSERSITE instead of using virtualenv. What do you think?

the build failed.

@mhsieh, any ideas or suggestions?

[wangyugui]

A suggestion.

If we provide a way/document to install FALCON to local machine without virtualenv/grid as the default option, this will help most of user.

Of cause, FALCON will use a lot of computer resource, but it is not a big problem.

1. people without enough skill can run FALCON on a big server.
2. people with enough skill can run FALCON on grid, they will need virtualenv, but there is no need to worry about them.

[pb-cdunn]

See FALCON-make. That's a simple Makefile which shows the basic steps. You can sudo install directly into your site-packages if you want. Or use your HOME dir with --user like this:

• http://stackoverflow.com/questions/7143077/how-can-i-install-packages-in-my-home-folder-with-pip

If you use your HOME dir, then you might need to alter a couple environment variables too, depending on your system.

There is no way to solve all Python integration problems for all users. Too complicated. Putting the packages into PyPI would work, of course, but only as long as there are no non-Python packages, which is the problem with most language-specific package-managers.

I'll make sure mhsieh sees your build problem when he is back.

[mhsieh]

make world is only for the testing purpose for apparent reason, you don't have to build that many stuffs. Besides pycrypto is not required by falcon at all.

The pycrypt problem looks real since the log says that at some point pycrypto starts to look for system python headers.

I have a tender and soft spot for biologists, let me know how I can help.

[aliceb5]

Hi all,

I am also having this problem, but with a new/clean install on 26 May 2016.

I am working on a cluster running CentOS release 6.7.

I'm new to virtual environments... and editing this type of code in general, but I couldn't figure out where perhaps edit the "repo source code"

Also, should my newly downloaded version have had this error?

here is what I get from "make virtualenv" (everything ok before this):

 make virtualenv
mkdir -p /cluster/scratch/adennis/FALCON-integrate/fc_env
python2.7 virtualenv/virtualenv.py --no-site-packages  --always-copy /cluster/scratch/adennis/FALCON-integrate/fc_env
New python executable in /cluster/scratch/adennis/FALCON-integrate/fc_env/bin/python2.7
Not overwriting existing python script /cluster/scratch/adennis/FALCON-integrate/fc_env/bin/python (you must use /cluster/scratch/adennis/FALCON-integrate/fc_env/bin/python2.7)
Installing setuptools, pip, wheel...
  Complete output from command /cluster/scratch/ade...fc_env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel:
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/cluster/scratch/adennis/FALCON-integrate/virtualenv/virtualenv_support/pip-7.0.3-py2.py3-none-any.whl/pip/__init__.py", line 4, in <module>
  File "/cluster/apps/python/2.7.6/x86_64/lib64/python2.7/logging/__init__.py", line 26, in <module>
    import sys, os, time, cStringIO, traceback, warnings, weakref
ImportError: No module named time
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "virtualenv/virtualenv.py", line 2363, in <module>
    main()
  File "virtualenv/virtualenv.py", line 832, in main
    symlink=options.symlink)
  File "virtualenv/virtualenv.py", line 1004, in create_environment
    install_wheel(to_install, py_executable, search_dirs)
  File "virtualenv/virtualenv.py", line 969, in install_wheel
    'PIP_NO_INDEX': '1'
  File "virtualenv/virtualenv.py", line 910, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /cluster/scratch/ade...fc_env/bin/python2.7 -c "import sys, pip; sys...d\"] + sys.argv[1:]))" setuptools pip wheel failed with error code 1
make: *** [virtualenv] Error 1

--

I also tried your suggestions to 
cd FALCON-integrate/virtualenv
git fetch origin
git checkout -b develop origin/develop # This will take a minute.

But this gave me:FALCON-integrate/virtualenv: No such file or directory

Thanks for any suggestions!!
Alice

[pb-cdunn]

I don't know why virtualenv has problems on Centos7, but that's a known problem:

• Breakage in --always-copy - fails with: ImportError: No module named time pypa/virtualenv#565

I was going to try the virtualenv code from pitchfork. Maybe MJ is using some old version which happens to work. You are free to find some other way of installing these Python packages. Ask your sysadmin what he suggests for installing Python packages onto your particular system.

As mentioned earlier in this thread, you could also try pitchfork, which will build everything that you need. MJ has worked to ensure that it succeeds on as many platforms as possible, and he is willing to help.