FAQ

Installation

How do I install dinopy?

The recommended ways to install dinopy are with conda (using the bioconda channel) or from PyPI using pip. You can also download the code from out bitbucket repository or PyPI and compile it yourself.

Do I need Cython to use dinopy?

Since version 2.2.0, Cython is required to install dinopy.

Prior to version 2.2.0 Cython was not a requirement as all extension modules were packaged as .c-files. For these versions, you only need to compile these extension modules, which does not require Cython. This is handled by setuptools and is automatically executed when you install dinopy, as long as you have the build-essential package installed.

During testing a lot of KeyErrors are raised.

Running

nosetests -v --nocapture

yields a lot of the following KeyErrors:

ERROR: Does the iteration over qgrams work correctly?
----------------------------------------------------------------------
Traceback (most recent call last):
    File "/vol/home/timm/repos/dna_handler/test/fasta_parser_test.py", line 295, in test_iterate_qgrams
        chr1_sequence = cnv.string_to_bytes("ACGTTGCATCTACGTTGCATCTACGTTGCATCACGTTGCATCTACGTTGCATCTACGTTGCATCACGTTGCATCTACGTTGCATCTACGTTGCATC")
    File "conversion.pyx", line 112, in dinopy.conversion.string_to_bytes (dinopy/conversion.c:3308)
    File "conversion.pyx", line 127, in dinopy.conversion.string_to_bytes (dinopy/conversion.c:3168)
KeyError: 65

This is due to an old (<0.20) Cython version.

To fix this issue, please download a newer Cython version and install it.

The version used can either be checked by importing the Cython module in the interpreter.:

>>> import Cython
>>> Cython.__version__
'0.22'

or by running:

user@machine:~$ cython --version
Cython version 0.22

on a terminal of your choice.

Building dinopy fails with: gcc: error trying to exec 'cc1plus'

The installation of dinopy fails during the compilation step with the following error.:

me@machine:~$ pip install dinopy
[...]
.linux-x86_64-3.5/dinopy/wrap_sais.o -O3
  gcc: error trying to exec 'cc1plus': execvp: No such file or directory
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for dinopy
Failed to build dinopy
Installing collected packages: dinopy
  Running setup.py install for dinopy
    Complete output from command /home/timm/anaconda3/envs/dinoenv/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-5oyzgq9i/dinopy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-p5ogilb7-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    running build_ext
    building 'dinopy.wrap_sais' extension
    gcc -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Idinopy/cpp/ -I/home/timm/anaconda3/envs/dinoenv/include/python3.5m -c dinopy/wrap_sais.cpp -o build/temp.linux-x86_64-3.5/dinopy/wrap_sais.o -O3
    gcc: error trying to exec 'cc1plus': execvp: No such file or directory
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/home/timm/anaconda3/envs/dinoenv/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-5oyzgq9i/dinopy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-p5ogilb7-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-5oyzgq9i/dinopy

Dinopy requires cc1plus, the c++ compiler, which is not pre-installed on a vanilla ubuntu. This can be solved by installing the build-essential package.:

me@machine:~$ sudo apt-get install build-essential
me@machine:~$ pip install dinopy

Collecting dinopy
  Using cached dinopy-1.2.0.tar.gz
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.9 in /home/timm/anaconda3/envs/dinoenv/lib/python3.5/site-packages (from dinopy)
Requirement already satisfied (use --upgrade to upgrade): cython>=0.22 in /home/timm/anaconda3/envs/dinoenv/lib/python3.5/site-packages (from dinopy)
Building wheels for collected packages: dinopy
  Running setup.py bdist_wheel for dinopy
  Stored in directory: /home/timm/.cache/pip/wheels/e3/7d/99/a140b9cef01ff2bf07ef28a0df1595a1b1f5cd7165afc20ae2
Successfully built dinopy
Installing collected packages: dinopy
Successfully installed dinopy-1.2.0

ImportError: No module named 'dinopy.output_opener' is raised when importing dinopy in the interpreter

After installing dinopy from the sources using me@machine:~$ python setup.py install dinopy can not be imported.

Python interpreter version of the error:

>>> import dinopy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/timm/repos/dinofoo/dinopy/__init__.py", line 15, in <module>
    from .output_opener import OutputOpener
ImportError: No module named 'dinopy.output_opener'

IPython version:

In [1]: import dinopy
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-e19a24a8118b> in <module>()
----> 1 import dinopy

/home/timm/repos/dinofoo/dinopy/__init__.py in <module>()
     13 __all__ = ['fasta_reader', 'fasta_writer', 'fastq_reader', 'fastq_writer', 'auxiliary', 'shaping', 'shape',
     14            'processors', 'definitions', 'exceptions', 'output_opener', 'nameline_parser', 'sambam', 'sam_reader', 'sam_writer']
---> 15 from .output_opener import OutputOpener
     16 from .fastq_reader import FastqReader
     17 from .fasta_reader import FastaReader

ImportError: No module named 'dinopy.output_opener'

This error occurs, when the interpreter is started in the dinopy repository. For example if you checked out to ~/repos/dinopy, installed me@machine:~/repos/dinopy$ python setup.py install and started python me@machine:~/repos/dinopy$ python.

Python will first check the local folder for imports according to sys.path. There it will find a folder named dinopy that contains an __init__.py and will happily try to use the folder as the dinopy package. The Cython extensions are not compiled locally though, so the .so are missing there, rendering output_opener an undefined import.

To solve this, leave the dinopy repository structure and retry. For projects that use dinopy as a dependency this should not be a problem.

After locally building the documentation all module pages are empty

When building the documentation, sphinx traverses the package and imports all autodoc modules (extensions) in order to get the docstrings. As the extensions are implemented in Cython the modules are compiled as .so files, which can be imported by python. After running python setup.py install the .so files are copied to the installation target. The documentation configuration file expects those extensions to be in the local dinopy package though.

To solve this run python setup.py build_ext --inplace. This compiles the .c extensions and places the .so files next to their .pyx and .c files. After this sphinx should find the modules and build the autodoc pages.

numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216 is raised when importing dinopy

If installed together with other packages, conda can install a numpy version incompatible with the one used to generate the .c files. This results in the following error message:

Python 3.6.7 | packaged by conda-forge | (default, Jul  2 2019, 02:18:42)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import dinopy as dp
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-60b1ad065020> in <module>
----> 1 import dinopy as dp

~/miniconda3/envs/dino2/lib/python3.6/site-packages/dinopy/__init__.py in <module>
     14            'processors', 'definitions', 'exceptions', 'output_opener', 'nameline_parser', 'sam_reader',  'sam_writer', 'sambam']
     15 from .output_opener import OutputOpener
---> 16 from .fastq_reader import FastqReader
     17 from .fasta_reader import FastaReader
     18 from .fastq_writer import FastqWriter

__init__.pxd in init dinopy.fastq_reader()

ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

This can be solved by updating the numpy version, for example using conda update numpy or pip install numpy --upgrade. A similar error is described here.