New upstream version 2.4

Changes

=======

v2.4 (2019-07-09)

-----------------

* :issue:`292`: Implement support for demultiplexing paired-end reads that use

  :ref:`combinatorial indexing (“combinatorial demultiplexing”)

  <combinatorial-demultiplexing>`.

* :pr:`384`: Speed up reading compressed files by requiring an xopen version

  that uses an external pigz process even for *reading* compressed input files

  (not only for writing).

* :issue:`381`: Fix ``--report=minimal`` not working.

* :issue:`380`: Add a ``--fasta`` option for forcing that FASTA is written

  to standard output even when input is FASTQ. Previously, forcing

  FASTA was only possible by providing an output file name.

v2.3 (2019-04-25)

-----------------

Metadata-Version: 2.1

Name: cutadapt

Version: 2.3

Version: 2.4

Summary: trim adapters from high-throughput sequencing reads

Home-page: https://cutadapt.readthedocs.io/

Author: Marcel Martin

release = version

issues_uri = 'https://github.com/marcelm/cutadapt/issues/{issue}'

issues_pr_uri = 'https://github.com/marcelm/cutadapt/pull/{pr}'

suppress_warnings = ['image.nonlocal_uri']

# The language for content autogenerated by Sphinx. Refer to documentation

    venv/bin/tox

Development installation (without virtualenv)

---------------------------------------------

Alternatively, if you do not want to use virtualenv, running the following may

work from within the cloned repository::

    python3 setup.py build_ext -i

    pytest

This requires Cython and pytest to be installed. Avoid this method and use a

virtualenv instead if you can.

Code style

----------

Cutadapt tries to follow PEP8, with some exceptions:

* Indentation is made with tabs, not with spaces

* The maximum line length for code 100 characters, not 80, but try to wrap

  comments at 80 characters for readability.

Yes, there are inconsistencies in the current code base since it’s a few years old already.

Making a release

----------------

Since version 1.17, Travis CI is used to automatically deploy a new Cutadapt release

(both as an sdist and as wheels) whenever a new tag is pushed to the Git repository.

Cutadapt uses `versioneer <https://github.com/warner/python-versioneer>`_ to automatically manage

Cutadapt uses `setuptools_scm <https://github.com/pypa/setuptools_scm>`_ to automatically manage

version numbers. This means that the version is not stored in the source code but derived from

the most recent Git tag. The following procedure can be used to bump the version and make a new

release.

#. Wait for Travis to finish and to deploy to PyPI.

#. Update the `bioconda recipe <https://github.com/bioconda/bioconda-recipes/blob/master/recipes/cutadapt/meta.yaml>`_.

   It is probly easiest to edit the recipe via the web interface and send in a

   pull request. Ensure that the list of dependencies (the ``requirements:``

   section in the recipe) is in sync with the ``setup.py`` file.

   Since this is just a version bump, the pull request does not need a

   review by other bioconda developers. As soon as the tests pass and if you

   have the proper permissions, it can be merged directly.

Releases to bioconda still need to be made manually.

Making a release manually

-------------------------

.. note:

    This section is outdated, see the previous section!

If this is the first time you attempt to upload a distribution to PyPI, create a

configuration file named ``.pypirc`` in your home directory with the following

contents::

    [distutils]

    index-servers =

        pypi

    [pypi]

    username=my-user-name

    password=my-password

See also `this blog post about getting started with

PyPI <http://peterdowns.com/posts/first-time-with-pypi.html>`_. In particular,

note that a ``%`` in your password needs to be doubled and that the password

must *not* be put between quotation marks even if it contains spaces.

Cutadapt uses `versioneer <https://github.com/warner/python-versioneer>`_ to automatically manage

version numbers. This means that the version is not stored in the source code but derived from

the most recent Git tag. The following procedure can be used to bump the version and make a new

release.

#. Update ``CHANGES.rst`` (version number and list of changes)

#. Ensure you have no uncommitted changes in the working copy.

#. Run a ``git pull``.

#. Run ``tox``, ensuring all tests pass.

#. Tag the current commit with the version number (there must be a ``v`` prefix)::

       git tag v0.1

#. Create a distribution (``.tar.gz`` file). Double-check that the auto-generated version number in

   the tarball is as you expect it by looking at the name of the generated file in ``dist/``::

       python3 setup.py sdist

#. If necessary, pip install ``twine`` and then upload the generated tar file to PyPI::

       twine upload dist/cutadapt-0.1.tar.gz  # adjust version number

#. Push the tag::

       git push --tags

#. The `bioconda recipe <https://github.com/bioconda/bioconda-recipes/blob/master/recipes/cutadapt/meta.yaml>`_

   also needs to be updated, but the bioconda bot will likely do this automatically

   if you just wait a little while.

#. Update the `bioconda recipe <https://github.com/bioconda/bioconda-recipes/blob/master/recipes/cutadapt/meta.yaml>`_.

   It is probly easiest to edit the recipe via the web interface and send in a

   pull request. Ensure that the list of dependencies (the ``requirements:``

   Ensure that the list of dependencies (the ``requirements:``

   section in the recipe) is in sync with the ``setup.py`` file.

   Since this is just a version bump, the pull request does not need a

   review by other bioconda developers. As soon as the tests pass and if you

   have the proper permissions, it can be merged directly.

If something went wrong *after* you uploaded a tarball, fix the problem and follow the

above instructions again, but with an incremented revision in the version number.

That is, go from version x.y to x.y.1. Do not change a version that has already

If something went wrong *after* a version has already been tagged and published to

PyPI, fix the problem and tag a new version. Do not change a version that has already

been uploaded.

.. include:: ../CONTRIBUTING.rst

Input and output file formats

-----------------------------

Input files for Cutadapt need to be in one the these formats:

Input and output files need to be in FASTA or FASTQ format. Reading and writing

compressed file formats ``.gz``, ``.bz2`` or ``.xz`` is also supported. Cutadapt

uses ``pigz`` internally if possible to speed up writing and reading of

gzipped files.

* FASTA with extensions ``.fasta``, ``.fa`` or ``.fna``

* FASTQ with extensions ``.fastq`` or ``.fq``

* Any of the above, but compressed as ``.gz``, ``.bz2`` or ``.xz``

The input file format is recognized from the file name extension. If the

extension was not recognized or when Cutadapt reads from standard input,

the contents are inspected instead.

Input and output file formats are recognized from the file name extension. You

can override the input format with the ``--format`` option.

The output file format is also recognized from the file name extension. If the

extensions was not recognized or when Cutadapt writes to standard input, the

same format as the input is used for the output.

You can use the automatic format detection to convert from FASTQ to FASTA

(without doing any adapter trimming)::

You can use this to convert from FASTQ to FASTA (without doing any adapter

trimming)::

    cutadapt -o output.fasta.gz input.fastq.gz

When you want to do the same (read FASTQ, write FASTA), but want to write to

standard output, you need to use ``--fasta`` instead because there is no

output file name::

    cutadapt --fasta input.fastq.gz > out.fasta

.. _compressed-files:

Cutadapt supports compressed input and output files. Whether an input file

needs to be decompressed or an output file needs to be compressed is detected

automatically by inspecting the file name: If it ends in ``.gz``, then gzip

compression is assumed. This is why the example given above works::

automatically by inspecting the file name: For example, if it ends in ``.gz``,

then gzip compression is assumed ::

    cutadapt -a AACCGGTT -o output.fastq.gz input.fastq.gz

The supported compression formats are gzip (``.gz``), bzip2 (``.bz2``)

and xz (``.xz``).

The default compression level for gzip output is 6. Use option ``-Z`` to

change this to level 1. The files need more space, but it is faster and

therefore a good choice for short-lived intermediate files.

Standard input and output

-------------------------

.. versionadded:: 1.18

    ``--cores=0`` for autodetection

Speed-up tricks

---------------

There are several tricks for limiting wall-clock time while using cutadapt.

``-Z`` (alternatively ``--compression-level=1``) can be used to limit the

amount of CPU time which is spent on the compression of output files.

Alternatively, choosing filenames not ending with ``.gz``, ``.bz2`` or ``.xz``

will make sure no cpu time is spent on compression at all.  On systems

with slow I/O, it can actually be faster to set a higher compression-level

than 1.

Increasing the number of cores with ``-j`` will increase the number of reads per

minute at near-linear rate.

It is also possible to use pipes in order to bypass the filesystem and pipe

cutadapt's output into an aligner such as BWA. The ``mkfifo`` command allows

you to create named pipes in bash.

.. code-block::bash

    mkfifo R1.fastq R2.fastq

    cutadapt -a ${ADAPTER_R1} -A ${ADAPTER_R2} -o R1.fastq -p R2.fastq ${READ1} ${READ2} > cutadapt.report & \

    bwa mem -o output.sam ${INDEX} R1.fastq R2.fastq

This command will run cutadapt and BWA simultaneously, using cutadapts output as

BWA's input, and capturing cutadapts report in ``cutadapt.report``.

Read processing stages

======================

Linked adapters (combined 5' and 3' adapter)

--------------------------------------------

If your sequence of interest ist “framed” by a 5' and a 3' adapter, and you want

If your sequence of interest is “framed” by a 5' and a 3' adapter, and you want

to remove both adapters, then you may want to use a *linked adapter*. A linked

adapter combines a 5' and a 3' adapter. By default, the adapters are not anchored,

but in many cases, you should anchor the 5’ adapter by prefixing it with ``^``.

indices, which are only used in pairs, that is, the first i5 index is always used with the first i7

index and so on.

.. note::

    If the adapters do not come in pairs, but all combinations are possible, see

    :ref:`the section about combinatorial demultiplexing <combinatorial-demultiplexing>`.

An example::

    cutadapt --pair-adapters -a AAAAA -a GGGG -A CCCCC -a TTTT -o out.1.fastq -p out.2.fastq in.1.fastq in.2.fastq

There is one limitation of the algorithm at the moment: The program looks for the best-matching R1 adapter

first and then checks whether the corresponding R2 adapter can be found. If not, the read pair

remains unchanged. However, it is in theory possible that a different R1 adapter that does not

fit as well has a partner that *can* be found. Some read pairs may therefore remain untrimmed.

fit as well would have a partner that *can* be found. Some read pairs may therefore remain untrimmed.

.. versionadded:: 2.1

* If you want to demultiplex, but keep the barcode in the reads, use the option ``--action=none``.

.. _combinatorial-demultiplexing:

Demultiplexing paired-end reads with combinatorial dual indexes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Illumina’s combinatorial dual indexing strategy uses a set of indexed adapters on R1 and another one

on R2. Unlike

`unique dual indexes (UDI) <https://support.illumina.com/bulletins/2018/08/understanding-unique-dual-indexes--udi--and-associated-library-p.html>`_,

all combinations of indexes are possible.

For demultiplexing this type of data ("combinatorial demultiplexing"), it is necessary to write each

read pair to an output file depending on the adapters found on R1 *and* R2.

Doing this with Cutadapt is similar to doing normal demultiplexing as described above, but you need

to use ``{name1}}`` and ``{name2}`` in both output file name templates. For example::

    cutadapt \

        -e 0.15 --no-indels \

        -g file:barcodes_fwd.fasta \

        -G file:barcodes_rev.fasta \

        -o trimmed-{name1}-{name2}.1.fastq.gz -p trimmed-{name1}-{name2}.2.fastq.gz \

        input.1.fastq.gz input.2.fastq.gz

The ``{name1}`` will be replaced with the name of the best-matching R1 adapter and ``{name2}}`` will

be replaced with the name of the best-matching R2 adapter.

If there was no match of an R1 adapter, ``{name1}`` is set to "unknown", and if there is no match of

an R2 adapter, ``{name2}`` is set to "unknown". To discard read pairs for which one or both adapters

could not be found, use ``--discard-untrimmed``.

The ``--untrimmed-output`` and ``--untrimmed-paired-output`` options cannot be used.

Read the :ref:`demultiplexing <demultiplexing>` section for how to choose the error rate etc.

Also, the tips below about how to speed up demultiplexing apply even with combinatorial

demultiplexing.

.. versionadded:: 2.4

.. _speed-up-demultiplexing:

Speeding up demultiplexing