New upstream version 2.2.2+dfsg

# swarm #

# swarm

A robust and fast clustering method for amplicon-based studies.

starting from raw fastq files, clustering with **swarm** and producing

a filtered OTU table.

**swarm** 2.0 introduces several novelties and improvements over

  swarm 1.0:

swarm 2.0 introduces several novelties and improvements over swarm

1.0:

* built-in breaking phase now performed automatically,

* possibility to output OTU representatives in fasta format (option

  `-w`),

* a new option called *fastidious* that refines *d* = 1 results and

  reduces the number of small OTUs,

Table of Contents

=================

* [Common misconceptions](#common_misconceptions)

* [Quick start](#quick_start)

* [Install](#install)

* [Prepare amplicon fasta files](#prepare_amplicon)

  * [Linearization](#linearization)

  * [Dereplication](#dereplication)

  * [Launch swarm](#launch)

* [Frequently asked questions](#FAQ)

  * [Refine swarm OTUs](#refine_OTUs)

  * [Count the number of amplicons per OTU](#OTU_sizes)

  * [Get the seed sequence for each swarm](#extract_seeds)

  * [Get fasta sequences for all amplicons in a swarm](#extract_all)

* [Troubleshooting](#troubleshooting)

* [Citation](#citation)

* [Contact](#contact)

* [Third-party pipelines](#pipelines)

* [Alternatives](#alternatives)

* [Version history](#history)

<a name="common_misconceptions"/>

## Common misconceptions ##

## Common misconceptions

**swarm** is a single-linkage clustering method, with some superficial

  similarities with other clustering methods (e.g.,

  that molecular markers have limitations too.

<a name="quick_start"/>

## Quick start ##

## Quick start

**swarm** most simple usage is:

cap swarm's memory consumption.

<a name="install"/>

## Install ##

Get the latest binaries for GNU/Linux or MacOSX from

Get the latest binaries for GNU/Linux or macOS from

[the release page](https://github.com/torognes/swarm/releases "swarm

tagged releases"). Get the source code from

[GitHub](https://github.com/torognes/swarm "swarm public repository")

swarm`.

<a name="prepare_amplicon"/>

## Prepare amplicon fasta files ##

To facilitate the use of **swarm**, we provide examples of shell

converted to fasta.

<a name="linearization"/>

### Linearization ###

**swarm** accepts wrapped fasta files as well as linear fasta

```

<a name="dereplication"/>

### Dereplication (mandatory) ###

In a sample, or collection of sample, a given sequence is likely to

**swarm**'s `--append-abundance` (`-a`) option to be used when

abundance information is missing from a sequence.

<a name="launch"/>

### Launch swarm ###

Here is a typical way to use **swarm**:

for details on swarm's options and parameters.

<a name="FAQ"/>

## Frequently asked questions ##

To facilitate the use of **swarm**, we provide examples of options or

(linearization and dereplication).

<a name="refine_OTUs"/>

### Refine swarm OTUs ###

The chain-breaking, which used to be performed in a second step in

![](https://github.com/frederic-mahe/swarm/blob/master/figures/swarm_2.0_fastidious_reduced.png)

<a name="OTU_sizes"/>

### Count the number of amplicons per OTU ###

You might want to check the size distribution of OTUs (number of

complete description.

<a name="extract_seeds"/>

### Get the seed sequence for each OTU ###

It is frequent for subsequent analyses to keep only one representative

filename` option.

<a name="extract_all"/>

### Get fasta sequences for all amplicons in a OTU ###

For each OTU, get the fasta sequences for all amplicons. Warning, this

```

<a name="troubleshooting"/>

## Troubleshooting ##

If **swarm** exits with an error message saying `This program

released since 2004.

<a name="citation"/>

## Citation ##

To cite **swarm**, please refer to:

PeerJ 3:e1420 doi: [10.7717/peerj.1420](http://dx.doi.org/10.7717/peerj.1420)

<a name="contact"/>

## Contact ##

You are welcome to:

* compose a friendly e-mail to: Frédéric Mahé <mahe@rhrk.uni-kl.de> and Torbjørn Rognes <torognes@ifi.uio.no>

<a name="pipelines"/>

## Third-party pipelines ##

**swarm** is available in third-party pipelines:

  performing microbiome analysis from raw DNA sequencing data.

<a name="alternatives"/>

## Alternatives ##

If you want to try alternative free and open-source clustering

methods, here are some links:

* [Vsearch](https://github.com/torognes/vsearch)

* [VSEARCH](https://github.com/torognes/vsearch)

* [Oligotyping](http://merenlab.org/projects/oligotyping/)

* [DNAclust](http://dnaclust.sourceforge.net/)

* [Sumaclust](http://metabarcoding.org/sumatra)

* [Crunchclust](https://code.google.com/p/crunchclust/)

<a name="history"/>

## Version history##

<a name="version2112"/>

### version 2.2.2 ###

**swarm** 2.2.2 fixes a bug causing Swarm to wait forever in very rare

cases when multiple threads were used.

### version 2.2.1 ###

**swarm** 2.2.1 fixes a memory allocation bug for d=1.

### version 2.2.0 ###

**swarm** 2.2.0 fixes several problems and improves usability.

Corrected output to structure and uclust files when using fastidious

mode. Corrected abundance output in some cases. Added check for

duplicated sequences and fixed check for duplicated sequence

IDs. Checks for empty sequences. Sorts sequences by additional fields

to improve stability. Improves compatibility with compilers and

operating systems.  Outputs sequences in upper case. Allows 64-bit

abundances. Shows message when waiting for input from stdin. Improves

error messages and warnings. Improves checking of command line

options. Fixes remaining errors reported by test suite. Updates

documentation.

### version 2.1.13 ###

**swarm** 2.1.13 removes a bug in the progress bar when writing seeds.

### version 2.1.12 ###

**swarm** 2.1.12 removes a debugging message.

<a name="version2111"/>

### version 2.1.11 ###

**swarm** 2.1.11 fixes two bugs related to the SIMD implementation

of alignment that might result in incorrect alignments and scores.

The bug only applies when d>1.

<a name="version2110"/>

### version 2.1.10 ###

**swarm** 2.1.10 fixes two bugs related to gap penalties of

default gap extension penalty used have actually been 4.5 instead of

4.

<a name="version219"/>

### version 2.1.9 ###

**swarm** 2.1.9 fixes a problem when compiling with GCC version 6.

<a name="version218"/>

### version 2.1.8 ###

**swarm** 2.1.8 fixes a rare bug triggered when clustering extremely

short undereplicated sequences. Also, alignment parameters are not

shown when d=1.

<a name="version217"/>

### version 2.1.7 ###

**swarm** 2.1.7 fixes more problems with seed output. Ignore CR

  characters in FASTA files. Improved help and error messsages.

<a name="version216"/>

### version 2.1.6 ###

**swarm** 2.1.6 fixes problems with older compilers that do not have

the x86intrin.h header file. It also fixes a bug in the output of seeds

with the `-w` option when d>1.

<a name="version215"/>

### version 2.1.5 ###

**swarm** 2.1.5 fixes minor bugs.

<a name="version214"/>

### version 2.1.4 ###

**swarm** 2.1.4 fixes minor bugs in the swarm algorithm used for d=1.

<a name="version213"/>

### version 2.1.3 ###

**swarm** 2.1.3 adds checks of numeric option arguments.

<a name="version212"/>

### version 2.1.2 ###

**swarm** 2.1.2 adds the -a (--append-abundance) option to set a

abundance information will result in a fatal error. The error message

in that case is improved.

<a name="version211"/>

### version 2.1.1 ###

**swarm** 2.1.1 fixes a bug with the fastidious option that caused it

to ignore some connections between heavy and light swarms.

<a name="version210"/>

### version 2.1.0 ###

**swarm** 2.1.0 marks the first official release of swarm 2.

<a name="version207"/>

### version 2.0.7 ###

**swarm** 2.0.7 writes abundance information in usearch style when using

options `-w` (`--seeds`) in combination with `-z` (`--usearch-abundance`).

<a name="version206"/>

### version 2.0.6 ###

**swarm** 2.0.6 fixes a minor bug.

<a name="version205"/>

### version 2.0.5 ###

**swarm** 2.0.5 improves the implementation of the fastidious option

abundances inside each OTU). This version also enables dereplication

when `d = 0`.

<a name="version204"/>

### version 2.0.4 ###

**swarm** 2.0.4 includes a fully parallelized fastidious option.

<a name="version203"/>

### version 2.0.3 ###

**swarm** 2.0.3 includes a working fastidious option.

<a name="version202"/>

### version 2.0.2 ###

**swarm** 2.0.2 fixes SSSE3 problems.

<a name="version201"/>

### version 2.0.1 ###

**swarm** 2.0.1 is a development release that partially implements the 

fastidious option.

<a name="version200"/>

### version 2.0.0 ###

**swarm** 2.0.0 simplifies the usage of swarm by using the fast

algorithm and the built-in OTU breaking by default. Some options are

changed and some new output options are introduced.

<a name="version1221"/>

### version 1.2.21 ###

**swarm** 1.2.21 is supposed to fix some problems related to the use of the

SSSE3 CPU instructions which are not always available.

<a name="version1220"/>

### version 1.2.20 ###

**swarm** 1.2.20 presents a production-ready version of the

swarm. That release also introduces new options to control swarm

output (options `-i` and `-l`).

<a name="version1219"/>

### version 1.2.19 ###

**swarm** 1.2.19 fixes a problem related to abundance information when

  the sequence identifier includes multiple underscore characters.

<a name="version1218"/>

### version 1.2.18 ###

**swarm** 1.2.18 reenables the possibility of reading sequences from

  `stdin` if no file name is specified on the command line. It also

  fixes a bug related to CPU features detection.

<a name="version1217"/>

### version 1.2.17 ###

**swarm** 1.2.17 fixes a memory allocation bug introduced in

  version 1.2.15.

<a name="version1216"/>

### version 1.2.16 ###

**swarm** 1.2.16 fixes a bug in the abundance sort introduced in

  version 1.2.15.

<a name="version1215"/>

### version 1.2.15 ###

**swarm** 1.2.15 sorts the input sequences in order of decreasing

  the alternative algorithm for *d* = 1 it also sorts all subseeds in

  order of decreasing abundance.

<a name="version1214"/>

### version 1.2.14 ###

**swarm** 1.2.14 fixes a bug in the output with the swarm breaker

  option (`-b`) when using the alternative algorithm (`-a`).

<a name="version1213"/>

### version 1.2.13 ###

**swarm** 1.2.13 updates the citation.

<a name="version1212"/>

### version 1.2.12 ###

**swarm** 1.2.12 improves speed of new search strategy for *d* = 1.

<a name="version1211"/>

### version 1.2.11 ###

**swarm** 1.2.11 corrects the number of differences reported in the

  break swarms output.

<a name="version1210"/>

### version 1.2.10 ###

**swarm** 1.2.10 allows amplicon abundances to be specified using the

  `-z` option is chosen. Also fixes the bad URL shown in the previous

  version of swarm.

<a name="version129"/>

### version 1.2.9 ###

**swarm** 1.2.9 includes a parallelized variant of the new search

  differences between the seed and the amplicon is indicated in the

  last column.

<a name="version128"/>

### version 1.2.8 ###

**swarm** 1.2.8 fixes an error with the gap extension

  sequences or more. The new strategy can be turned on with the `-a`

  option.

<a name="version127"/>

### version 1.2.7 ###

**swarm** 1.2.7 incorporates a few small changes and improvements to

  make it ready for integration into QIIME.

<a name="version126"/>

### version 1.2.6 ###

**swarm** 1.2.6 add an option (`-r` or `--mothur`) to format the

  input sequences it will now report the illegal character and the

  line number.

<a name="version125"/>

### version 1.2.5 ###

**swarm** 1.2.5 can be run on CPUs without the POPCNT feature. It

  the appropriate code.  The code that avoids POPCNT is just slightly

  slower. Only basic SSE2 is now required.

<a name="version124"/>

### version 1.2.4 ###

**swarm** 1.2.4 changes the name of the new option from

  options, and also adds a companion script `swarm_breaker.py` to

  refine swarm results (`scripts` folder).

<a name="version123"/>

### version 1.2.3 ###

**swarm** 1.2.3 adds an option (`-b` or `--break_swarms`) to output

  the inline assembly code is also changed for compatibility with more

  compilers.

<a name="version122"/>

### version 1.2.2 ###

**swarm** 1.2.2 fixes an issue with incorrect values in the statistics

  file (maximum generation and radius of swarms). This version is also

  a bit faster.

<a name="version121"/>

### version 1.2.1 ###

**swarm** 1.2.1 removes the need for a SSE4.1 capable CPU and should

  now be able to run on most servers, desktops and laptops.

<a name="version120"/>

### version 1.2.0 ###

**swarm** 1.2.0 introduces a pre-filtering of similar amplicons based

  a computational overhead, but becomes more and more efficient as the

  size of the amplicon set increases.

<a name="version111"/>

### version 1.1.1 ###

**swarm** now works on Apple computers. This version also corrects an

  sub-optimal alignments. Slightly different alignments may result

  relative to previous version, giving slightly different swarms.

<a name="version110"/>

### version 1.1.0 ###

**swarm** 1.1.0 introduces new optimizations and is 20% faster than

.\" ============================================================================

.TH swarm 1 "January 16, 2017" "version 2.1.12" "USER COMMANDS"

.TH swarm 1 "December 12, 2017" "version 2.2.2" "USER COMMANDS"

.\" ============================================================================

.SH NAME

swarm \(em find clusters of nearly-identical nucleotide amplicons

SIMD Extensions (SSE2) of modern x86-64 CPUs. If SSE2 instructions are

not available, \fBswarm\fR exits with an error message.

.PP

\fBswarm\fR reads the named input \fIfilename\fR, a fasta file of

nucleotide amplicons. The amplicon identifier is defined as the string

comprised between the ">" symbol and the first space or the end of the

\fBswarm\fR can read nucleotide amplicons in fasta format from a

normal file or from the standard input (using a pipe or a

redirection). The amplicon identifier is defined as the string

comprised between the '>' symbol and the first space or the end of the

line, whichever comes first. As \fBswarm\fR outputs lists of amplicon

identifiers, amplicon identifiers must be unique to avoid ambiguity;

swarm exits with an error message if identifiers are not

unique. Amplicon identifiers must end with a "_" followed by a

positive integer representing the amplicon copy number (or abundance

annotation; usearch/vsearch users can use the option \-z to change

that behavior). Abundance annotations play a crucial role in the

clustering process, and swarm exits with an error message if that

swarm exits with an error message if identifiers are not unique.

Amplicon identifiers must end with a '_' followed by a positive

integer representing the amplicon copy number (or abundance

annotation; usearch/vsearch users can use the option \-z to use

the ';size=' annotation). Abundance annotations play a crucial role in

the clustering process, and swarm exits with an error message if that

information is not available. The amplicon sequence is defined as a

string of [acgt] or [acgu] symbols (case insensitive), starting after

the end of the identifier line and ending before the next identifier

line or the file end; \fBswarm\fR exits with an error message if any

other symbol is present.

string of [ACGT] or [ACGU] symbols (case insensitive, 'U' is replaced

with 'T' internally), starting after the end of the identifier line

and ending before the next identifier line or the file end;

\fBswarm\fR silently removes newline symbols ('\\n' or '\\r') and

exits with an error message if any other symbol is present.

.\" ----------------------------------------------------------------------------

.SS General options

.TP 9

.TP

.B \-\-

delimit the option list. Later arguments, if any, are treated as

operands even if they begin with "\-". For example, "swarm \-\-

\-file.fasta" reads from the file "\-file.fasta".

operands even if they begin with '\-'. For example, 'swarm \-\-

\-file.fasta' reads from the file '\-file.fasta'.

.LP

.\" ----------------------------------------------------------------------------

.SS Clustering options

less) differences. This is \fBswarm\fR's most important parameter. The

number of differences is calculated as the number of mismatches

(substitutions, insertions or deletions) between the two amplicons

once the optimal pairwise global alignment has been found (see

"pairwise alignment advanced options" to influence that step). Any

\fIinteger\fR between 0 and 256 can be used, but high \fId\fR values

once the optimal pairwise global alignment has been found

(see 'pairwise alignment advanced options' to influence that step).

Any \fIinteger\fR from 0 to 255 can be used, but high \fId\fR values

will decrease the taxonomical resolution of \fBswarm\fR

results. Commonly used \fId\fR values are 1, 2 or 3, rarely

higher. When using \fId\fR = 0, \fBswarm\fR will output results

when using the option \-\-fastidious (\-f), define \fBswarm\fR's

maximum memory footprint (in megabytes). \fBswarm\fR will adjust the

\-\-bloom\-bits (\-y) value of the Bloom filter to fit within the

specified amount of memory.

specified amount of memory. The value must be at least 3.

.TP

.B \-f\fP,\fB\ \-\-fastidious

when working with \fId\fR = 1, perform a second clustering pass to

(\-l), \-\-output\-file (\-o), \-\-mothur (\-r), \-\-uclust\-file, and

\-\-seeds (\-w) are updated to reflect these modifications; the file

\-\-statistics\-file (\-s) is partially updated (columns 6 and 7 are

not updated); the output file \-\-internal\-structure (\-i) is not

updated.

not updated); the output file \-\-internal\-structure (\-i) is

partially updated (column 5 is not updated for amplicons that belonged to

the small OTU).

.TP

.BI \-y\fP,\fB\ \-\-bloom\-bits\~ "positive integer"

when using the option \-\-fastidious (\-f), define the size (in bits)

.TP 9

.BI \-a\fP,\fB\ \-\-append\-abundance\~ "positive integer"

set abundance value to use when some or all amplicons in the input

file lack abundance values. Warning, it is not recommended to use

\fBswarm\fR on datasets where abundance values are all identical. We

provide that option as a courtesy to advanced users, please use it

file lack abundance values (_\fIinteger\fR, or ;size=\fIinteger\fR;

when using \-z). Warning, it is not recommended to use \fBswarm\fR on

datasets where abundance values are all identical. We provide that

option as a courtesy to advanced users, please use it

carefully. \fBswarm\fR exits with an error message if abundance values

are missing and if this option is not used.

.TP

of delineation, starting from 1. All pairs of amplicons belonging to

the same OTU will receive the same number.

.IP \n+[step].

number of steps from the OTU seed to amplicon B (\fIpositive

integer\fR).

cummulated number of steps from the OTU seed to amplicon B

(\fIpositive integer\fR). When using the option \-\-fastidious (\-f),

the actual number of steps between grafted amplicons and the OTU seed

cannot be re-computed efficiently and is always set to 2 for the

amplicon pair linking the small OTU to the big OTU. Cummulated number

of steps in the small OTU (if any) are left unchanged.

.RE

.RE

.TP

number of amplicons with an abundance of 1 in the OTU,

.IP \n+[step].

maximum number of iterations before the OTU reached its natural

limit),

limit,

.IP \n+[step].

theoretical maximum radius of the OTU (i.e., number of cummulated

differences between the seed and the furthermost amplicon in the

OTU). The actual maximum radius of the OTU is often much smaller.

cummulated number of steps along the path joining the seed and the

furthermost amplicon in the OTU. Please note that the actual number of

differences between the seed and the furthermost amplicon is usually

much smaller. When using the option \-\-fastidious (\-f), grafted

amplicons are not taken into account.

.RE

.RE

.TP

.BI \-u\fP,\fB\ \-\-uclust\-file \0filename

output clustering results in uclust-like file format to the specified

file. That option does not modify \fBswarm\fR's default output format.

output clustering results in \fIfilename\fR using a tab-separated

uclust-like format with 10 columns and 3 different type of entries (S,

H or C). That option does not modify \fBswarm\fR's default output

format. Each fasta sequence in the input file can be either a cluster

centroid (S) or a hit (H) assigned to a cluster. Cluster records (C)

summarize information (size, centroid label) for each cluster. Column

content varies with the type of entry (S, H or C):

.RS

.RS

.nr step 1 1

.IP \n[step]. 4

Record type: S, H, or C.

.IP \n+[step].

Cluster number (zero-based).

.IP \n+[step].

Centroid length (S), query length (H), or cluster size (C).

.IP \n+[step].

Percentage of similarity with the centroid sequence (H), or set to '*'

(S, C).

.IP \n+[step].

Match orientation + or - (H), or set to '*' (S, C).

.IP \n+[step].

Not used, always set to '*' (S, C) or to zero (H).

.IP \n+[step].

Not used, always set to '*' (S, C) or to zero (H).

.IP \n+[step].

set to '*' (S, C) or, for H, compact representation of the pairwise

alignment using the CIGAR format (Compact Idiosyncratic Gapped

Alignment Report): M (match), D (deletion) and I (insertion). The

equal sign '=' indicates that the query is identical to the centroid

sequence.

.IP \n+[step].

Label of the query sequence (H), or of the centroid sequence (S, C).

.IP \n+[step].

Label of the centroid sequence (H), or set to '*' (S, C).

.RE

.RE

.TP

.BI \-w\fP,\fB\ \-\-seeds \0filename

output OTU representatives to \fIfilename\fR in fasta format. The

or minor bug releases are not mentioned):

.RS

.TP

.BR v2.2.2\~ "released December 12, 2017"

Version 2.2.2 fixes a bug that would cause Swarm to wait forever in

very rare cases when multiple threads were used.

.TP

.BR v2.2.1\~ "released October 27, 2017"

Version 2.2.1 fixes a memory allocation bug for \fId\fR = 1 and

duplicated sequences.

.TP

.BR v2.2.0\~ "released October 17, 2017"

Version 2.2.0 fixes several problems and improves usability. Corrected

output to structure and uclust files when using fastidious

mode. Corrected abundance output in some cases. Added check for

duplicated sequences and fixed check for duplicated sequence

IDs. Checks for empty sequences. Sorts sequences by additional fields

to improve stability. Improves compatibility with compilers and

operating systems.  Outputs sequences in upper case. Allows 64-bit

abundances. Shows message when waiting for input from stdin. Improves

error messages and warnings. Improves checking of command line

options. Fixes remaining errors reported by test suite. Updates

documentation.

.TP

.BR v2.1.13\~ "released March 8, 2017"

Version 2.1.13 removes a bug with the progress bar when writing seeds.

.TP

.BR v2.1.12\~ "released January 16, 2017"

Version 2.1.12 removes a debugging message.

.TP

.BR v2.1.11\~ "released January 16, 2017"

Version 2.1.11 fixes two bugs related to the SIMD implementation

of alignment that might result in incorrect alignments and scores.

The bug only applies when d>1.

The bug only applies when \fId\fR > 1.

.TP

.BR v2.1.10\~ "released December 22, 2016"

Version 2.1.10 fixes two bugs related to gap penalties of alignments.

.BR v2.1.7\~ "released February 24, 2016"

Version 2.1.7 fixes a bug in the output of seeds with the \-w option

when \fId\fR > 1 that was not properly fixed in version 2.1.6. It also

handles ascii character n°13 (CR) in FASTA files better. Swarm will

handles ascii character #13 (CR) in FASTA files better. Swarm will

now exit with status 0 if the \-h or the \-v option is specified. The

help text and some error messages have been improved.

.TP

.TP

.BR v1.2.10\~ "released August 8, 2014"

Version 1.2.10 allows amplicon abundances to be specified using the

usearch style in the sequence header (e.g. ">id;size=1") when the \-z

usearch style in the sequence header (e.g. '>id;size=1') when the \-z

option is chosen.

.TP

.BR v1.2.8\~ "released August 5, 2014"

#COMMON=-pg -g

COMMON=-g

COMPILEOPT=-Wall -Wsign-compare -O3 -msse2 -mtune=core2 -Icityhash

LIBS=-lpthread

LINKFLAGS=$(COMMON)

CXX=g++

CXXFLAGS=$(COMPILEOPT) $(COMMON)

WARNINGS=-Wall -Wsign-compare -Wextra -Wpedantic -Wno-long-long

CXXFLAGS=$(COMMON) $(WARNINGS) -O3 -msse2 -mtune=core2 -Icityhash

PROG=swarm

DEPS=Makefile swarm.h bitmap.h bloom.h cityhash/config.h cityhash/city.h \

	threads.h

.SUFFIXES:.o .cc

%.o : %.cc $(DEPS)

	$(CXX) $(CXXFLAGS) -c -o $@ $<

all : $(PROG)

swarm : $(OBJS)

	rm -rf swarm *.o *~ ../bin/ gmon.out cityhash/*.o ../man/*~ ../*~

ssse3.o : ssse3.cc $(DEPS)

	$(CXX) -mssse3 $(CXXFLAGS) -c -o $@ $<

	$(CXX) $(CXXFLAGS) -mssse3 -c -o $@ $<