New upstream version 0.11.3

# travis.yml for github.com/jts/nanopolish

dist: trusty

services: docker

sudo: false

language: generic

cache: apt

git:

    depth: 1

.nanopolish.ci.matrix-definitions:

    - &cron-only

      if: type = cron OR env(CI_CRON) = true

    - &arch

      before_install:

          - |

            sed -i "/^FROM / s/arm64/${ARCH}/" Dockerfile-arm

          - |

            docker run --rm --privileged \

                multiarch/qemu-user-static:register --reset && \

                docker build --rm -t nanopolish -f Dockerfile-arm .

      script:

          - |

            docker run --rm -t \

                -e HDF5="${HDF5:-install}" \

                -e H5_CFLAGS="${H5_CFLAGS}" \

                -e HDF5_VERSION="1.10.4" \

                -e H5_INCLUDE="${H5_INCLUDE}" \

                -e LDFLAGS="${LDFLAGS}" \

                nanopolish

matrix:

  include:

        # Set env for both nanoplish and the dependency hdf5.

        - env:

              - CC=gcc-4.8

              - CXX=g++-4.8

              - AR=gcc-ar-4.8

              - NM=gcc-nm-4.8

              - RANLIB=gcc-ranlib-4.8

        - env:

              - CC=gcc-8

              - CXX=g++-8

              - AR=gcc-ar-8

              - NM=gcc-nm-8

              - RANLIB=gcc-ranlib-8

        # aarch64 - ARM 64-bit

        - name: aarch64

          sudo: required

    - os: linux

      arch: amd64

      addons:

        apt:

          sources:

            - ubuntu-toolchain-r-test

          packages:

            - g++-4.9

      env: "CC=gcc-4.9 CXX=g++-4.9 AR=gcc-ar-4.9 NM=gcc-nm-4.9 RANLIB=gcc-ranlib-4.9"

    - os: linux

      arch: amd64

      addons:

        apt:

          sources:

            - ubuntu-toolchain-r-test

          packages:

            - g++-8

      env: "CC=gcc-8 CXX=g++-8 AR=gcc-ar-8 NM=gcc-nm-8 RANLIB=gcc-ranlib-8"

    - os: linux

      arch: arm64

      addons:

        apt:

          packages:

            - libhdf5-serial-dev

      env:

              - ARCH=arm64

          <<: *arch

          <<: *cron-only

        - name: aarch64-system-hdf5

          sudo: required

          env:

              - ARCH=arm64

        - HDF5="system"

        - H5_INCLUDE="-I/usr/include/hdf5/serial"

        - LDFLAGS="-L/usr/lib/aarch64-linux-gnu/hdf5/serial"

          <<: *arch

        # armv7l - ARM 32-bit

        - name: armv7l

          sudo: required

          env:

              - ARCH=armhf

          <<: *arch

          <<: *cron-only

        - name: armv7l-system-hdf5

          sudo: required

          env:

              - ARCH=armhf

              - HDF5="system"

              - H5_INCLUDE="-I/usr/include/hdf5/serial"

              - LDFLAGS="-L/usr/lib/arm-linux-gnueabihf/hdf5/serial"

          <<: *arch

    allow_failures:

        # The jobs installing hdf5 from source in docker finishes with error

        # because of the job exceeded the maximum time limit (50 minutes).

        - name: aarch64

        - name: armv7l

# Install and export newer gcc

before_install:

    - sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test

    - sudo apt-get update -qq

    - |

      if [[ "${CC}" =~ ^gcc- ]]; then

          echo "Installing ${CC}."

          sudo apt-get install -qq "${CC}"

      fi

    - |

      if [[ "${CXX}" =~ ^g\+\+- ]]; then

          echo "Installing ${CXX}."

          sudo apt-get install -qq "${CXX}"

      fi

before_script:

    # Suppress all compiler warnings for hdf5 Makefile

**Details**:

* Sample :	Human cell line (NA12878)

* Basecaller : Albacore v2.0.2

* Basecaller : Guppy v2.3.5

* Region: chr20:5,000,000-10,000,000

In the extracted example data you should find the following files:

The reads were basecalled using this albacore command: ::

    read_fast5_basecaller.py -c r94_450bps_linear.cfg -t 8 -i fast5_files -s basecalled/ -o fastq

    guppy_basecaller -c dna_r9.4.1_450bps_flipflop.cfg -i fast5_files/ -s basecalled/

After the basecaller finished, we merged all of the fastq files together into a single file: ::

    cat basecalled/workspace/pass/*.fastq > albacore_output.fastq

    cat basecalled/*.fastq > output.fastq

Data preprocessing

------------------------------------

nanopolish needs access to the signal-level data measured by the nanopore sequencer. To begin, we need to create an index file that links read ids with their signal-level data in the FAST5 files: ::

    nanopolish index -d fast5_files/ albacore_output.fastq

    nanopolish index -d fast5_files/ output.fastq

We get the following files: ``albacore_output.fastq.index``, ``albacore_output.fastq.index.fai``, ``albacore_output.fastq.index.gzi``, and ``albacore_output.fastq.index.readdb``.

We get the following files: ``output.fastq.index``, ``output.fastq.index.fai``, ``output.fastq.fastq.index.gzi``, and ``output.fastq.index.readdb``.

Aligning reads to the reference genome

--------------------------------------

Next, we need to align the basecalled reads to the reference genome. We use minimap2 as it is fast enough to map reads to the human genome. In this example we'll pipe the output directly into ``samtools sort`` to get a sorted bam file: ::

    minimap2 -a -x map-ont reference.fasta albacore_output.fastq | samtools sort -T tmp -o albacore_output.sorted.bam

    samtools index albacore_output.sorted.bam

    minimap2 -a -x map-ont reference.fasta output.fastq | samtools sort -T tmp -o output.sorted.bam

    samtools index output.sorted.bam

Calling methylation

-------------------

Now we're ready to use nanopolish to detect methylated bases (in this case 5-methylcytosine in a CpG context). The command is fairly straightforward - we have to tell it what reads to use (``albacore_output.fastq``), where the alignments are (``albacore_output.sorted.bam``), the reference genome (``reference.fasta``) and what region of the genome we're interested in (``chr20:5,000,000-10,000,000``)::

Now we're ready to use nanopolish to detect methylated bases (in this case 5-methylcytosine in a CpG context). The command is fairly straightforward - we have to tell it what reads to use (``output.fastq``), where the alignments are (``output.sorted.bam``), the reference genome (``reference.fasta``) and what region of the genome we're interested in (``chr20:5,000,000-10,000,000``)::

    nanopolish call-methylation -t 8 -r albacore_output.fastq -b albacore_output.sorted.bam -g reference.fasta -w "chr20:5,000,000-10,000,000" > methylation_calls.tsv

    nanopolish call-methylation -t 8 -r output.fastq -b output.sorted.bam -g reference.fasta -w "chr20:5,000,000-10,000,000" > methylation_calls.tsv

The output file contains a lot of information including the position of the CG dinucleotide on the reference genome, the ID of the read that was used to make the call, and the log-likelihood ratio calculated by our model: ::

    mkdir data && mkdir data/fastqs

    wget ftp://ftp.sra.ebi.ac.uk/vol1/run/ERR276/ERR2764784/30xpolyA.tar.gz -O 30xpolyA.tar.gz && mv 30xpolyA.tar.gz data/

    tar -xzf data/30xpolyA.tar.gz -C data/

    read_fast5_basecaller.py --worker_threads=8 -f FLO-MIN107 -k SQK-RNA001 -q 0 -s data/fastqs -i data/30xpolyA/fast5/pass

    cat data/fastqs/workspace/pass/*.fastq > data/30xpolyA.fastq

    guppy_basecaller -c rna_r9.4.1_70bps_hac.cfg -i data/30xpolyA/fast5/pass -s data/fastqs

    cat data/fastqs/*.fastq > data/30xpolyA.fastq

In the above, change the value of the `-f` and `-k` arguments based on your flow-cell and sequencing kit, as the basecaller's accuracy is highly dependent upon these settings.

Now we'll align our basecalled mRNA sequences in the fastq file with our reference. First download a reference fasta: ::

    wget https://raw.githubusercontent.com/paultsw/polya_analysis/master/data/references/enolase_reference.fas

    wget https://raw.githubusercontent.com/paultsw/polya_analysis/master/data/references/enolase_reference.fai

    wget https://raw.githubusercontent.com/paultsw/polya_analysis/master/data/references/enolase_reference.fas.fai

    mv enolase_reference.* data/

Note that our directory structure should now look like this: ::

#! /usr/bin/env python3

import sys