New upstream version 3.1.2

sudo: false

python:

- '2.7'

- '3.4'

- '3.5'

- '3.6'

- '3.7'

- '3.8'

services:

- redis

branches:

3.1.2:

  #18 Add a protocol option to set CURLOPT_FTP_FILEMETHOD

  #19 Rename protocol options to options

  Fix copy of production files instead of download when files are in subdirectories

3.1.1:

  #17 Support MDTM command in directftp

3.1.0:

  #16 Don't change name after download in DirectHTTPDownloader

  PR #7 Refactor downloaders (*WARNING* breaks API)

Web processes should be behind a proxy/load balancer, API base url /api/download

Prometheus endpoint metrics are exposed via /metrics on web server

# Download options

Since version 3.0.26, you can use the `set_options` method to pass a dictionary of downloader-specific options.

The following list shows some options and their effect (the option to set is the key and the parameter is the associated value):

  * **skip_check_uncompress**:

    * parameter: bool.

    * downloader(s): all.

    * effect: If true, don't test the archives after download.

    * default: false (i.e. test the archives).

  * **ssl_verifyhost**:

    * parameter: bool.

    * downloader(s): `CurlDownloader`, `DirectFTPDownload`, `DirectHTTPDownload`.

    * effect: If false, don't check that the name of the remote server is the same than in the SSL certificate.

    * default: true (i.e. check host name).

    * note: It's generally a bad idea to disable this verification. However some servers are badly configured. See [here](https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html) for the corresponding cURL option.

  * **ssl_verifypeer**:

    * parameter: bool.

    * downloader(s): `CurlDownloader`, `DirectFTPDownload`, `DirectHTTPDownload`.

    * effect: If false, don't check the authenticity of the peer's certificate.

    * default: true (i.e. check authenticity).

    * note: It's generally a bad idea to disable this verification. However some servers are badly configured. See [here](https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html) for the corresponding cURL option.

  * **ssl_server_cert**:

    * parameter: filename of the certificate.

    * downloader(s): `CurlDownloader`, `DirectFTPDownload`, `DirectHTTPDownload`.

    * effect: Pass a file holding one or more certificates to verify the peer with.

    * default: use OS certificates.

    * note: See [here](https://curl.haxx.se/libcurl/c/CURLOPT_CAINFO.html) for the corresponding cURL option.

  * **tcp_keepalive**:

    * parameter: int.

    * downloader(s): `CurlDownloader`, `DirectFTPDownload`, `DirectHTTPDownload`.

    * effect: Sets the interval, in seconds, that the operating system will wait between sending keepalive probes.

    * default: cURL default (60s at the time of this writing).

    * note: See [here](https://curl.haxx.se/libcurl/c/CURLOPT_TCP_KEEPINTVL.html) for the corresponding cURL option.

  * **ftp_method**:

    * parameter: one of `default`, `multicwd`, `nocwd`, `singlecwd` (case insensitive).

    * downloader(s): `CurlDownloader`, `DirectFTPDownload`, `DirectHTTPDownload`.

    * effect: Sets the method to use to reach a file on a FTP(S) server (`nocwd` and `singlecwd` are usually faster but not always supported).

    * default: `default` (which is `multicwd` at the time of this writing)

    * note: See [here](https://curl.haxx.se/libcurl/c/CURLOPT_FTP_FILEMETHOD.html) for the corresponding cURL option.

Those options can be set in bank properties.

See file `global.properties.example` in [biomaj module](https://github.com/genouest/biomaj).

        ftputil.stat.MSParser(),

    ]

    # Valid values for ftp_method options as string and int

    VALID_FTP_FILEMETHOD = {

        "default": pycurl.FTPMETHOD_DEFAULT,

        "multicwd": pycurl.FTPMETHOD_MULTICWD,

        "nocwd": pycurl.FTPMETHOD_NOCWD,

        "singlecwd": pycurl.FTPMETHOD_SINGLECWD,

    }

    def __init__(self, curl_protocol, host, rootdir, http_parse=None):

        """

        Initialize a CurlDownloader.

        # This object is shared by all operations to use the cache.

        # Before using it, call method:`_basic_curl_configuration`.

        self.crl = pycurl.Curl()

        #

        # Initialize options

        #

        # Should we skip SSL verification (cURL -k/--insecure option)

        self.ssl_verifyhost = True

        self.ssl_verifypeer = True

        self.ssl_server_cert = None

        # Keep alive

        self.tcp_keepalive = 0

        # FTP method (cURL --ftp-method option)

        self.ftp_method = pycurl.FTPMETHOD_DEFAULT  # Use cURL default

    def _basic_curl_configuration(self):

        """

            # CURLOPT_CAPATH is for a directory of certificates.

            self.crl.setopt(pycurl.CAINFO, self.ssl_server_cert)

        # Configure ftp method

        self.crl.setopt(pycurl.FTP_FILEMETHOD, self.ftp_method)

        # Configure timeouts

        self.crl.setopt(pycurl.CONNECTTIMEOUT, 300)

        self.crl.setopt(pycurl.TIMEOUT, self.timeout)

        super(CurlDownload, self).set_server(server)

        self.url = self.curl_protocol + '://' + self.server

    def set_options(self, protocol_options):

        super(CurlDownload, self).set_options(protocol_options)

        if "ssl_verifyhost" in protocol_options:

            self.ssl_verifyhost = Utils.to_bool(protocol_options["ssl_verifyhost"])

        if "ssl_verifypeer" in protocol_options:

            self.ssl_verifypeer = Utils.to_bool(protocol_options["ssl_verifypeer"])

        if "ssl_server_cert" in protocol_options:

            self.ssl_server_cert = protocol_options["ssl_server_cert"]

        if "tcp_keepalive" in protocol_options:

            self.tcp_keepalive = Utils.to_int(protocol_options["tcp_keepalive"])

    def set_options(self, options):

        super(CurlDownload, self).set_options(options)

        if "ssl_verifyhost" in options:

            self.ssl_verifyhost = Utils.to_bool(options["ssl_verifyhost"])

        if "ssl_verifypeer" in options:

            self.ssl_verifypeer = Utils.to_bool(options["ssl_verifypeer"])

        if "ssl_server_cert" in options:

            self.ssl_server_cert = options["ssl_server_cert"]

        if "tcp_keepalive" in options:

            self.tcp_keepalive = Utils.to_int(options["tcp_keepalive"])

        if "ftp_method" in options:

            # raw_val is a string which contains the name of the option as in the CLI.

            # We always convert raw_val to a valid integer

            raw_val = options["ftp_method"].lower()

            if raw_val not in self.VALID_FTP_FILEMETHOD:

                raise ValueError("Invalid value for ftp_method")

            self.ftp_method = self.VALID_FTP_FILEMETHOD[raw_val]

    def _append_file_to_download(self, rfile):

        # Add url and root to the file if needed (for safety)

import re

import hashlib

import sys

import os

from biomaj_download.download.curl import CurlDownload

from biomaj_core.utils import Utils

            raise ValueError(msg)

        return super(DirectFTPDownload, self).set_files_to_download(files_to_download)

    def _file_url(self, rfile):

        # rfile['root'] is set to self.rootdir if needed but may be different.

        # We don't use os.path.join because rfile['name'] may starts with /

        return self.url + '/' + rfile['root'] + rfile['name']

    def list(self, directory=''):

        '''

        FTP protocol does not give us the possibility to get file date from remote url

        '''

        # TODO: are we sure about this implementation ?

        self._basic_curl_configuration()

        for rfile in self.files_to_download:

            if self.save_as is None:

                self.save_as = os.path.basename(rfile['name'])

            rfile['save_as'] = self.save_as

            file_url = self._file_url(rfile)

            try:

                self.crl.setopt(pycurl.URL, file_url)

            except Exception:

                self.crl.setopt(pycurl.URL, file_url.encode('ascii', 'ignore'))

            self.crl.setopt(pycurl.URL, file_url)

            self.crl.setopt(pycurl.OPT_FILETIME, True)

            self.crl.setopt(pycurl.NOBODY, True)

            # Very old servers may not support the MDTM commands. Therefore,

            # cURL will raise an error. In that case, we simply skip the rest

            # of the function as it was done before. Download will work however.

            # Note that if the file does not exist, it will be skipped too

            # (that was the case before too). Of course, download will fail in

            # this case.

            try:

                self.crl.perform()

            except Exception:

                continue

            timestamp = self.crl.getinfo(pycurl.INFO_FILETIME)

            dt = datetime.datetime.fromtimestamp(timestamp)

            size_file = int(self.crl.getinfo(pycurl.CONTENT_LENGTH_DOWNLOAD))

            rfile['year'] = dt.year

            rfile['month'] = dt.month

            rfile['day'] = dt.day

            rfile['size'] = size_file

            rfile['hash'] = hashlib.md5(str(timestamp).encode('utf-8')).hexdigest()

        return (self.files_to_download, [])

    def match(self, patterns, file_list, dir_list=None, prefix='', submatch=False):