New upstream version 3.1.14

sudo: false

python:

- '2.7'

- '3.4'

- '3.5'

- '3.6'

- '3.7'

- '3.8'

services:

- mongodb

- elasticsearch

branches:

  except:

  - "/^feature.*$/"

before_install:

  - sudo apt-get install -y libgnutls-dev

addons:

  apt:

    update: true

install:

- pip install flake8

- pip install -r requirements.txt

3.1.14:

  Add repair option

3.1.13:

  Add process name and status in logs

  PR #116 update to use download 3.1.0

3.1.12:

  In case of multiple matches for release regexp, try to determine most recent one

  #115 Correctly use save_as for release file name

3.1.11:

  Increase one log level

  #110 Allow ftps and directftps protocols (needs biomaj-download 3.0.26 and biomaj-core 3.0.19)

  #111 locked bank after bad update command

  Ignore UTF-8 errors in release file

  Add plugin support via biomaj-plugins repo (https://github.com/genouest/biomaj-plugins) to get release and list of files to download from a plugin script.

  Add support for protocol options in global and bank properties (options.names=x,y   options.x=val options.y=val). Options may be ignored or used differently depending on used protocol.

3.1.10:

  Allow to use hardlinks when reusing files from previous releases

3.1.9:

  Fix remote.files recursion

3.1.8:

  Fix uncompress when saved files contains subdirectory

3.1.7:

====================

* Synchronisation:

 * Multiple remote protocols (ftp, sftp, http, local copy, etc.)

  * Multiple remote protocols (ftp, ftps, http, local copy, etc.)

  * Data transfers integrity check

  * Release versioning using a incremental approach

  * Multi threading

  * Data extraction (gzip, tar, bzip)

  * Data tree directory normalisation

  * Plugins support for custom downloads

* Pre &Post processing :

  * Advanced workflow description (D.A.G)

  * Post-process indexation for various bioinformatics software (blast, srs, fastacmd, readseq, etc.)

  * Easy integration of personal scripts for bank post-processing automation

* Supervision:

  * Optional Administration web interface (biomaj-watcher)

  * CLI management

  * Monolithic (local install) or microservice architecture (remote access to a BioMAJ server)

  * Microservice installation allows per process scalability and supervision (number of process in charge of download, execution, etc.)

* Remote access:

  * Optional FTP server providing authenticated or anonymous data access

============

Packages:

* Debian: libcurl-dev, gcc

* CentOs: libcurl-devel, openldap-devel, gcc

    pip install biomaj biomaj-cli biomaj-daemon

You should consider using a Python virtual environment (virtualenv) to install BioMAJ.

In tools/examples, copy the global.properties and update it to match your local

The tools/process contains example process files (python and shell).

Docker

======

You can use BioMAJ with Docker (genouest/biomaj)

    docker pull genouest/biomaj

    docker pull mongo

    docker run --name biomaj-mongodb -d mongo

Examples are available at https://github.com/genouest/biomaj-data

Import bank templates

=====================

Once biomaj is installed, it is possible to import some bank examples with the biomaj client

    # List available templates

    biomaj-cli ... --data-list

    # Import a bank template

    biomaj-cli ... --data-import --bank alu

    # then edit bank template in config directory if needed and launch bank update

    biomaj-cli ... --update --bank alu

Plugins

=======

BioMAJ support python plugins to manage custom downloads where supported protocols

are not enough (http page with unformatted listing, access to protected pages, etc.).

Example of plugins and how to configure them are available on [biomaj-plugins](https://github.com/genouest/biomaj-plugins) repository.

Plugins can define a specific way to:

* retreive release

* list remote files to download

* download remote files

Plugin can define one or many of those features.

Basically, one defined in bank property file:

    # Location of plugins

    plugins_dir=/opt/biomaj-plugins

    # Use plugin to fetch release

    release.plugin=github

    # List of arguments of plugin function with key=value format, comma separated

    release.plugin_args=repo=osallou/goterra-cli

Plugins are used when related workflow step is used:

* release.plugin <= returns remote release

* remote.plugin <= returns list of files to download

* download.plugin <= download files from list of files

API documentation

=================

    nosetests -a '!network'

Monitoring

==========

Remarks

=======

Biomaj uses libcurl, for sftp libcurl must be compiled with sftp support

To delete elasticsearch index:

 curl -XDELETE 'http://localhost:9200/biomaj_test/'

 curl -XDELETE '<http://localhost:9200/biomaj_test/'>

Credits

======

=======

Special thanks for tuco at Pasteur Institute for the intensive testing and new ideas.

Thanks to the old BioMAJ team for the work they have done.

from biomaj.session import Session

from biomaj.workflow import UpdateWorkflow

from biomaj.workflow import RemoveWorkflow

from biomaj.workflow import RepairWorkflow

from biomaj.workflow import Workflow

from biomaj.workflow import ReleaseCheckWorkflow

from biomaj_core.config import BiomajConfig

        # Insert session

        if self.session.get('action') == 'update':

            action = 'last_update_session'

        if self.session.get('action') == 'remove':

        elif self.session.get('action') == 'remove':

            action = 'last_remove_session'

        else:

            action = 'last_update_session'

        cache_dir = self.config.get('cache.dir')

        download_files = self.session.get('download_files')

        return res

    def repair(self):

        """

        Launch a bank repair

        :return: bool

        """

        logging.warning('Bank:' + self.name + ':Repair')

        start_time = datetime.now()

        start_time = time.mktime(start_time.timetuple())

        if not self.is_owner():

            logging.error('Not authorized, bank owned by ' + self.bank['properties']['owner'])

            raise Exception('Not authorized, bank owned by ' + self.bank['properties']['owner'])

        self.run_depends = False

        self.controls()

        if self.options.get_option('release'):

            logging.info('Bank:' + self.name + ':Release:' + self.options.get_option('release'))

            s = self.get_session_from_release(self.options.get_option('release'))

            # No session in prod

            if s is None:

                logging.error('Release does not exists: ' + self.options.get_option('release'))

                return False

            self.load_session(UpdateWorkflow.FLOW, s)

        else:

            logging.info('Bank:' + self.name + ':Release:latest')

            self.load_session(UpdateWorkflow.FLOW)

        self.session.set('action', 'update')

        res = self.start_repair()

        self.session.set('workflow_status', res)

        self.save_session()

        try:

            self.__stats()

        except Exception:

            logging.exception('Failed to send stats')

        end_time = datetime.now()

        end_time = time.mktime(end_time.timetuple())

        self.history.insert({

            'bank': self.name,

            'error': not res,

            'start': start_time,

            'end': end_time,

            'action': 'repair',

            'updated': self.session.get('update')

        })

        return res

    def start_repair(self):

        """

        Start an repair workflow

        """

        workflow = RepairWorkflow(self)

        if self.options and self.options.get_option('redis_host'):

            redis_client = redis.StrictRedis(

                host=self.options.get_option('redis_host'),

                port=self.options.get_option('redis_port'),

                db=self.options.get_option('redis_db'),

                decode_responses=True

            )

            workflow.redis_client = redis_client

            workflow.redis_prefix = self.options.get_option('redis_prefix')

            if redis_client.get(self.options.get_option('redis_prefix') + ':' + self.name + ':action:cancel'):

                logging.warn('Cancel requested, stopping update')

                redis_client.delete(self.options.get_option('redis_prefix') + ':' + self.name + ':action:cancel')

                return False

        return workflow.start()

    def update(self, depends=False):

        """

        Launch a bank update

                        if not reset:

                            logging.info("Process %s not found in %s" % (str(proc), task['name']))

                            return False

            if not set_to_false:

                logging.error('No task found named %s' % (self.options.get_option('from_task')))

                return False

        self.session.set('action', 'update')

        res = self.start_update()

        self.session.set('workflow_status', res)

                    processes_status[bprocess] = res

                    self.set_progress(bmaj_process.name, res)

                    if not res:

                        logging.info("PROC:META:RUN:PROCESS:ERROR:" + bmaj_process.name)

                        self.global_status = False

                        break

                    else:

                        logging.info("PROC:META:RUN:PROCESS:OK:" + bmaj_process.name)

                    if not self.simulate:

                        if self._lock:

                            self._lock.acquire()