New upstream version 1.1.20

  # Build the compiled extension and run the project tests

  - "python.exe scratch/test/test_pyflow.py"

branches:

  only:

    - master

    - travis

    - /v\d\.\d\.\d/

notifications:

  - provider: Email

    to:

language: python

# Note PYVER drives hack to use python 2.4, this is

# actually pretty ugly on travis -- process is:

# 1) install # python2.4 from deadsnakes ppa

# 2) shove 2.4 in /usr/bin/python

# 3) set PATH back to /usr/bin

#

# This removes the system python link which is probably not

# smart, but the test works so leaving it for now.

# 

matrix: 

  include: 

    - os: linux

      python: "2.7"

    - os: linux

      sudo: required

      python: "2.7"

      env: PYVER="2.4"

      python: "2.6"

before_install:

  - date -u

  - uname -a

  - lsb_release -a

  - if [ "$PYVER" == "2.4" ]; then sudo add-apt-repository -y ppa:fkrull/deadsnakes && sudo apt-get update -qq; fi

install:

  - if [ "$PYVER" == "2.4" ]; then sudo apt-get install python2.4 -y && python2.4 -V; fi

  - if [ "$PYVER" == "2.4" ]; then sudo rm -f /usr/bin/python && sudo ln -s /usr/bin/python2.4 /usr/bin/python; fi

  - if [ "$PYVER" == "2.4" ]; then export PATH=/usr/bin:$PATH; fi

  - python -V

script:

branches:

  only:

    - master

    - travis

    - /v\d\.\d\.\d/

notifications:

  email:

License

-------

pyFlow source code is provided under the [BSD 2-Clause License]

(pyflow/COPYRIGHT.txt).

pyFlow source code is provided under the [BSD 2-Clause License](pyflow/COPYRIGHT.txt).

Releases

  required for each task.

- Recursive workflow specification: take any existing pyFlow object and

  use it as a task in another pyFlow.

- Dynamic workflow specification: define a wait on task specification rather

  than just tasks, so that tasks can be defined based on the results of 

  upstream tasks (note: recursive workflows are an even better way to do this)

- Dynamic workflow specification:

  - Define a wait on task specification rather than just tasks, so that tasks can be defined based on the results of upstream tasks (note: recursive workflows are an even better way to do this)

  - Tasks/Task-trees can be canceled during the workflow based on the results of other tasks

- Detects and reports all failed tasks with consistent workflow-level logging.

- Task-level logging: All task stderr is logged and decorated,

  eg. [time][host][workflow_run][taskid]

intallation. To do so unpack the tarball and use the setup script

as follows:

```

```bash

tar -xzf pyflow-X.Y.Z.tar.gz

cd pyflow-X.Y.Z

python setup.py build install

src/ directory to the system search path. For instance:

usepyflow.py:

```

```python

import sys

sys.path.append("/path/to/pyflow/src")

is composed of (1) the run() PID and (2) the number of times run() has been called on

the workflow by the same process. These two values are joined by an underscore

- 'component' - the name of the pyflow thread, the primary threads are

  'WorkflowManager' which runs the worklow() method, and 'TaskManager' which

  'WorkflowManager' which runs the workflow() method, and 'TaskManager' which

  polls the task graph and launches jobs.

In the task logs, only the stderr stream is decorated. The prefix in

workflows.

The runstate of each task is in

pyflow.data/state/pyflow_tasks_runstate.txt, the description of each

task is in pyflow.data/state/pyflow_tasks_info.txt. At the beginning of

`pyflow.data/state/pyflow_tasks_runstate.txt`, the description of each

task is in `pyflow.data/state/pyflow_tasks_info.txt`. At the beginning of

each run any existing task files are backed up in

pyflow.data/state/backup.

    pyflow.data/state/make_pyflow_task_graph.py

This script can be run without arguments to produce the current task graph in

dot format based on the data files in the pyflow.data/state/ directory.

dot format based on the data files in the `pyflow.data/state/` directory.

#### Site configuration:

The file ${pyflowDir}/src/pyflowConfig.py contains any pyflow variables or

The file `${pyflowDir}/src/pyflowConfig.py` contains any pyflow variables or

functions which would be likely to need configuration at a new site. This

currently incudes:

#

# pyFlow - a lightweight parallel task engine

#

# Copyright (c) 2012-2015 Illumina, Inc.

# Copyright (c) 2012-2017 Illumina, Inc.

# All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE

# POSSIBILITY OF SUCH DAMAGE.

#

#

"""

    # TODO: what if there's an NFS delay updating all the bams while

    # we're reading them out here? make this process more robust -- we

    # should know how many BAM's we're expecting, in a way that's

    # robust to interuption/restart

    # robust to interruption/restart

    #

    bams = {}

    bamDigger = FileDigger(".bam", ["Flowcell_", "bam", "Project_", "Sample_"])

    #

    # mergedBams contains all bams from the current run, we also add any from a

    # previous interupted run:

    # previous interrupted run:

    mergedBamDigger = FileDigger(mergedBamExt, ["Project_", "Sample_"])

    for (project, sample, bamFile) in mergedBamDigger.getNextFile(mergedBamDir) :

        key = (project, sample)