Michael R. Crusoe · Michael R. Crusoe · Michael R. Crusoe · Michael R. Crusoe · Michael R. Crusoe · Michael R. Crusoe
--- a/.gitignore
+++ b/.gitignore
@@ -6,10 +6,12 @@
 __pycache__
 .eggs/
 .cache/
+.pytest_cache/
 .idea/
 /test-report.xml
 /test-report-*.xml
 /venv/
+tmp/
 /src/toil/test/cwl/spec
 /cwltool_deps/
 /docs/generated_rst/

--- a/Makefile
+++ b/Makefile
@@ -23,8 +23,8 @@ The 'prepare' target installs Toil's build requirements into the current virtual
 The 'develop' target creates an editable install of Toil and its runtime requirements in the
 current virtualenv. The install is called 'editable' because changes to the source code
 immediately affect the virtualenv. Set the 'extras' variable to ensure that the 'develop' target
-installs support for extras. Consult setup.py for the list of supported extras. To install Toil
-in develop mode with all extras, run
+installs support for extras; some tests require extras to be installed. Consult setup.py for the
+list of supported extras. To install Toil in develop mode with all extras, run

 	make develop extras=[all]

@@ -37,6 +37,9 @@ uploaded to PyPI.

 The 'docs' target uses Sphinx to create HTML documentation in the docs/_build directory

+Targets are provided to run Toil's tests. Note that these targets do *not* automatically install
+Toil's dependencies; it is recommended to 'make develop' before running any of them.
+
 The 'test' target runs Toil's unit tests serially with pytest. It will run some docker tests and
 setup. If you wish to avoid this, use the 'test_offline' target instead. Note: this target does not
 capture output from the terminal. For any of the test targets, set the 'tests' variable to run a
@@ -45,7 +48,9 @@ particular test, e.g.
 	make test tests=src/toil/test/sort/sortTest.py::SortTest::testSort

 The 'test_offline' target is similar to 'test' but it skips the docker dependent tests and their
-setup.
+setup. It can also be used to invoke individual tests, e.g.
+
+    make test_offline tests_local=src/toil/test/sort/sortTest.py::SortTest::testSort

 The 'integration_test_local' target runs toil's integration tests. These are more thorough but also
 more costly than the regular unit tests. For the AWS integration tests to run, the environment
@@ -146,7 +151,7 @@ test_offline: check_venv check_build_reqs
 # The auto-deployment test needs the docker appliance
 test: check_venv check_build_reqs docker
 	TOIL_APPLIANCE_SELF=$(docker_registry)/$(docker_base_name):$(docker_tag) \
-	    $(python) -m pytest $(pytest_args_local) $(tests)
+	    $(python) -m pytest --cov=toil $(pytest_args_local) $(tests)

 # For running integration tests locally in series (uses the -s argument for pyTest)
 integration_test_local: check_venv check_build_reqs sdist push_docker
@@ -262,12 +267,12 @@ check_build_reqs:


 prepare: check_venv
-	$(pip) install sphinx==1.5.5 mock==1.0.1 pytest==2.8.3 stubserver==1.0.1 \
+	$(pip) install sphinx==1.5.5 mock==1.0.1 pytest==3.6.2 stubserver==1.0.1 \
 		pytest-timeout==1.2.0 cwltest


 check_venv:
-	@$(python) -c 'import sys; sys.exit( int( not (hasattr(sys, "real_prefix") or ( hasattr(sys, "base_prefix") and sys.base_prefix != sys.prefix ) ) ) )' \
+	@$(python) -c 'import sys, os; sys.exit( int( 0 if "VIRTUAL_ENV" in os.environ else 1 ) )' \
 		|| ( printf "$(red)A virtualenv must be active.$(normal)\n" ; false )



--- a/README.rst
+++ b/README.rst
-.. image:: https://badge.waffle.io/BD2KGenomics/toil.svg?label=ready&title=Ready
-   :target: https://waffle.io/BD2KGenomics/toil
-   :alt: 'Stories in Ready'
+ATTENTION: Toil has moved from https://github.com/BD2KGenomics/toil to https://github.com/DataBiosphere/toil as of July 5th, 2018.

 Toil is a scalable, efficient, cross-platform pipeline management system,
 written entirely in Python, and designed around the principles of functional
@@ -13,7 +11,7 @@ programming.
 * Google Groups discussion `forum`_ and videochat `invite list`_.

 .. _website: http://toil.ucsc-cgl.org/
-.. _Read the Docs: http://toil.readthedocs.org/
+.. _Read the Docs: https://toil.readthedocs.io/en/latest
 .. _forum: https://groups.google.com/forum/#!forum/toil-community
 .. _invite list: https://groups.google.com/forum/#!forum/toil-community-videochats
 .. _blog: https://toilpipelines.wordpress.com/

--- a/attic/toil-sort-example.py
+++ b/attic/toil-sort-example.py
 from __future__ import absolute_import
+from six.moves import xrange
 from argparse import ArgumentParser
 import os
 import logging

--- a/debian/changelog
+++ b/debian/changelog
+toil (3.18.0-1) UNRELEASED; urgency=medium
+
+  * New upstream version.
+  * skip galaxy-lib, not yet packaged for Debian
+  * Adjust docs for a Debian installed Toil.
+  * cgcloud is no longer needed (moved into Toil itself).
+  * Add patch to adjust to newer version of the CWL reference runner.
+  * Fix spelling.
+
+ -- Michael R. Crusoe <michael.crusoe@gmail.com>  Mon, 07 Jan 2019 02:04:03 -0800
+
 toil (3.16.0-1) experimental; urgency=medium

  * New upstream version.

--- a/debian/clean
+++ b/debian/clean
 version_template.pyc
 src/toil/version.pyc
 src/toil.egg-info/
+docs/vendor/sphinxcontrib/__pycache__/
+src/toil/__pycache__/
+src/toil/batchSystems/__pycache__/
+src/toil/cwl/__pycache__/
+src/toil/jobStores/__pycache__/
+src/toil/lib/__pycache__/
+src/toil/provisioners/__pycache__/
+src/toil/provisioners/aws/__pycache__/
+src/toil/test/__pycache__/
--- a/debian/control
+++ b/debian/control
@@ -6,35 +6,32 @@ Uploaders: Steffen Moeller <moeller@debian.org>,
           Michael R. Crusoe <michael.crusoe@gmail.com>
 Build-Depends: debhelper (>= 10),
               dh-python,
-               python-all,
-               python-setuptools,
-               python-docker,
-               python-galaxy-lib,
+               python3-all,
+               python3-setuptools,
+               python3-docker,
+	       python3-dill,
+	       python3-future,
               cwltool,
 # documentation
-               python-urllib3,
-               python-chardet,
-               python-sphinx
+               python3-urllib3,
+               python3-chardet,
+               python3-sphinx
 Standards-Version: 4.1.4
 Homepage: https://github.com/BD2KGenomics/toil
-X-Python-Version: >= 2.6
-#X-Python3-Version: >= 3.2
 Vcs-Git: https://salsa.debian.org/med-team/toil.git
 Vcs-Browser: https://salsa.debian.org/med-team/toil
 #Testsuite: autopkgtest-pkg-python

 Package: toil
 Architecture: all
-Depends: ${python:Depends}, ${misc:Depends},
-         python-cgcloud-lib,
-         python-schema-salad,
+Depends: ${python3:Depends}, ${misc:Depends},
         cwltool
 Recommends:
-         python-boto, python-boto3,
-         python-protobuf,
-         python-futures,
-         python-libcloud
-Suggests: python-azure, python-azure-storage
+         python3-boto, python3-boto3,
+         python3-protobuf,
+         python3-futures,
+         python3-libcloud
+Suggests: python3-azure, python3-azure-storage
 #Suggests: python-toil-doc
 Description: cross-platform workflow engine
 Toil is a scalable, efficient, cross-platform and easy-to-use workflow
@@ -42,19 +39,6 @@ Description: cross-platform workflow engine
 balancers like Slurm or the Sun Grid Engine.  Toil is compatible with
 the Common Workflow Language (CWL).

-#Package: python3-toil
-#Architecture: all
-#Depends: ${python3:Depends}, ${misc:Depends}
-#Suggests: python-toil-doc
-#Conflicts: python-toil
-#Description: cross-platform workflow engine (Python 3)
-# Toil is a scalable, efficient, cross-platform and easy-to-use workflow
-# engine in pure Python.  It works with several well established load
-# balancers like Slurm or the Sun Grid Engine.  Toil is compatible with
-# the Common Workflow Language (CWL).
-# .
-# This package installs the library for Python 3.
-
 #Package: python-toil-doc
 #Architecture: all
 #Section: doc

--- a/debian/patches/adjust_to_newer_cwltool
+++ b/debian/patches/adjust_to_newer_cwltool
+Author: Michael R. Crusoe <michael.crusoe@gmail.com>
+Description: Adjust to newer version of cwltool
+--- toil.orig/src/toil/cwl/cwltoil.py
+++ toil/src/toil/cwl/cwltoil.py
+@@ -40,6 +40,7 @@
+ 
+ from schema_salad import validate
+ import schema_salad.ref_resolver
+from schema_salad.schema import Names
+ 
+ import cwltool.errors
+ import cwltool.load_tool
+@@ -58,7 +59,7 @@
+                                 get_listing, MapperEnt, visit_class,
+                                 normalizeFilesDirs)
+ from cwltool.process import (shortname, fill_in_defaults, compute_checksums,
+-                             collectFilesAndDirs, add_sizes)
+                             add_sizes)
+ from cwltool.secrets import SecretStore
+ from cwltool.software_requirements import (
+     DependenciesConfiguration, get_container_from_software_requirements)
+@@ -389,8 +390,21 @@
+     """Copy input files out of the global file store and update location and
+     path."""
+ 
+-    jobfiles = []  # type: List[Dict[Text, Any]]
+-    collectFilesAndDirs(cwljob, jobfiles)
+    def _collectDirEntries(obj):
+        # type: (Union[Dict[Text, Any], List[Dict[Text, Any]]]) -> Iterator[Dict[Text, Any]]
+        if isinstance(obj, dict):
+            if obj.get("class") in ("File", "Directory"):
+                yield obj
+            else:
+                for sub_obj in obj.values():
+                    for dir_entry in _collectDirEntries(sub_obj):
+                        yield dir_entry
+        elif isinstance(obj, list):
+            for sub_obj in obj:
+                for dir_entry in _collectDirEntries(sub_obj):
+                    yield dir_entry
+
+    jobfiles = list(_collectDirEntries(cwljob))
+     pm = ToilPathMapper(
+         jobfiles, "", outdir, separateDirs=False, stage_listing=True)
+     for f, p in pm.items():
+@@ -455,6 +469,7 @@
+         self.addChild(realjob)
+         return realjob.rv()
+ 
+
+ def _makeNestedTempDir(top, seed, levels=2):
+     """
+     Gets a temporary directory in the hierarchy of directories under a given
+@@ -497,12 +512,29 @@
+         if runtime_context.builder:
+             builder = runtime_context.builder
+         else:
+-            builder = cwltool.builder.Builder(cwljob)
+-            builder.requirements = self.cwltool.requirements
+-            builder.outdir = None
+-            builder.tmpdir = None
+-            builder.timeout = runtime_context.eval_timeout
+-            builder.resources = {}
+            builder = cwltool.builder.Builder(
+                job=cwljob,
+                files=[],
+                bindings=[],
+                schemaDefs={},
+                names=Names(),
+                requirements=self.cwltool.requirements,
+                hints=[],
+                resources={},
+                mutation_manager=None,
+                formatgraph=None,
+                make_fs_access=runtime_context.make_fs_access,
+                fs_access=runtime_context.make_fs_access(''),
+                job_script_provider=None,
+                timeout=runtime_context.eval_timeout,
+                debug=False,
+                js_console=False,
+                force_docker_pull=False,
+                loadListing=u'',
+                outdir=u'',
+                tmpdir=u'',
+                stagedir=u''
+            )
+         req = tool.evalResources(builder, runtime_context)
+         # pass the default of None if basecommand is empty
+         unitName = self.cwltool.tool.get("baseCommand", None)
+@@ -530,7 +562,8 @@
+         fill_in_defaults(
+             self.step_inputs, cwljob,
+             self.runtime_context.make_fs_access(""))
+-        for inp_id in cwljob.keys():
+        immobile_cwljob_dict = copy.deepcopy(cwljob)
+        for inp_id in immobile_cwljob_dict.keys():
+             found = False
+             for field in self.cwltool.inputs_record_schema['fields']:
+                 if field['name'] == inp_id:
+@@ -576,6 +609,7 @@
+ 
+         return output
+ 
+
+ def makeJob(tool, jobobj, step_inputs, runtime_context):
+     """Create the correct Toil Job object for the CWL tool (workflow, job, or job
+     wrapper for dynamic resource requirements.)
+@@ -1079,7 +1113,8 @@
+ 
+     if options.provisioner and not options.jobStore:
+         raise NoSuchJobStoreException(
+-            'Please specify a jobstore with the --jobStore option when specifying a provisioner.')
+            'Please specify a jobstore with the --jobStore option when '
+            'specifying a provisioner.')
+ 
+     use_container = not options.no_container
+ 
+@@ -1141,9 +1176,9 @@
+             document_loader, avsc_names, processobj, metadata, uri = \
+                 cwltool.load_tool.validate_document(
+                     document_loader, workflowobj, uri,
+                    loading_context.overrides_list, loading_context.metadata,
+                     loading_context.enable_dev, loading_context.strict, False,
+                     loading_context.fetcher_constructor, False,
+-                    loading_context.overrides_list,
+                     do_validate=loading_context.do_validate)
+             loading_context.overrides_list.extend(
+                 metadata.get("cwltool:overrides", []))
+@@ -1215,7 +1250,8 @@
+                 return 33
+ 
+             wf1.cwljob = initialized_job_order
+-            if wf1 is CWLJob:  # Clean up temporary directories only created with CWLJobs.
+            if wf1 is CWLJob:
+                # Clean up temporary directories only created with CWLJobs.
+                 wf1.addFollowOnFn(cleanTempDirs, wf1)
+             outobj = toil.start(wf1)
+ 
+--- toil.orig/src/toil/common.py
+++ toil/src/toil/common.py
+@@ -1083,7 +1083,7 @@
+         So far there is no reason to store any old pids.
+         """
+         with self._jobStore.writeSharedFileStream('pid.log') as f:
+-            f.write(str(os.getpid()))
+            f.write(str(os.getpid()).encode('utf-8'))
+ 
+ 
+ class ToilRestartException(Exception):
+--- toil.orig/src/toil/jobGraph.py
+++ toil/src/toil/jobGraph.py
+@@ -100,6 +100,9 @@
+         # this job
+         self.chainedJobs = chainedJobs
+ 
+    def __hash__(self):
+        return hash(self.jobStoreID)
+
+     def setupJobAfterFailure(self, config):
+         """
+         Reduce the remainingRetryCount if greater than zero and set the memory
--- a/debian/patches/debianize_docs
+++ b/debian/patches/debianize_docs
+Author: Michael R. Crusoe <michael.crusoe@gmail.com>
+Description: Update docs to reflect a local install
+--- toil.orig/docs/index.rst
+++ toil/docs/index.rst
+@@ -31,7 +31,6 @@
+    :caption: Getting Started
+    :maxdepth: 2
+ 
+-   gettingStarted/install
+    gettingStarted/quickStart
+ 
+ .. toctree::
+--- toil.orig/docs/gettingStarted/quickStart.rst
+++ toil/docs/gettingStarted/quickStart.rst
+@@ -8,11 +8,9 @@
+ Running a basic workflow
+ ------------------------
+ 
+-A Toil workflow can be run with just three steps:
+A Toil workflow can be run with just two steps:
+ 
+-1. Install Toil (see :ref:`installation-ref`)
+-
+-2. Copy and paste the following code block into a new file called ``helloWorld.py``:
+1. Copy and paste the following code block into a new file called ``helloWorld.py``:
+ 
+    .. code-block:: python
+ 
+@@ -30,14 +28,9 @@
+           print output
+ 
+ 
+-3. Specify the name of the :ref:`job store <jobStoreOverview>` and run the workflow::
+-
+-       (venv) $ python helloWorld.py file:my-job-store
+2. Specify the name of the :ref:`job store <jobStoreOverview>` and run the workflow::
+ 
+-.. note::
+-
+-   Don't actually type ``(venv) $`` in at the beginning of each command. This is intended only to remind the user that
+-   they should have their :ref:`virtual environment <venvPrep>` running.
+       python helloWorld.py file:my-job-store
+ 
+ Congratulations! You've run your first Toil workflow using the default :ref:`Batch System <batchsysteminterface>`, ``singleMachine``,
+ using the ``file`` job store.
+@@ -64,13 +57,6 @@
+ workflows that are portable across multiple workflow engines and platforms.
+ Running CWL workflows using Toil is easy.
+ 
+-#. First ensure that Toil is installed with the
+-   ``cwl`` extra (see :ref:`extras`)::
+-
+-       (venv) $ pip install 'toil[cwl]'
+-
+-   This installs the ``toil-cwl-runner`` and ``cwltoil`` executables.
+-
+ #. Copy and paste the following code block into ``example.cwl``:
+ 
+    .. code-block:: yaml
+@@ -96,11 +82,11 @@
+ 
+ #. To run the workflow simply enter ::
+ 
+-        (venv) $ toil-cwl-runner example.cwl example-job.yaml
+        $ toil-cwl-runner example.cwl example-job.yaml
+ 
+    Your output will be in ``output.txt``::
+ 
+-        (venv) $ cat output.txt
+        $ cat output.txt
+         Hello world!
+ 
+ To learn more about CWL, see the `CWL User Guide`_ (from where this example was
+@@ -110,7 +96,7 @@
+ 
+ For information on using CWL with Toil see the section :ref:`cwl`
+ 
+-.. _CWL User Guide: http://www.commonwl.org/v1.0/UserGuide.html
+.. _CWL User Guide: https://www.commonwl.org/user_guide/
+ 
+ Running a basic WDL workflow
+ ----------------------------
+@@ -118,13 +104,6 @@
+ The `Workflow Description Language`_ (WDL) is another emerging language for writing workflows that are portable across multiple workflow engines and platforms.
+ Running WDL workflows using Toil is still in alpha, and currently experimental.  Toil currently supports basic workflow syntax (see :ref:`wdlSupport` for more details and examples).  Here we go over running a basic WDL helloworld workflow.
+ 
+-#. First ensure that Toil is installed with the
+-   ``wdl`` extra (see :ref:`extras`)::
+-
+-        (venv) $ pip install 'toil[wdl]'
+-
+-   This installs the ``toil-wdl-runner`` executable.
+-
+ #. Copy and paste the following code block into ``wdl-helloworld.wdl``::
+ 
+         workflow write_simple_file {
+@@ -144,11 +123,11 @@
+ 
+ #. To run the workflow simply enter ::
+ 
+-        (venv) $ toil-wdl-runner wdl-helloworld.wdl wdl-helloworld.json
+        $ toil-wdl-runner wdl-helloworld.wdl wdl-helloworld.json
+ 
+    Your output will be in ``wdl-helloworld-output.txt``::
+ 
+-        (venv) $ cat wdl-helloworld-output.txt
+        $ cat wdl-helloworld-output.txt
+         Hello world!
+ 
+ To learn more about WDL, see the main `WDL website`_ .
+@@ -175,7 +154,7 @@
+ 
+ #. Run it with the default settings::
+ 
+-      (venv) $ python sort.py file:jobStore
+      $ python sort.py file:jobStore
+ 
+    The workflow created a file called ``sortedFile.txt`` in your current directory.
+    Have a look at it and notice that it contains a whole lot of sorted lines!
+@@ -192,7 +171,7 @@
+ 
+ 3. Run with custom options::
+ 
+-      (venv) $ python sort.py file:jobStore --numLines=5000 --lineLength=10 --overwriteOutput=True --workDir=/tmp/
+      $ python sort.py file:jobStore --numLines=5000 --lineLength=10 --overwriteOutput=True --workDir=/tmp/
+ 
+    Here we see that we can add our own options to a Toil script. As noted above, the first two
+    options, ``--numLines`` and ``--lineLength``, determine the number of lines and how many characters are in each line.
+@@ -308,7 +287,7 @@
+ with the ``--logLevel`` flag. For example, to only log ``CRITICAL`` level
+ messages to the screen::
+ 
+-   (venv) $ python sort.py file:jobStore --logLevel=critical --overwriteOutput=True
+   $ python sort.py file:jobStore --logLevel=critical --overwriteOutput=True
+ 
+ This hides most of the information we get from the Toil run. For more detail,
+ we can run the pipeline with ``--logLevel=debug`` to see a comprehensive
+@@ -332,7 +311,7 @@
+ 
+ When we run the pipeline, Toil will show a detailed failure log with a traceback::
+ 
+-   (venv) $ python sort.py file:jobStore
+   $ python sort.py file:jobStore
+    ...
+    ---TOIL WORKER OUTPUT LOG---
+    ...
+@@ -354,11 +333,11 @@
+ failure, the job store is preserved so that the workflow can be restarted,
+ starting from the previously failed jobs. We can restart the pipeline by running ::
+ 
+-   (venv) $ python sort.py file:jobStore --restart --overwriteOutput=True
+   $ python sort.py file:jobStore --restart --overwriteOutput=True
+ 
+ We can also change the number of times Toil will attempt to retry a failed job::
+ 
+-   (venv) $ python sort.py --retryCount 2 --restart --overwriteOutput=True
+   $ python sort.py --retryCount 2 --restart --overwriteOutput=True
+ 
+ You'll now see Toil attempt to rerun the failed job until it runs out of tries.
+ ``--retryCount`` is useful for non-systemic errors, like downloading a file that
+@@ -369,7 +348,7 @@
+ 
+ ::
+ 
+-    (venv) $ python sort.py --restart --overwriteOutput=True
+    $ python sort.py --restart --overwriteOutput=True
+ 
+ The pipeline will run successfully, and the job store will be removed on the
+ pipeline's completion.
+@@ -428,7 +407,7 @@
+ 
+ #. Use the :ref:`destroyCluster` command to destroy the cluster::
+ 
+-        (venv) $ toil destroy-cluster --zone us-west-2a <cluster-name>
+        $ toil destroy-cluster --zone us-west-2a <cluster-name>
+ 
+    Note that this command will destroy the cluster leader
+    node and any resources created to run the job, including the S3 bucket.
--- a/debian/patches/no_galaxy_lib
+++ b/debian/patches/no_galaxy_lib
+Author: Michael R. Crusoe <michael.crusoe@gmail.com>
+Description: skip galaxy-lib, not yet packaged for Debian
+--- toil.orig/setup.py
+++ toil/setup.py
+@@ -81,8 +81,7 @@
+         gcs]
+     cwl_reqs = [
+         cwltool,
+-        schemaSalad,
+-        galaxyLib]
+        schemaSalad]
+     wdl_reqs = []
+     htcondor_reqs = [
+         htcondor]
--- a/debian/patches/no_virtualenv_to_create_manpages.patch
+++ b/debian/patches/no_virtualenv_to_create_manpages.patch
+Author: Steffen Moeller <moeller@debian.org>
+Description: ignore requirement to use a virtualenv
 Index: toil/docs/conf.py
 ===================================================================
 --- toil.orig/docs/conf.py

--- a/debian/patches/series
+++ b/debian/patches/series
 version.patch
 no_virtualenv_to_create_manpages.patch
 setting_version.patch
+no_galaxy_lib
+debianize_docs
+spelling
+adjust_to_newer_cwltool
--- a/debian/patches/setting_version.patch
+++ b/debian/patches/setting_version.patch
@@ -3,11 +3,24 @@ Author: Steffen Moeller <moeller@debian.org>
 Bug-Debian: https://bugs.debian.org/851365
 Forwarded: not-needed
 Last-Update: 2018-06-14
+
+Generated by running `python version_template.py` from a git checkout of the
+source to the tag corresponding to the release
 ---

-Index: toil/src/toil/version.py
-===================================================================
 --- /dev/null
 +++ toil/src/toil/version.py
-@@ -0,0 +1 @@
-+distVersion="3.16.0"
+@@ -0,0 +1,13 @@
+dockerShortTag = '3.18.0-84239d8'
+baseVersion = '3.18.0'
+dockerTag = '3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be'
+dockerName = 'toil'
+buildNumber = None
+cgcloudVersion = '1.6.0a1.dev393'
+version = '3.18.0-84239d802248a5f4a220e762b3b8ce5cc92af0be'
+dirty = False
+shortVersion = '3.18.0-84239d8'
+currentCommit = '84239d802248a5f4a220e762b3b8ce5cc92af0be'
+dockerMinimalTag = '3.18.0'
+distVersion = '3.18.0'
+dockerRegistry = 'quay.io/ucsc_cgl'
--- a/debian/patches/spelling
+++ b/debian/patches/spelling
+From: Michael R. Crusoe <michael.crusoe@gmail.com>
+Subject: Lintian-inspired spelling fixes
+--- toil.orig/docs/running/cloud/amazon.rst
+++ toil/docs/running/cloud/amazon.rst
+@@ -165,7 +165,7 @@
+ 
+     When using the Toil provisioner, the appliance image will be automatically chosen
+     based on the pip-installed version of Toil on your system. That choice can be
+-    overriden by setting the environment variables ``TOIL_DOCKER_REGISTRY`` and ``TOIL_DOCKER_NAME`` or
+    overridden by setting the environment variables ``TOIL_DOCKER_REGISTRY`` and ``TOIL_DOCKER_NAME`` or
+     ``TOIL_APPLIANCE_SELF``. See :ref:`envars` for more information on these variables. If
+     you are developing with autoscaling and want to test and build your own
+     appliance have a look at :ref:`appliance_dev`.
+--- toil.orig/docs/running/cloud/gce.rst
+++ toil/docs/running/cloud/gce.rst
+@@ -32,7 +32,7 @@
+ 
+        $ chmod 400 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub
+ 
+-#. Add your newly formated public key to Google. To do this, log into your Google Cloud account
+#. Add your newly formatted public key to Google. To do this, log into your Google Cloud account
+    and go to `metadata`_ section under the Compute tab.
+ 
+    .. image:: googleScreenShot.png
+--- toil.orig/docs/running/cwl.rst
+++ toil/docs/running/cwl.rst
+@@ -33,7 +33,7 @@
+ ------------------------------------
+ 
+ A CWL workflow can be run indirectly in a native Toil script. However, this is not the :ref:`standard <cwl>` way to run
+-CWL workflows with Toil and doing so comes at the cost of job efficency. For some use cases, such as running one process on
+CWL workflows with Toil and doing so comes at the cost of job efficiency. For some use cases, such as running one process on
+ multiple files, it may be useful. For example, if you want to run a CWL workflow with 3 YML files specifying different
+ samples inputs, it could look something like::
+ 
+--- toil.orig/docs/running/wdl.rst
+++ toil/docs/running/wdl.rst
+@@ -117,7 +117,7 @@
+         A cromwell.jar file is needed in order to run a WDL workflow.
+ 
+ A WDL workflow can be run indirectly in a native Toil script. However, this is not the :ref:`standard <wdl>` way to run
+-WDL workflows with Toil and doing so comes at the cost of job efficency. For some use cases, such as running one process on
+WDL workflows with Toil and doing so comes at the cost of job efficiency. For some use cases, such as running one process on
+ multiple files, it may be useful. For example, if you want to run a WDL workflow with 3 JSON files specifying different
+ samples inputs, it could look something like::
+ 
+--- toil.orig/src/toil/jobStores/abstractJobStore.py
+++ toil/src/toil/jobStores/abstractJobStore.py
+@@ -758,7 +758,7 @@
+     def jobs(self):
+         """
+         Best effort attempt to return iterator on all jobs in the store. The iterator may not
+-        return all jobs and may also contain orphaned jobs that have already finished succesfully
+        return all jobs and may also contain orphaned jobs that have already finished successfully
+         and should not be rerun. To guarantee you get any and all jobs that can be run instead
+         construct a more expensive ToilState object
+ 
+--- toil.orig/src/toil/lib/threading.py
+++ toil/src/toil/lib/threading.py
+@@ -39,7 +39,7 @@
+ class ExceptionalThread(threading.Thread):
+     """
+     A thread whose join() method re-raises exceptions raised during run(). While join() is
+-    idempotent, the exception is only during the first invocation of join() that succesfully
+    idempotent, the exception is only during the first invocation of join() that successfully
+     joined the thread. If join() times out, no exception will be re reraised even though an
+     exception might already have occured in run().
+ 
+--- toil.orig/src/toil/job.py
+++ toil/src/toil/job.py
+@@ -273,11 +273,11 @@
+             exhausting all their retries, remove any successor jobs and rerun this job to restart the
+             subtree. Job must be a leaf vertex in the job graph when initially defined, see
+             :func:`toil.job.Job.checkNewCheckpointsAreCutVertices`.
+-        :type cores: int or string convertable by toil.lib.humanize.human2bytes to an int
+-        :type disk: int or string convertable by toil.lib.humanize.human2bytes to an int
+        :type cores: int or string convertible by toil.lib.humanize.human2bytes to an int
+        :type disk: int or string convertible by toil.lib.humanize.human2bytes to an int
+         :type preemptable: bool
+-        :type cache: int or string convertable by toil.lib.humanize.human2bytes to an int
+-        :type memory: int or string convertable by toil.lib.humanize.human2bytes to an int
+        :type cache: int or string convertible by toil.lib.humanize.human2bytes to an int
+        :type memory: int or string convertible by toil.lib.humanize.human2bytes to an int
+         """
+         requirements = {'memory': memory, 'cores': cores, 'disk': disk,
+                         'preemptable': preemptable}
+--- toil.orig/docs/appendices/environment_vars.rst
+++ toil/docs/appendices/environment_vars.rst
+@@ -28,7 +28,7 @@
+ | TOIL_DOCKER_REGISTRY   | The URL of the registry of the Toil Appliance      |
+ |                        | image you wish to use. Docker will use Dockerhub   |
+ |                        | by default, but the quay.io registry is also       |
+-|                        | very popular and easily specifiable by settting    |
+|                        | very popular and easily specifiable by setting     |
+ |                        | this option to ``quay.io``.                        |
+ +------------------------+----------------------------------------------------+
+ | TOIL_DOCKER_NAME       | The name of the Toil Appliance image you           |
+--- toil.orig/docs/developingWorkflows/developing.rst
+++ toil/docs/developingWorkflows/developing.rst
+@@ -559,7 +559,7 @@
+ 
+     Just like regular promises, the return value must be determined prior to
+     scheduling any job that depends on the return value. In our example above, notice
+-    how the dependant jobs were follow ons to the parent while promising jobs are
+    how the dependent jobs were follow ons to the parent while promising jobs are
+     children of the parent. This ordering ensures that all promises are
+     properly fulfilled.
+ 
--- a/debian/patches/version.patch
+++ b/debian/patches/version.patch
-Index: toil/setup.py
-===================================================================
+Author: Michael R. Crusoe <michael.crusoe@gmail.com>
+Description: allow the use of Debian packaged dependencies
 --- toil.orig/setup.py
 +++ toil/setup.py
-@@ -34,9 +34,9 @@ def runSetup():
+@@ -34,8 +34,8 @@
     gcs = 'google-cloud-storage==1.6.0'
     gcs_oauth2_boto_plugin = 'gcs_oauth2_boto_plugin==1.14'
     apacheLibcloud = 'apache-libcloud==2.2.1'
-    cwltool = 'cwltool==1.0.20180518123035'
-+    cwltool = 'cwltool>=1.0.20180518123035'
-     schemaSalad = 'schema-salad >= 2.6, < 3'
-    galaxyLib = 'galaxy-lib==17.9.3'
-+    galaxyLib = 'galaxy-lib>=17.9.3'
+-    cwltool = 'cwltool==1.0.20180820141117'
+-    schemaSalad = 'schema-salad>=2.6, <3'
+    cwltool = 'cwltool>=1.0.20180820141117'
+    schemaSalad = 'schema-salad>=3, <3.1'
+     galaxyLib = 'galaxy-lib==17.9.3'
     htcondor = 'htcondor>=8.6.0'
- 
-     mesos_reqs = [
-@@ -137,33 +137,6 @@ def importVersion():
+     dill = 'dill==0.2.7.1'
+@@ -150,33 +150,6 @@
     required.
     """
     import imp
@@ -42,7 +41,7 @@ Index: toil/setup.py
 -                raise
 -
 -        if old != new:
-            with NamedTemporaryFile(dir='src/toil', prefix='version.py.', delete=False) as f:
+-            with NamedTemporaryFile(mode='w',dir='src/toil', prefix='version.py.', delete=False) as f:
 -                f.write(new)
 -            os.rename(f.name, 'src/toil/version.py')
     # Unfortunately, we can't use a straight import here because that would also load the stuff

--- a/debian/rules
+++ b/debian/rules
@@ -3,17 +3,21 @@

 export PYBUILD_NAME=toil

-export PYBUILD_DESTDIR_python2=debian/toil/
+export PYBUILD_DESTDIR_python3=debian/toil/

 export PYBUILD_DISABLE=test
-export PYBUILD_DISABLE_python3=1
-
+export PYBUILD_DISABLE_python2=1
+#CURDIR ?= $(realpath $(dir $(firstword $(MAKEFILE_LIST)))/..)
 %:
-	dh $@ --with python2 --buildsystem=pybuild
+	dh $@ --with python3 --buildsystem=pybuild

 override_dh_auto_build:
 	dh_auto_build
-	$(MAKE) -C docs man
+	PYTHONPATH=$(CURDIR)/src $(MAKE) -C docs man
+
+override_dh_auto_install:
+	dh_auto_install
+	find $(CURDIR)/debian -name cwltoil -delete


 # If you need to rebuild the Sphinx documentation

--- a/docs/appendices/environment_vars.rst
+++ b/docs/appendices/environment_vars.rst
@@ -52,6 +52,8 @@ There are several environment variables that affect the way Toil runs.
 |                        | deleted until all associated nodes have been       |
 |                        | terminated.                                        |
 +------------------------+----------------------------------------------------+
+| TOIL_AZURE_ZONE        | A specified region for provisioning instances.     |
+------------------------+----------------------------------------------------+
 | TOIL_SLURM_ARGS        | Arguments for sbatch for the slurm batch system.   |
 |                        | Do not pass CPU or memory specifications here.     |
 |                        | Instead, define resource requirements for the job. |

--- a/docs/contributing/contributing.rst
+++ b/docs/contributing/contributing.rst
@@ -24,28 +24,23 @@ environment variables are set (see :ref:`test_env_vars`).
 |                         | the terminal as expected.                         |
 +-------------------------+---------------------------------------------------+

-Run all tests (including slow tests):
-
-::
+Run all tests (including slow tests)::

    $ make test

-Run only quick tests (as of Sep 18, 2017, this was < 30 minutes):

-::
+Run only quick tests (as of Jul 25, 2018, this was ~ 20 minutes)::

    $ export TOIL_TEST_QUICK=True; make test

-Run an individual test with:
-
-::
+Run an individual test with::

    $ make test tests=src/toil/test/sort/sortTest.py::SortTest::testSort

 The default value for ``tests`` is ``"src"`` which includes all tests in the
 ``src/`` subdirectory of the project root. Tests that require a particular
 feature will be skipped implicitly. If you want to explicitly skip tests that
-depend on a currently installed *feature*, use:
+depend on a currently installed *feature*, use

 ::

@@ -55,28 +50,18 @@ This will run only the tests that don't depend on the ``azure`` extra, even if
 that extra is currently installed. Note the distinction between the terms
 *feature* and *extra*. Every extra is a feature but there are features that are
 not extras, such as the ``gridengine`` and ``parasol`` features.  To skip tests
-involving both the Parasol feature and the Azure extra, use the following
-
-::
+involving both the Parasol feature and the Azure extra, use the following::

    $ make test tests="-m 'not azure and not parasol' src"

-Running Tests (pytest)
-~~~~~~~~~~~~~~~~~~~~~~
-
-Often it is simpler to use pytest directly, instead of calling the ``make`` wrapper.
-This usually works as expected, but some tests need some manual preparation.
-
- - Running tests that make use of Docker (e.g. autoscaling tests and Docker tests)
-   require an appliance image to be hosted. This process first requires :ref:`quaySetup`.
-   Then to build and host the appliance image run the ``make`` targets ``docker``
-   and ``push_docker`` respectively.

- - Running integration tests require setting the environment variable ::

-       export TOIL_TEST_INTEGRATIVE=True
+Running Tests with pytest
+~~~~~~~~~~~~~~~~~~~~~~~~~

-To run a specific test with pytest ::
+Often it is simpler to use pytest directly, instead of calling the ``make`` wrapper.
+This usually works as expected, but some tests need some manual preparation. To run a specific test with pytest,
+use the following::

    python -m pytest src/toil/test/sort/sortTest.py::SortTest::testSort

@@ -86,6 +71,38 @@ For more information, see the `pytest documentation`_.

 .. _test_env_vars:

+Running Integration Tests
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These tests are generally only run using in our CI workflow due to their resource requirements and cost. However, they
+can be made available for local testing:
+
+ - Running tests that make use of Docker (e.g. autoscaling tests and Docker tests) require an appliance image to be
+   hosted. First, make sure you have gone through the set up found in :ref:`quaySetup`.
+   Then to build and host the appliance image run the ``make`` target ``push_docker``. ::
+
+        $ make push_docker
+
+ - Running integration tests require activation via an environment variable as well as exporting information relevant to
+   the desired tests. Enable the integration tests::
+
+        $ export TOIL_TEST_INTEGRATIVE=True
+
+ - Finally, set the environment variables for keyname and desired zone::
+
+        $ export TOIL_X_KEYNAME=[Your Keyname]
+        $ export TOIL_X_ZONE=[Desired Zone]
+
+   Where ``X`` is one of our currently supported cloud providers (``AZURE``, ``GCE``, ``AWS``).
+
+ - For example, to prepare for running Azure related integration tests in the ``westus`` region::
+
+       $ export TOIL_TEST_INTEGRATIVE=True
+       $ export TOIL_AZURE_KEYNAME=[Your keyname]
+       $ export TOIL_AZURE_ZONE=westus
+
+ - See the above sections for guidance on running tests.
+
 Test Environment Variables
 ~~~~~~~~~~~~~~~~~~~~~~~~~~

@@ -128,7 +145,7 @@ Test Environment Variables

 .. _standard temporary directory: https://docs.python.org/2/library/tempfile.html#tempfile.gettempdir

-.. admonition:: Partial install and failing tests.
+.. admonition:: Partial install and failing tests

    Some tests may fail with an ImportError if the required extras are not installed. 
    Install Toil with all of the extras
@@ -196,13 +213,13 @@ Making Your Own Toil Docker Image
 **Note!**  Toil checks if the docker image specified by TOIL_APPLIANCE_SELF
 exists prior to launching by using the docker v2 schema.  This should be
 valid for any major docker repository, but there is an option to override
-this if desired using the option: `--forceDockerAppliance`.
+this if desired using the option: `-\\-forceDockerAppliance`.

 Here is a general workflow (similar instructions apply when using Docker Hub):

-1. Make some changes to the provisioner of your local version of Toil.
+#. Make some changes to the provisioner of your local version of Toil

-2. Go to the location where you installed the Toil source code and run::
+#. Go to the location where you installed the Toil source code and run ::

        $ make docker

@@ -210,31 +227,31 @@ Here is a general workflow (similar instructions apply when using Docker Hub):
   your personal `Quay`_ account. If you have not installed Toil source
   code yet see :ref:`buildFromSource`.

-3. If it's not already you will need Docker installed and need
+#. If it's not already you will need Docker installed and need
   to `log into Quay`_. Also you will want to make sure that your Quay
   account is public.

-4. Set the environment variable ``TOIL_DOCKER_REGISTRY`` to your Quay
+#. Set the environment variable ``TOIL_DOCKER_REGISTRY`` to your Quay
   account. If you find yourself doing this often you may want to add ::

        export TOIL_DOCKER_REGISTRY=quay.io/<MY_QUAY_USERNAME>

   to your ``.bashrc`` or equivalent.

-5. Now you can run::
+#. Now you can run ::

        $ make push_docker

   which will upload the docker image to your Quay account. Take note of
   the image's tag for the next step.

-6. Finally you will need to tell Toil from where to pull the Appliance
+#. Finally you will need to tell Toil from where to pull the Appliance
   image you've created (it uses the Toil release you have installed by
   default). To do this set the environment variable
   ``TOIL_APPLIANCE_SELF`` to the url of your image. For more info see
   :ref:`envars`.

-7. Now you can launch your cluster! For more information see
+#. Now you can launch your cluster! For more information see
   :ref:`Autoscaling`.

 Running a Cluster Locally

--- a/docs/developingWorkflows/developing.rst
+++ b/docs/developingWorkflows/developing.rst
@@ -449,7 +449,7 @@ Running this workflow results in three log messages from the jobs: ``i is 1``
 from ``j1``, ``i is 2`` from ``j2`` and ``i is 3`` from ``j3``.

 The return value from the first job is *promised* to the second job by the call
-to :func:`toil.job.Job.rv` in the line::
+to :func:`toil.job.Job.rv` in the following line::

    j2 = j1.addChildFn(fn, j1.rv())

@@ -569,7 +569,7 @@ multiple jobs' output values::
 FileID
 ------

-This object is a small wrapper around Python's builtin string class. It is used to
+The :class:`toil.fileStore.FileID` class is a small wrapper around Python's builtin string class. It is used to
 represent a file's ID in the file store, and has a ``size`` attribute that is the
 file's size in bytes. This object is returned by ``importFile`` and ``writeGlobalFile``.


--- a/docs/developingWorkflows/toilAPIBatchsystem.rst
+++ b/docs/developingWorkflows/toilAPIBatchsystem.rst
@@ -26,7 +26,7 @@ requirements, and another - for resources requirements (without the `-l` prefix)
    export TOIL_TORQUE_REQS="walltime=1:00:00"

 For GridEngine (SGE, UGE), there is an additional environmental variable to define the
-`parallel environment <https://blogs.oracle.com/templedf/entry/configuring_a_new_parallel_environment>`_
+`parallel environment <http://www.softpanorama.org/HPC/Grid_engine/parallel_environment.shtml#Important_details>`_
 for running multicore jobs::

    export TOIL_GRIDENGINE_PE='smp'