
pyFlow - a lightweight parallel task engine 
===========================================

Chris Saunders (csaunders@illumina.com)  
Version: ${VERSION}


SUMMARY:
--------

pyFlow manages running tasks in the context of a task dependency
graph. It has some similarities to make. pyFlow is not a program -- it
is a python module, and workflows are defined using pyFlow by writing
regular python code with the pyFlow API.

FEATURES:
---------

- Define workflows as python code
- Run workflows on localhost or sge
- Continue workflows which have partially completed
- Task resource management: Specify number of threads and memory
  required for each task.
- Recursive workflow specification: take any existing pyFlow object and
  use it as a task in another pyFlow.
- Dynamic workflow specification: define a wait on task specification rather
  than just tasks, so that tasks can be defined based on the results of 
  upstream tasks (note: recursive workflows are an even better way to do this)
- Detects and reports all failed tasks with consistent workflow-level logging.
- Task-level logging: All task stderr is logged and decorated,
  eg. [time][host][workflow_run][taskid]
- Task timing: Task wrapper function provides wall time for every task
- Task priority: Tasks which are simultanously eligible to run can be assigned
relative priorities to be run or queued first. 
- Task mutex sets: define sets of tasks which access an exclusive resource
- Email notification on job completion/error/exception
- Provide ongoing task summary report at specified intervals
- Output task graph in dot format

LICENSE:
--------

pyFlow source code is provided under the [BSD 2-Clause License] (COPYRIGHT.txt).

INSTALL:
--------

pyFlow can be installed and used on python versions in the 2.4 to
2.7 series

The pyflow module can be installed using standard python distutils
intallation. To do so unpack the tarball and use the setup script
as follows:

```
tar -xzf pyflow-X.Y.Z.tar.gz
cd pyflow-X.Y.Z
python setup.py build install
```

If installation in not convenient, you can simply add the pyflow 
src/ directory to the system search path. For instance:

usepyflow.py:
```
import sys
sys.path.append("/path/to/pyflow/src")

from pyflow import WorkflowRunner
```



WRITING WORKFLOWS:
------------------

Briefly, pyFlow workflows are written by creating a new class which
inherits from pyflow.WorkflowRunner. This class then defines its
workflow by overloading the WorkflowRunner.workflow()
method. Workflows are run by instantiating a workflow class and
calling the WorkflowRunner.run() method.

A very simple demonstration of the minimal workflow setup and run
described above is available in the directory: `${pyflowDir}/demo/helloWorld/`

Several other demonstration workflows are available:
`${pyflowDir}/demo/simpleDemo` – a basic feature sandbox
`${pyflowDir}/demo/subWorkflow` – shows how recursive workflow invocation works

The developer documentation for the pyflow API can be generated by running
`${pyflowDir}/doc/getApiDoc.py` or `python ${pyflowDir}/src/pydoc.py`

An advanced proof-of-concept demonstration of bclToBam conversion
is also available in `${pyflowDir}/demo/bclToBwaBam`



USING WORKFLOWS:
----------------

When running a pyFlow workflow, all logs and state information are
written into a single "pyflow.data" directory. The root of this
directory is specified in the workflow.run() call.

### Logging:

pyFlow creates a primary workflow-level log, and 2 log files to
capture all task stdout and stderr, respectively.

Workflow-level log information is copied to both stderr and
pyflow.data/logs/pyflow_log.txt.  All workflow log messages are
prefixed with "[time] [hosname] [workflow_run] [component] ". Where:

- 'time' is UTC in ISO 8601 format.
- 'workflow_run' is an id that's weakly unique for each run of the workflow. It 
is composed of (1) the run() PID and (2) the number of times run() has been called on
the workflow by the same process. These two values are joined by an underscore
- 'component' - the name of the pyflow thread, the primary threads are
  'WorkflowManager' which runs the worklow() method, and 'TaskManager' which
  polls the task graph and launches jobs. 

In the task logs, only the stderr stream is decorated. The prefix in
this case is: "[time] [hostname] [workflow_run] [taskname] ". The
'taskname" is usually the label provided for each task in its
addTask() call. All tasks are launched by a task wrapping function,
and any messages from the taskWrapper (as opposed to the task command
itself) will use an extended taskname:
"pyflowTaskWrapper:${tasklabel}". One example where the task wrapper
writes to the log is to report the total runtime for its task.

All logging is append only -- pyFlow does not overwrite logs even over
multiple runs. The workflow_run id can be used to select out the 
information from a specific run if restarting/continuing a run 
multiple times.

### State:

pyFlow continues jobs by marking their status in a file, *not* by
looking for the presence of file targets. This is a major difference
from make and must be kept in mind when restarting interrupted
workflows.

The runstate of each task is in
pyflow.data/state/pyflow_tasks_runstate.txt, the description of each
task is in pyflow.data/state/pyflow_tasks_info.txt. At the beginning of
each run any existing task files are backed up in
pyflow.data/state/backup.
 
### Other:

#### Email notification:

When running a workflow with one or more email addresses given in the
mailTo argument, pyflow will attempt to send a notification describing the
outcome of the run under any circumstance short of host hardware failure.
The email should result from 1 of 3 outcomes: (1) successful run completion
(2) the first unrecoverable task failure, with a description of the error
(3) an unhandled software exception. Mail comes by default from
"pyflow-bot@YOURDOMAIN" (configurable). Note that (1) you may 
have to change the email address from the automatically detected domain to
to recieve emails, and (2) you may need to check your junk-mail
filter to recieve notifications. It is best to configure one of the demo
scripts to email you on a new machine to test out any issues before starting
a production run.

#### Graph output:

pyFlow provides a script which can be used to produce a graph of the current
task dependencies, where each node colored by the task status. The graph
generation script is automatically created for each run in the pyflow state
directory here:

pyflow.data/state/make_pyflow_task_graph.py

This script can be run without arguments to produce the current task graph in
dot format based on the data files in the pyflow.data/state/ directory.

#### Site configuration:

The file ${pyflowDir}/src/pyflowConfig.py contains any pyflow variables or
functions which would be likely to need configuration at a new site. This
currently incudes:

- from: email address from pyflow
- default memory per task
- default memory available per thread in localhost mode
- qsub arguments given in response to a resource request.

