Skip to content

Commit

Permalink
More updates in README
Browse files Browse the repository at this point in the history
  • Loading branch information
samuell committed Aug 31, 2015
1 parent d4323ab commit f1b27ec
Showing 1 changed file with 14 additions and 10 deletions.
24 changes: 14 additions & 10 deletions README.md
Expand Up @@ -10,19 +10,18 @@ workflow system that aims to make writing scientific workflows (consisting of
numerous interdependent commandline applications) more fluent, flexible and
modular.

While Luigi is a great, fun-to-use, and very flexible library, its default
way of defining dependencies by hard coding them in each task's requires()
function, is not optimal for the type of workflows that are common in
scientific fields such as bioinformatics, where multiple inputs and outputs,
complex dependencies between tools, and the need to quickly try different
workflow connectivity (such as plugging in extra filtering steps) in an
explorative fashion, is central to the way of working.

SciLuigi was designed to solve these problem we were facing when trying
Luigi is a great, flexible, and very fun-to-use library. It has turned out though,
that its default way of defining dependencies by hard coding them in each task's
requires() function is not optimal for some type of workflows common in scientific
fields such as bioinformatics, where multiple inputs and outputs, complex dependencies,
and the need to quickly try different workflow connectivity (e.g. plugging in extra
filtering steps) in an explorative fashion is central to the way of working.

SciLuigi was designed to solve some these problem we were facing when trying
to use luigi for defining complex workflows for data preprocessing,
machine-learning and cross-validation.

To achieve that, SciLuigi provides the following features over vanilla Luigi:
To achieve that, SciLuigi provides the following "features" over vanilla Luigi:

- Separation of dependency definitions from the tasks themselves,
for improved modularity and composability.
Expand All @@ -38,6 +37,11 @@ To achieve that, SciLuigi provides the following features over vanilla Luigi:
- Integration with some HPC workload managers.
(So far only [SLURM](http://slurm.schedmd.com/) though).

Because of Luigi's great easy-to-use API, these changes have been implemented
as a very thin layer on top of luigi's own API, and no changes to the luigi
core is needed at all, so you can continue leveraging the work already being
put into maintaining and further developing luigi, by the team at Spotify and others.

## Workflow code quick demo

Just to give a quick feel for how a workflow definition might look like in SciLuigi, check this code example
Expand Down

0 comments on commit f1b27ec

Please sign in to comment.