Some tasks need to know too much about their processing pipeline

[ixjlyons]

For example, a pipeline contains a scikit-learn StandardScaler. The task making use of the pipeline needs to use data from a previous task to fit the StandardScaler. In order to do this, it needs to know which task has the feature data, the name of the block that needs to be fit, etc.

Basically any time a pipeline has blocks that need special attention (any time a task needs to look at pipeline.named_blocks), there is a bad coupling that will make it messy to implement generic task classes/mixins.

Something like this is a workaround that could probably be made much simpler with some support from AxoPy:

pipeline = copper.Pipeline(...)
task = SomeTask(pipeline)
exp = Experiment([task])

def on_finish():
    reader = exp.storage.require_task('SomeTask')
    pipeline.named_blocks['Transformer'].fit(reader.array('data'))

task.finish.connect(on_finish)

exp.run()

Notice the ordering here is strict and ugly. Maybe having hooks in Experiment on task start/finish/whatever could work.