This article covers internal technical details of
Flow executions. The implementation of a
Flow may change at any time without warning.
Flow executions are coordinated across two different actors:
Compiles the DAG, excecutes layers in a specific order, and polls for completion.
Executes a single
The following is a rough outline of the call structure of a
Flow.scheduleis called and the first set of runnable tasks are identified
Layer, the entry hook is invoked.
The number of splits for each
Layersplit, the submission hook is invoked.
Layeris submitted to the executor it is being run on (thread, docker, batch, etc.)
Flow.executeis called and the
Layeris prepared for execution.
The parameters for
The execution hook is invoked.
Layerartifacts are written to the
Layeris waited on for completion.
Layerfails, the retry hook is invoked and the
Recordis written to the
Layeris marked as complete by the scheduler.
New runnable layers are identified and scheduled.