Executions¶
Warning
This article covers internal technical details of Flow
executions. The implementation of a Flow
may change at any time without warning.
Overview¶
Laminar Flow
executions are coordinated across two different actors:
Actor |
Description |
---|---|
Scheduler |
Compiles the DAG, excecutes layers in a specific order, and polls for completion. |
Executor |
Executes a single |
Anatomy¶
The following is a rough outline of the call structure of a Flow
execution.
Scheduler
Flow.__call__
Flow.schedule
is called and the first set of runnable tasks are identifiedFor each
Layer
, the entry hook is invoked.The number of splits for each
Layer
is determined.For each
Layer
split, the submission hook is invoked.For each
Layer
split, theLayer
is submitted to the executor it is being run on (thread, docker, batch, etc.)
Executor
Flow.__call__
Flow.execute
is called and theLayer
is prepared for execution.The parameters for
Layer.__call__
are prepared.The execution hook is invoked.
Layer.__call__
Layer
artifacts are written to theDatastore
.
Scheduler
The
Layer
is waited on for completion.If the
Layer
fails, the retry hook is invoked and theLayer
retries.The layer’s
Record
is written to theDataStore
.The
Layer
is marked as complete by the scheduler.New runnable layers are identified and scheduled.