Skip to content

common_dagster.flow

flow.py — DGFlow: the base every container’s job classes subclass.

The base COLLECTS (decorated methods → specs), VALIDATES at class creation (illegal classes refuse at import — laws are physics, not discipline), and CONSTRUCTS fresh per step execution (no state ever crosses steps except through tables/the store).

class EventsFlow(DGFlow):
CONTAINER = "core_data_updater"
class Config(RunConfig):
run: str | None = None # None → the dg/run tag → DEFAULT_RUN
def setup(self, cfg): ... # per-STEP-execution, live DB reads live here
@step(produces="dividends")
async def dividends(self, cfg): ...
JOBS = {
"nightly": Job(
steps=["dividends"], job_type="update", schedule="30 0 * * *"
)
}

Classes

DGFlow

Subclass me. Methods do; deps draw; JOBS crop; build.py resolves.

__init__

def __init__() -> None

Initialize execution-local state; nothing crosses step instances.

Fields and properties declared here; inherited members belong to the bases above.

AFTER

AFTER: dict[str, list[str]] = {}

CONTAINER

CONTAINER: str | None = None

Config

Config: type[RunConfig] = RunConfig

DB_FACTORY

DB_FACTORY: Any = None

DEFAULT_RUN

DEFAULT_RUN: str = 'default'

JOBS

JOBS: dict[str, Job] = {}

MAX_CONCURRENT

MAX_CONCURRENT: int = 4

MONITORING

MONITORING: bool = False

ON_MATERIALIZE

ON_MATERIALIZE: dict[str, list[str]] = {}

RESOURCES

RESOURCES: dict = {'requests': {'cpu': '500m', 'memory': '4Gi'}}

TABLE_SCHEMA

TABLE_SCHEMA: str = 'raw'

context

context: Any

monitoring

monitoring: Any | None = None

monitoring_stage_id

monitoring_stage_id: str | None = None

jobs

@classmethod
def jobs() -> tuple[str, ...]

resolve_run

def resolve_run(cfg: Any, context: Any = None) -> str

Which configs-DB row: launchpad override > the job’s dg/run tag > DEFAULT_RUN. The selector travels as a TAG, never baked op config.

run

@staticmethod
def run(key: str | None = None, tags: dict | None = None) -> RunRequest

set_product_metadata

def set_product_metadata(product: str, **metadata: Any) -> None

Attach JSON-safe causal metadata to this step’s canonical product.

The descriptive operation carries the metadata through the current run; the generated confirmer copies only the named product’s values to the canonical materialization. Consumers such as dbt readiness can therefore distinguish “a run happened today” from “this run revised history beginning on 2020-08-31” without coupling to container code.

setup

def setup(cfg: RunConfig) -> None

Override me: read the config row, build feeds/repos. Runs fresh per STEP EXECUTION in the run pod — live reads by construction (audit rule 3: nothing here is ever frozen in the code server).

skip

@staticmethod
def skip(reason: str) -> SkipReason

skip_asset

@staticmethod
def skip_asset(reason: str) -> SkipAsset

Build the deliberate step exception: raise self.skip_asset(...).

steps

@classmethod
def steps() -> tuple[str, ...]

suppress_materialization

def suppress_materialization(reason: str) -> None

Finish the operation but publish no canonical product event.

This is for successful side effects that intentionally target a scratch destination (for example a schema override). It differs from raising :class:SkipAsset: the method is allowed to finish all of its work, then the generated boundary emits only a descriptive operation-completion event. That truthful internal signal releases dependent scratch steps; canonical product confirmation and its checks remain suppressed.

suppress_product_materialization

def suppress_product_materialization(product: str, reason: str) -> None

Suppress only one declared canonical product after an operation.

Multi-output operations can commit one table while truthfully leaving a sibling unchanged. The descriptive operation still completes, but the named product’s generated confirmer skips both checks and materialization. If another selected writer in the same run did mutate that product, its unsuppressed evidence wins and the shared confirmer still runs.

tables

@classmethod
def tables() -> tuple[str, ...]

Every table this flow lands (the produces census).

teardown

def teardown() -> None

Override to release per-execution resources, even after failure.

today

@staticmethod
def today() -> str

RunConfig

Bases: DagsterConfig

The launchpad-form base. Fields default None = ‘use the DB row value’; a filled field is a one-off override for that launch.