common_dagster.flow
flow.py — DGFlow: the base every container’s job classes subclass.
The base COLLECTS (decorated methods → specs), VALIDATES at class creation (illegal classes refuse at import — laws are physics, not discipline), and CONSTRUCTS fresh per step execution (no state ever crosses steps except through tables/the store).
class EventsFlow(DGFlow): CONTAINER = "core_data_updater" class Config(RunConfig): run: str | None = None # None → the dg/run tag → DEFAULT_RUN def setup(self, cfg): ... # per-STEP-execution, live DB reads live here @step(produces="dividends") async def dividends(self, cfg): ... JOBS = { "nightly": Job( steps=["dividends"], job_type="update", schedule="30 0 * * *" ) }Classes
DGFlow
Subclass me. Methods do; deps draw; JOBS crop; build.py resolves.
__init__
def __init__() -> NoneInitialize execution-local state; nothing crosses step instances.
Fields and properties declared here; inherited members belong to the bases above.
AFTER
AFTER: dict[str, list[str]] = {}CONTAINER
CONTAINER: str | None = NoneConfig
Config: type[RunConfig] = RunConfigDB_FACTORY
DB_FACTORY: Any = NoneDEFAULT_RUN
DEFAULT_RUN: str = 'default'JOBS
JOBS: dict[str, Job] = {}MAX_CONCURRENT
MAX_CONCURRENT: int = 4MONITORING
MONITORING: bool = FalseON_MATERIALIZE
ON_MATERIALIZE: dict[str, list[str]] = {}RESOURCES
RESOURCES: dict = {'requests': {'cpu': '500m', 'memory': '4Gi'}}TABLE_SCHEMA
TABLE_SCHEMA: str = 'raw'context
context: Anymonitoring
monitoring: Any | None = Nonemonitoring_stage_id
monitoring_stage_id: str | None = Nonejobs
@classmethoddef jobs() -> tuple[str, ...]resolve_run
def resolve_run(cfg: Any, context: Any = None) -> strWhich configs-DB row: launchpad override > the job’s dg/run tag > DEFAULT_RUN. The selector travels as a TAG, never baked op config.
run
@staticmethoddef run(key: str | None = None, tags: dict | None = None) -> RunRequestset_product_metadata
def set_product_metadata(product: str, **metadata: Any) -> NoneAttach JSON-safe causal metadata to this step’s canonical product.
The descriptive operation carries the metadata through the current run; the generated confirmer copies only the named product’s values to the canonical materialization. Consumers such as dbt readiness can therefore distinguish “a run happened today” from “this run revised history beginning on 2020-08-31” without coupling to container code.
setup
def setup(cfg: RunConfig) -> NoneOverride me: read the config row, build feeds/repos. Runs fresh per STEP EXECUTION in the run pod — live reads by construction (audit rule 3: nothing here is ever frozen in the code server).
skip
@staticmethoddef skip(reason: str) -> SkipReasonskip_asset
@staticmethoddef skip_asset(reason: str) -> SkipAssetBuild the deliberate step exception: raise self.skip_asset(...).
steps
@classmethoddef steps() -> tuple[str, ...]suppress_materialization
def suppress_materialization(reason: str) -> NoneFinish the operation but publish no canonical product event.
This is for successful side effects that intentionally target a scratch
destination (for example a schema override). It differs from raising
:class:SkipAsset: the method is allowed to finish all of its work, then
the generated boundary emits only a descriptive operation-completion
event. That truthful internal signal releases dependent scratch steps;
canonical product confirmation and its checks remain suppressed.
suppress_product_materialization
def suppress_product_materialization(product: str, reason: str) -> NoneSuppress only one declared canonical product after an operation.
Multi-output operations can commit one table while truthfully leaving a sibling unchanged. The descriptive operation still completes, but the named product’s generated confirmer skips both checks and materialization. If another selected writer in the same run did mutate that product, its unsuppressed evidence wins and the shared confirmer still runs.
tables
@classmethoddef tables() -> tuple[str, ...]Every table this flow lands (the produces census).
teardown
def teardown() -> NoneOverride to release per-execution resources, even after failure.
today
@staticmethoddef today() -> strRunConfig
Bases: DagsterConfig
The launchpad-form base. Fields default None = ‘use the DB row value’; a filled field is a one-off override for that launch.