Skip to content

common_dagster.specs

specs.py — the declaration vocabulary: @step · @sensor · @check · Job.

Everything here is a RECORD, not machinery: decorators attach specs to methods; Job declares a cropping. Resolution into real Dagster objects happens in build.py — declared data, constructed machinery (the house law).

Classes

CheckSpec

A custom check method bound to one product written by a step.

__init__

def __init__(fn: Callable, name: str, step: str, product: str | None = None, blocking: bool = False) -> None

Fields and properties declared here; inherited members belong to the bases above.

blocking

blocking: bool = False

fn

fn: Callable

name

name: str

product

product: str | None = None

step

step: str

Job

A cropping of the flow’s steps, runnable on its own terms.

config carries ONLY the row selector (e.g. {"run": "weekly"}) — it travels as a run TAG (dg/run), never as baked per-op config, so subset re-execution stays clean (audit: the old H1 rewind killer). Knob VALUES live in the configs DB row, read live inside the step.

__init__

def __init__(steps: tuple[str, ...] | list[str], config: dict[str, str] | None = None, schedule: str | None = None, timezone: str = 'UTC', manual: bool = False, executor: str = 'multiprocess', resources: dict | None = None, max_concurrent: int | None = None, tags: dict[str, str] | None = None, job_type: JobType | None = None) -> None

Fields and properties declared here; inherited members belong to the bases above.

config

config: dict[str, str] | None = None

executor

executor: str = 'multiprocess'

job_type

job_type: JobType | None = None

manual

manual: bool = False

max_concurrent

max_concurrent: int | None = None

resources

resources: dict | None = None

schedule

schedule: str | None = None

steps

steps: tuple[str, ...] | list[str]

tags

tags: dict[str, str] | None = None

timezone

timezone: str = 'UTC'

SensorSpec

A real sensor method — arbitrary logic, resolved to a Dagster sensor.

requires upgrades the method to a HYBRID: the sensor monitors those asset keys AND runs the body — the body receives upstream_done: bool as its third parameter and decides (probe AND completion). Cursors advance only when the body returns a RunRequest, so an unfired probe never eats the upstream materialization.

__init__

def __init__(fn: Callable, name: str, job: str, interval: int = 30, requires: tuple[str, ...] = ()) -> None

Fields and properties declared here; inherited members belong to the bases above.

after

Read-only property.

after: tuple[str, ...]

Compatibility spelling for declarations written before requires.

fn

fn: Callable

interval

interval: int = 30

job

job: str

name

name: str

requires

requires: tuple[str, ...] = ()

SkipAsset

Bases: Exception

A deliberate successful no-output outcome for an optional operation.

Raise this from a step when there is no product to confirm. The compiler catches it at the Dagster boundary, emits no optional output, and therefore lets Dagster skip every dependent product confirmation honestly.

StepSpec

One operation and the persisted products it confirms. Captured by @step.

The operation always keeps name as its own Dagster node/asset identity. produces is a separate, canonical product identity: the compiler creates one confirmation asset per product and lets any number of operations write it without creating duplicate Dagster definitions.

__init__

def __init__(fn: Callable, name: str, produces: tuple[str, ...] = (), deps: tuple[Any, ...] = (), group: str | None = None, checks: tuple[Any, ...] | dict[str, tuple[Any, ...]] = (), retry: RetryPolicy | None = None, isolate: bool = False, resources: dict | None = None, image: str | None = None, description: str | None = None, tags: dict[str, str] | None = None) -> None

Fields and properties declared here; inherited members belong to the bases above.

checks

checks: tuple[Any, ...] | dict[str, tuple[Any, ...]] = ()

deps

deps: tuple[Any, ...] = ()

description

description: str | None = None

fn

fn: Callable

group

group: str | None = None

image

image: str | None = None

isolate

isolate: bool = False

key_name

Read-only property.

key_name: str

Backward-compatible operation key leaf (always the descriptive name).

name

name: str

param_deps

Read-only property.

param_deps: tuple[str, ...]

pytest-style data deps: method params beyond the reserved set.

produces

produces: tuple[str, ...] = ()

product_names

Read-only property.

product_names: tuple[str, ...]

Canonical persisted products confirmed after this operation.

products

Read-only property.

products: tuple[str, ...]

Preferred spelling for the normalized persisted-product tuple.

resources

resources: dict | None = None

retry

retry: RetryPolicy | None = None

tags

tags: dict[str, str] | None = None

Functions

check

def check(*, step: str, product: str | None = None, blocking: bool = False, name: str | None = None) -> Callable

Declare a custom check for one product written by step.

product is optional only when the step declares exactly one product. Requiring it for multi-product operations prevents a check from silently attaching to the wrong persisted table.

sensor

def sensor(*, job: str, interval: int = 30, requires: tuple | list = (), after: tuple | list = (), name: str | None = None) -> Callable

Declare a real sensor method — probes, windows, cursors, Skip/Run.

requires=["container/asset", ...] makes it a HYBRID sensor: the body’s signature becomes (self, context, upstream_done) and fires only on its own verdict — the daily_ready pattern (vendor-served AND islands-done). after= remains a compatibility alias and cannot be mixed with it.

step

def step(fn: Callable | None = None, *, produces: str | tuple[str, ...] | list[str] | None = None, deps: tuple | list = (), group: str | None = None, checks: tuple | list | dict[str, tuple | list] = (), retry: RetryPolicy | None = None, isolate: bool = False, resources: dict | None = None, image: str | None = None, description: str | None = None, tags: dict | None = None, name: str | None = None) -> Callable

Declare a flow method as a step (one asset, one node).

Laws enforced at class creation (DGFlow.init_subclass): resources/image without isolate=True refuse loudly (audit rule 5 — they’d be silently inert under the multiprocess executor). Isolate pods run on the config-DB row’s defaults: launchpad overrides beyond run never reach them (only the dg/run tag crosses the pod).