Skip to content

common_dagster.store

store.py — the IO manager: step returns/params travel through Postgres.

Wraps DBClient’s stash/retrieve (arrow-ipc faithful frames). The four audit IO rules live HERE, once: (1) addressed by ASSET KEY + partition + the PRODUCER run id Dagster resolves through reexecution lineage; (2) connection state built per process in setup_for_execution, never a pydantic field; (3) partition RANGES handled (asset_partition_keys, not the raising singular); (4) one connection per concurrent step — DBClient’s short-lived ops keep the budget honest.

PodWritten is the isolate hand-off sentinel: an isolated step’s pod writes its own output to the store; the in-location wrapper returns PodWritten() and handle_output SKIPS — downstream load_input reads what the pod wrote.

Classes

DGStore

Bases: ConfigurableIOManager

The Postgres-backed hand-off. Configure table to redirect (tests, scratch schemas); prune_days bounds abandoned intermediates.

Fields and properties declared here; inherited members belong to the bases above.

prune_days

prune_days: int = 7

table

table: str = _TABLE

handle_output

def handle_output(context: OutputContext, obj: Any) -> None

load_input

def load_input(context: InputContext) -> Any

teardown_after_execution

def teardown_after_execution(context: Any) -> None

Release the process-local pool after Dagster finishes the run.

MemStore

Bases: ConfigurableIOManager

In-memory twin for tests/local runs — same addressing, no Postgres.

clear

@classmethod
def clear() -> None

handle_output

def handle_output(context: OutputContext, obj: Any) -> None

load_input

def load_input(context: InputContext) -> Any

PodOutcome

The explicit result an isolate pod returns to its parent launcher.

__init__

def __init__(status: Literal['success', 'skipped', 'suppressed'], message: str | None = None) -> None

Fields and properties declared here; inherited members belong to the bases above.

message

message: str | None = None

produced

Read-only property.

produced: bool

status

status: Literal['success', 'skipped', 'suppressed']

PodWritten

Sentinel: the isolated pod already stashed this step’s output.

Types and values

POD_OUTCOME_MESSAGE

POD_OUTCOME_MESSAGE = 'common_dagster/pod_outcome'