Dagster
Containers declare flows; this package resolves them to genuine Dagster objects and records every run in a durable ledger. It carries the fleet’s single Dagster version pin.
Install
common-dagster: the authoring layer, the store, the checks, the monitoring ledger.common-dagster[k8s]: the Kubernetes pod launcher for isolated steps.common-dagster[dbt]: the dbt flow and its readiness gate.
A flow
from common_dagster import DGFlow, Job, RunConfig, build_definitions, fresh, nonempty, sensor, step
class EventsFlow(DGFlow): CONTAINER = "core_data_updater" MONITORING = True JOBS = { "daily": Job(steps=["update_splits", "update_dividends"], job_type="update"), }
@step(produces="raw.splits", checks=[nonempty(), fresh(days=3)]) async def update_splits(self, cfg: RunConfig): ...
@step(produces="raw.dividends") async def update_dividends(self, cfg: RunConfig): ...
@sensor(job="daily", cron="0 6 * * 1-5") def nightly(self, ctx): return self.run(key=ctx.today) if ctx.ready else self.skip("not ready")
defs = build_definitions(EventsFlow)Steps are methods and methods are assets. produces names the canonical product a step
confirms; checks run against it and their results are persisted. JOBS crop the steps into
Dagster jobs; sensors and schedules sit beside them. Everything build_definitions returns is a
real Dagster object, so raw Dagster stays legal alongside through extra=.
Handoffs and isolation
Values passed between steps ride DGStore, an IO manager over common-db’s stash and retrieve.
MemStore stands in for tests. A step may run isolated in its own pod through the
K8sLauncher; pod.run_step is the entry inside the pod, and the outcome comes back as a
PodOutcome.
The monitoring ledger
from common_dagster.monitoring import MonitoringRepo, bind
Run, Stage, TargetReceipt, FindingSummary = bind("core_data_updater") # the container's tablesrepo = MonitoringRepo(db, container="core_data_updater", run_id=run_id)await repo.start_run(flow, job, "update")Four tables, run, stage, target receipt, finding summary, live in the named monitoring
database target under the container’s own schema. bind(schema) registers a container’s
concrete tables once on common-db’s shared Base, so provisioning can ensure them with the rest
of the census. Writes are fail-safe after validation: an observability outage is logged and
returned as False, never taking down business work. Report reads are fail-loud. A flow with
MONITORING = True records every run, stage, receipt and check result automatically.
Database access
The layer reaches the database through common-db’s DBClient. The connection comes from the
container’s settings; DBClient.from_env() is the established container contract for the
POSTGRES_* variables and is called here on the container’s behalf.