Skip to content

Dagster

Containers declare flows; this package resolves them to genuine Dagster objects and records every run in a durable ledger. It carries the fleet’s single Dagster version pin.

Install

  • common-dagster: the authoring layer, the store, the checks, the monitoring ledger.
  • common-dagster[k8s]: the Kubernetes pod launcher for isolated steps.
  • common-dagster[dbt]: the dbt flow and its readiness gate.

A flow

from common_dagster import DGFlow, Job, RunConfig, build_definitions, fresh, nonempty, sensor, step
class EventsFlow(DGFlow):
CONTAINER = "core_data_updater"
MONITORING = True
JOBS = {
"daily": Job(steps=["update_splits", "update_dividends"], job_type="update"),
}
@step(produces="raw.splits", checks=[nonempty(), fresh(days=3)])
async def update_splits(self, cfg: RunConfig): ...
@step(produces="raw.dividends")
async def update_dividends(self, cfg: RunConfig): ...
@sensor(job="daily", cron="0 6 * * 1-5")
def nightly(self, ctx):
return self.run(key=ctx.today) if ctx.ready else self.skip("not ready")
defs = build_definitions(EventsFlow)

Steps are methods and methods are assets. produces names the canonical product a step confirms; checks run against it and their results are persisted. JOBS crop the steps into Dagster jobs; sensors and schedules sit beside them. Everything build_definitions returns is a real Dagster object, so raw Dagster stays legal alongside through extra=.

Handoffs and isolation

Values passed between steps ride DGStore, an IO manager over common-db’s stash and retrieve. MemStore stands in for tests. A step may run isolated in its own pod through the K8sLauncher; pod.run_step is the entry inside the pod, and the outcome comes back as a PodOutcome.

The monitoring ledger

from common_dagster.monitoring import MonitoringRepo, bind
Run, Stage, TargetReceipt, FindingSummary = bind("core_data_updater") # the container's tables
repo = MonitoringRepo(db, container="core_data_updater", run_id=run_id)
await repo.start_run(flow, job, "update")

Four tables, run, stage, target receipt, finding summary, live in the named monitoring database target under the container’s own schema. bind(schema) registers a container’s concrete tables once on common-db’s shared Base, so provisioning can ensure them with the rest of the census. Writes are fail-safe after validation: an observability outage is logged and returned as False, never taking down business work. Report reads are fail-loud. A flow with MONITORING = True records every run, stage, receipt and check result automatically.

Database access

The layer reaches the database through common-db’s DBClient. The connection comes from the container’s settings; DBClient.from_env() is the established container contract for the POSTGRES_* variables and is called here on the container’s behalf.