common_db.client.archive
archive.py — ArchiveClient: the parquet dataset leg (full-roster archives).
The storage law: CORE data (curated, keyed, reconciled) lives in Postgres; FULL-ROSTER intraday bars (wide, append-heavy, scanned by slice) live in parquet datasets — hive-partitioned directories any engine can read in place.
The op mechanics:
archive to_arrow -> pyarrow.dataset.write_dataset (hive partitioning,
overwrite_or_ignore, fragments named {batch_id}-part-{i}) —
dataset-APPEND semantics: one batch’s fragments never rewrite
another’s; rerunning the SAME batch_id overwrites its own
fragments (idempotent rerun). Partition-snapshot mode uses one
stable fragment name in every touched hive partition: a full
date+interval re-fetch replaces only that composite snapshot
instead of duplicating an overlap. A _common_metadata sidecar pins
the full schema — types AND column order — because hive
inference alone reads date partitions back as strings.
replace_archive_partitions
manifest-driven exact replacement. Every declared composite
partition becomes one stable snapshot.parquet object,
including declared zero-row tombstones. Local files stage a
sibling object and move it over the stable name; object stores
overwrite the stable object directly. The guarantee is per
partition, not a fictional transaction across S3 objects.
scan ds.dataset -> the SAME where-dict language as the PG reads
(scalar equality · None -> is_null · (lo, hi) incl. open ends
· list membership incl. None member) -> to_table -> any output
(ReadConfig control). lazy=True hands back a polars LazyFrame
over the dataset instead — duration then covers construction
only, and rows stays 0 (nothing counted until collect).
Garage note: roots may be URIs — Garage is S3-compatible, so ‘s3://…’ (or a mounted/s3fs path) works; pyarrow filesystem inference handles the scheme. Bucket/layout init is a platform prerequisite, never done here.
SEAM — archive_table (PG table -> dataset, the offload op) waits on the pg read internals rewrite; it will ride read()‘s arrow output straight into archive(). Deliberately absent until that lands.
Classes
ArchiveClient
The dataset engine — a mixin DBClient composes; standalone-usable now.
archive
def archive(data: Any, ref: DatasetRef | str, *, partition_by: tuple[str, ...] | None = None, config: WriteConfig | None = None, batch_id: str | None = None, partition_snapshot: bool = False) -> WriteResultLand append fragments or one complete snapshot per partition.
partition_by resolves: explicit arg -> the ref’s declared
partitioning -> (‘utc_date’, ‘bar_interval’) for bars ->
(‘utc_date’,) for other dated data -> unpartitioned.
batch_id forced = idempotent rerun (same fragment names overwrite).
partition_snapshot=True requires partition columns and gives every
partition a stable snapshot-part-{i} basename. It is the UPSERT
face for complete partition snapshots: rolling date overlaps and
corrections replace their prior fragment rather than accumulating
duplicate primary keys. Callers must refuse incomplete partitions
before selecting this mode.
config is accepted for signature parity with the PG writes; the
dataset leg takes no knobs from it yet.
new_batch_id
@staticmethoddef new_batch_id() -> strreplace_archive_partitions
def replace_archive_partitions(data: Any, ref: DatasetRef | str, *, partition_by: tuple[str, ...], manifest: Sequence[Mapping[str, Any]], config: WriteConfig | None = None) -> WriteResultReplace exactly the concrete partitions named by manifest.
This is the authoritative-snapshot face, deliberately separate from
:meth:archive’s append/upsert compatibility surface. The complete
manifest is validated before the filesystem is mutated: keys are
concrete and unique, every input row belongs to exactly one declared
key, and no row may widen the replacement scope implicitly. A key
with no rows writes an empty parquet tombstone, retiring that key’s
previous rows without touching a sibling date or interval.
Each partition has one stable object name. A local write stages a
sibling temporary file before the filesystem move, preserving the old
stable file if serialization fails. S3/Garage writes overwrite the
stable object directly; object replacement is credible per key but the
method does not claim atomicity across several objects. Exact mode
refuses roots containing legacy/append fragments. Deployment must use
a clean root (or migrate it out of band) before enabling this path.
config remains a signature-parity placeholder.
scan
def scan(ref: DatasetRef | str, *, columns: tuple[str, ...] | None = None, where: dict[str, Any] | None = None, config: ReadConfig | None = None, lazy: bool = False) -> ReadResultThe dataset read: column scope + the PG dict predicates, partition-
pruned by pyarrow. lazy=True returns a polars LazyFrame instead
(output control does not apply; duration covers construction only).