DynaStore: canonical data and recoverable projections

A catalog may need exact geometry, spatial joins, discovery search and access to existing files. Those operations do not have identical storage requirements. A platform that hides the differences behind a universal backend risks making unreliable promises about behavior and fallback.

The programme began in 2023. This case describes the inspected public repository snapshot and its documented contracts, not a claim that every branch component is deployed or that repository tests establish production capacity.

HTTP extensions translate API requests; modules own domain logic and data; tasks handle asynchronous work. Canonical PostgreSQL/PostGIS storage is distinct from secondary search indexes and drivers with other capabilities, including DuckDB read-in-place for supported files.

Routing separates WRITE, READ and INDEX operations. SEARCH is derived from INDEX, with READ as a configuration fallback. Query hints can favor a capable read driver, but do not redirect the primary write lane. An empty Elasticsearch result is not a universal trigger to retry a query in PostgreSQL.

Logical feature identity is separate from catalog names and physical backend identifiers. That separation helps route the same domain object across storage and index representations.

In the PostgreSQL-primary bulk-upsert path, canonical rows, lightweight indexing obligations and the storage-drain task obligation use one transaction. The outbox stores IDs or write IDs; draining rereads canonical state before materialization rather than treating copied payloads as authoritative.

Ordinary item-upsert paths dispatch indexing after the primary write through a separate transaction. They do not inherit the bulk path’s atomicity. Cross-call outbox duplicates are also possible; retry-oriented materialization is not exactly-once delivery.

Tasks use capability-aware dispatch and database claiming, with heartbeat/visibility recovery and reaping of stale work. Collection lifecycle states constrain access during provisioning and deletion. External cleanup hooks can still be best-effort; a state machine does not prove every external side effect is fenced.

A retrospective alternative is synchronous dual writing to the primary database and search index. It gives a simpler visible request flow but creates a partial-failure window across systems. The outbox approach makes the pending obligation durable on the specified bulk path, while accepting index lag.

The inspected snapshot exposes the storage lanes, modular architecture, lifecycle state model and recoverable task machinery. Selected unit tests cover outbox rollback contracts and routing/lifecycle behavior. The atomicity tests use test seams and mocks; they are not a production or real-database rollback demonstration.

A published DuckDB case study illustrates querying supported geospatial files in place. It provides a complementary access pattern rather than evidence that all formats, all drivers or all queries share the same semantics.

DynaStore and its institutional deployments are FAO work delivered with colleagues and collaborators. This case does not turn design capacity, code coverage or conformance declarations into observed operational outcomes.

Write durability, search freshness and read capability are separate contracts. Naming them makes an architecture easier to reason about and avoids describing a derived index as a second canonical database.

For further operational validation I would test real database rollback, worker death during index updates, delayed visibility changes and stale cleanup work. These are concrete verification targets beyond the source evidence inspected here.