Skip to content

ADR-V2-001: Hybrid Kafka + Lakehouse architecture

Status: Accepted (proposed v2 — proposes evolution of ADR-001; does NOT supersede ADR-008) Source: prd-v2.1.md §B.15

Context

v1 ADR-001 chose Postgres + Redis Docker Compose for hackathon speed and explicitly deferred the production architecture decision. v2 must commit to a production scale path.

Decision

Adopt a hybrid architecture:

  • Kafka (or MSK) for event transport.
  • Databricks (Q-v2-2 resolved) for analytical lakehouse and ML lifecycle.
  • Postgres + Redis retained for OLTP and hot-path serving — not deprecated. Remains the system of record for transactional state.

Consequences

  • Production scale path is no longer hand-waved.
  • Adds significant operational surface (Kafka, Databricks, Schema Registry).
  • Cost shape changes from "negligible Docker Compose" to ongoing managed-service spend (~$5K-$15K/month estimated; needs validation).
  • Two systems can disagree; reconciliation strategy is now mandatory.
  • The pitch line changes from "real-time event-driven" to "hybrid transactional + analytical, real-time on the hot path".

Cross-references

  • ADR-001 — proposes evolution of (does not supersede in the v1 hackathon scope).
  • ADR-PROTO-001 — Part C defers Kafka to architecture diagram only; this ADR is the production target Part C points at.
  • ADR-PROTO-005 — per-metric routing is the prototype slice that proves the data-binding piece of this hybrid architecture.
  • prd-v2.1.md §B — full v2 production specification.