ADR-PROTO-004: Data dictionary in prompt context, not vector DB¶
Status: Accepted Source: prd-v2.1.md §C.8
Context¶
The original design included a vector DB (Pinecone/Qdrant) ingested with the data dictionary for RAG. Setup time for vector DB + ingestion + retrieval tuning is roughly 1 day by itself.
Decision¶
Load the entire data dictionary into a Python module, cache in memory at boot, inject the relevant subset into the Bedrock prompt context per request. No vector DB. The full Asurion data dictionary fits comfortably in Claude's 200K context window.
The dictionary lives in data-dictionary/ (4 CSVs + ai_query_guidelines.md) and is loaded by backend/app/sql_gen/dictionary_loader.py. DataDictionary.subset_for_tables(...) returns a scoped slice for any given metric. (Note: the original prd-v2.1.md text said "parsed from existing S3 schema files"; the implementation pivoted to local CSVs for the prototype — see docs/plans/active/prompt-2-dictionary-loader.md execution log.)
Consequences¶
- ~1 day saved.
- When the dictionary outgrows context (post-prototype, many use cases later), vector DB becomes a scale concern with its own ADR.
- Per-request token cost is higher than RAG would be — acceptable for prototype volume.
- Schema drift between the dictionary and Databricks is a real risk;
backend/scripts/validate_dictionary.pyreconciles them.
Cross-references¶
- ADR-PROTO-002 — the SQL generation path that consumes the dictionary.
- docs/data-dictionary.md — authoritative for the dictionary's content + workflows.
- docs/sql-generator.md — how the loader feeds the generator.
- Implementation: backend/app/sql_gen/dictionary_loader.py.