Case study · Both · 35 min
Migrating a Legacy Knowledge Base to RAG
Migrate a legacy FAQ/CMS knowledge base into a chunked, embedded RAG corpus with zero downtime and quality parity.
RAGmigrationFDE
Interview opener
Migrate a legacy KB into a RAG corpus without breaking trust.
- 1.Stale pages outnumber good ones — strategy?
- 2.How do you prove parity before cutover?
- 3.Rollback if retrieval quality tanks?
Strong signals
Hit 0/3
Weak signals (avoid)
- Embed everything indiscriminately
- Big-bang cutover with no shadow
Requirements
- Source: 80K CMS articles + 5K PDFs; mixed HTML quality
- Cutover: dual-run old keyword search + new RAG for 3 weeks
- Parity: ≥ 90% of top-200 support queries resolve as well or better
- Redirects: old URLs remain valid; citations map to canonical pages
Back-of-envelope Estimation
| Metric | Calculation | Result |
|---|---|---|
| Articles | 80K + 5K PDFs | 85K docs |
| Chunks | 85K × 4 | 340K |
| Embed tokens | 340K × 400 | 136M |
| Embed cost | 136M × $0.02/1M | ~$2.70 |
| Cleanup eng | HTML boilerplate strip | ~2 weeks |
| Eval labeling | 200 queries × 3 judges | ~3 days |
API Design
| Endpoint | Method | Description |
|---|---|---|
| POST /v1/migrate/jobs | POST | Start bulk crawl + transform |
| GET /v1/migrate/jobs/{id} | GET | Progress, error samples |
| POST /v1/search | POST | { mode: legacy|rag|shadow } for dual-run |
Data Model
| Entity | Key fields | Notes |
|---|---|---|
| source_articles | cms_id, html, updated_at | Raw snapshot |
| normalized_docs | doc_id, markdown, canonical_url | Cleaned |
| id_map | cms_id, doc_id, chunk_ids | Redirects |
| shadow_logs | query, legacy_rank, rag_rank | Compare |
High-level Design
Phase 1: crawl CMS → normalize HTML → store markdown Phase 2: chunk/embed → RAG index (not live) Phase 3: shadow mode — serve legacy, log RAG answers Phase 4: canary 10% RAG → compare CSAT/deflection Phase 5: 100% RAG; legacy read-only fallback 30 days Continuous: CMS webhook → incremental reindex
Deep Dive: Content Cleanup
- Strip nav/footer boilerplate or embeddings drown in chrome
- Split mega-FAQs into intent-sized articles before chunking
- Detect near-duplicate articles; pick canonical to avoid conflicting answers
- Preserve tables as markdown tables for better LLM fidelity
Bottlenecks at Scale
- Dirty HTML — invest in normalize before any model work
- Stakeholder fear of cutover — shadow metrics build trust
- Orphan media — host images or drop broken references
Failure Modes & Monitoring
| SLO | Target | Alert |
|---|---|---|
| Shadow win rate | ≥ 90% | Delay cutover |
| 404 after redirect | < 0.1% | Fix id_map |
| Ingest lag | < 30 min | Warn |
| CSAT delta | ≥ 0 | Rollback canary |