Case study · Architect · 40 min

Multimodal Document Understanding

Extract structured answers from PDFs with tables, charts, and scanned pages using OCR plus multimodal models.

visionOCRRAG

Interview opener

Ingest PDFs with tables and diagrams for Q&A.

1.Table cell answers are wrong — which pipeline stage failed?
2.OCR garbage on scans — degrade how?
3.Do you embed images, captions, or both?

Strong signals

Layout-aware parsing before chunking
Separate text vs vision paths with confidence
Eval set that includes table questions

Hit 0/3

Weak signals (avoid)

Naive text extract only
One embedding space for everything without thought

Requirements

Functional: upload PDF; ask questions; extract tables to CSV; cite page numbers
Non-functional: ingest 50-page PDF < 90s; Q&A p95 < 8s
Accuracy: table cell F1 ≥ 0.85 on finance corpus; OCR CER < 2% on clean scans
Scale: 5,000 docs/day; avg 20 pages; 20K questions/day

Back-of-envelope Estimation

Metric	Calculation	Result
Pages/day	5,000 × 20	100,000
Vision tokens/page	~1,000 (tile encoding)	1K
Ingest vision tokens	100K × 1K	100M/day
Vision cost @ $3/1M	100M × $3/1M	~$300/day
OCR CPU hours	100K pages × 0.5s	~14 CPU-hours
Q&A cost	20K × 4K tokens × $2/1M	~$160/day

API Design

Endpoint	Method	Description
POST /v1/documents	POST	Upload PDF → job_id
GET /v1/documents/{id}	GET	Status, page count, parse quality
POST /v1/documents/{id}/ask	POST	Question → answer + page citations
GET /v1/documents/{id}/tables/{tid}	GET	CSV/JSON table export

Data Model

Entity	Key fields	Notes
documents	doc_id, pages, parse_status, storage_uri	Object store for originals
pages	page_id, doc_id, text, layout_json, image_uri	Layout: blocks, tables, figures
tables	table_id, page_id, cells[][]	Normalized grid
chunks	chunk_id, page_id, modality, embedding	Text + caption embeddings

High-level Design

Ingest pipeline

PDF → Object store → Page splitter
         ├─► Digital text layer? ──yes──► Layout parser
         │                         no──► OCR (+ deskew)
         ├─► Table detector → structure model → cells
         ├─► Figure captioner (VLM) → text captions
         └─► Chunk + embed → Vector index

Ask: query → retrieve pages/chunks → VLM or text LLM → cite pages

Deep Dive: When to Use VLM vs OCR

Native PDF text: skip OCR; use layout-aware chunking
Scanned docs: OCR first; use VLM only for charts/figures
Complex tables: dedicated structure model cheaper than full-page VLM
Hybrid Q&A: retrieve text first; escalate page image to VLM if confidence low

Bottlenecks at Scale

VLM GPU queue — route charts-only pages; batch tiles
Large scanned PDFs — parallelize per page with concurrency caps
Storage of page images — compress; lifecycle policy 30/90 days

Failure Modes & Monitoring

SLO	Target	Alert
Ingest p95	< 90s for 50 pages	Warn if > 3 min
Table F1 (weekly)	≥ 0.85	Warn on drop
Q&A p95	< 8s	Warn if > 12s
Parse failure rate	< 1%	Page on spike

AI & FDE Interview Room