Case study · Architect · 40 min

IDE Code Assistant with Repo Context

Design an in-editor coding assistant that retrieves relevant repo context and completes or explains code under tight latency budgets.

codeRAGlatency

Interview opener

In-IDE coding assistant over a large private monorepo.

1.How do you avoid leaking secrets from the repo into prompts?
2.Repo is 40M LOC — what do you index vs fetch on demand?
3.How do you measure 'helpful' without gaming?

Strong signals

Symbol-aware retrieval + local context packing
Secret scanning / redaction before prompt
Acceptance rate + task success evals

Hit 0/3

Weak signals (avoid)

Dump whole files into context
No secret handling
Only vibe-based quality

Requirements

Functional: inline completion, chat explain/refactor, cite file:line
Non-functional: completion p95 < 400ms; chat p95 < 3s
Privacy: customer code never trains shared models; tenant isolation
Scale: 50K DAU; 30 completions/user/day; 2 chat turns/user/day
Context: monorepos up to 5M LOC; respect .gitignore and secret scanning

Back-of-envelope Estimation

Metric	Calculation	Result
Completions/day	50K × 30	1.5M
Completion QPS peak	1.5M / 8h / 3600 × 4×	~208/sec
Tokens/completion	~2K in + 80 out (fill-in-middle)	~2.1K
Completion tokens/day	1.5M × 2.1K	~3.15B
Cost if API @$0.5/1M	3.15B × $0.5/1M	~$1,575/day
Chat tokens/day	50K × 2 × 4K	~400M (~$200/day)

Completions dominate cost and latency — favor a small specialized model + local/prefix cache; reserve frontier models for chat.

API Design

Endpoint	Method	Description
POST /v1/complete	POST	{ prefix, suffix, path, language } → completion stream
POST /v1/chat	POST	{ messages, selections[], repoId } → answer + citations
POST /v1/index/sync	POST	Client reports file hashes; server embeds deltas

Data Model

Entity	Key fields	Notes
repos	repo_id, tenant_id, default_branch	Tenant-scoped
file_chunks	chunk_id, repo_id, path, start_line, embedding	Symbol-aware chunking preferred
symbols	symbol_id, repo_id, name, kind, path	For go-to-def style retrieval

High-level Design

Completion vs chat

IDE ─complete─► Edge GW ─► Prompt builder (FIM + local window)
                              └─► Small code model (GPU pool) ─► stream

IDE ─chat────► API ─► Repo retriever (symbols + embeddings)
                      └─► Rerank ─► Large model ─► citations

Indexer: git push / IDE sync → chunk + embed → per-tenant vector index

Deep Dive: Context Selection

Always include current file window (±200 lines) and open tabs heuristically
Retrieve by embedding of surrounding code + BM25 on identifiers
Prefer symbol graph neighbors over naive top-k similar comments
Strip secrets with regex + entropy detectors before leave-machine send

Latency budget

Budget 50ms retrieve, 20ms prompt build, 300ms model TTFT for completions — skip retrieval if local context is enough.

Bottlenecks at Scale

GPU saturation at peak — speculative decoding + continuous batching
Index freshness on large monorepos — incremental embed by content hash
Cross-tenant noise — hard partition indexes; never shared ANN collections
Chat context bloat — hard cap 32K tokens with priority schedule

Failure Modes & Monitoring

SLO	Target	Alert
Completion p95	< 400ms	Page if > 700ms
Accept rate	> 25%	Warn if weekly drop
Secret egress	0	Page on detector hit
Index lag	< 10 min	Warn if > 1 hour

AI & FDE Interview Room