Case study · Architect · 45 min

Enterprise RAG over Confluence + SharePoint

Design a permission-aware RAG system that answers employee questions from Confluence and SharePoint without leaking restricted documents.

RAGACLembeddings

Interview opener

Design permission-aware Q&A over Confluence + SharePoint.

1.ACL groups change hourly — how stale can filters be?
2.A citation points to a doc the user can't open — what went wrong?
3.How do you separate retrieval misses from generation hallucinations?

Strong signals

ACL filter at retrieval with identity context
Hybrid + rerank + citation contract
Golden eval with groundedness gate
Ingest vs query path drawn separately

Hit 0/4

Weak signals (avoid)

Security by obscurity on embeddings
No mention of chunk metadata or ACL
Jumping to multi-agent before baseline RAG

Requirements

Internal knowledge Q&A for 25,000 employees. Answers must cite sources and respect per-document ACLs from the source systems. Latency and hallucination risk dominate the interview discussion.

Functional: natural-language Q&A with citations; filter by space/site; feedback thumbs
Non-functional: p95 end-to-end < 4s; answer groundedness > 90% on eval set
Security: never return chunks the user cannot read in Confluence/SharePoint
Scale: ~8,000 queries/day; corpus ~2M pages, ~12M chunks after split
Out of scope (MVP): multi-hop agents, write-back to wiki, image OCR

Clarify ACL model early

Ask whether ACLs are group-based, nested, and how fast group membership changes. Design for ACL filter-at-query-time, not security-by-obscurity on embeddings.

Back-of-envelope Estimation

Metric	Calculation	Result
Avg QPS	8,000 / 86,400	~0.09/sec
Peak QPS	20× average (standup hour)	~2/sec
Embed corpus	12M chunks × 400 tokens	~4.8B tokens
Embed cost (once)	4.8B × $0.02 / 1M	~$96
Query tokens/day	8K × (2K in + 500 out)	~20M tokens
LLM cost/day	20M × $3 / 1M (mid-tier)	~$60/day
Vector storage	12M × 1536-dim × 4 B	~74 GB

At this QPS, a single retrieval replica and a modest LLM quota suffice. Cost is dominated by generation, not embeddings — prioritize cache and smaller models for FAQ-like queries.

API Design

Endpoint	Method	Description
POST /v1/ask	POST	Body: { query, filters? } → streamed answer + citations[]
POST /v1/feedback	POST	{ messageId, rating, comment? } for offline eval mining
POST /v1/ingest/webhook	POST	Confluence/SharePoint change events → reindex queue
GET /v1/sources/{id}	GET	Resolve citation to source URL if caller has ACL

Pass identity, not trust

Forward SSO subject + groups to the retrieval service. Never let the client specify allowed_doc_ids.

Data Model

Entity	Key fields	Notes
documents	doc_id, source, url, title, updated_at, acl_version	Canonical metadata; soft-delete on source delete
chunks	chunk_id, doc_id, text, token_count, embedding_id	512–800 token chunks with 10–15% overlap
acl_principals	doc_id, principal_type, principal_id	Denormalized allow-list for filter predicates
query_logs	query_id, user_id, retrieved_ids, answer, latency_ms	PII-redacted; retention 90 days

Store vectors in a managed ANN index with metadata filters on doc_id and principal intersection. Keep raw text in object storage or Postgres for citation snippets.

High-level Design

Query + ingest paths

Ingest: Confluence/SP webhooks → Queue → Chunker → Embed → Vector DB
                                              └─► ACL sync worker → acl_principals

Query:  Client (SSO) → API GW → Query svc
                              ├─► Retriever (ANN + ACL filter) → top-k chunks
                              ├─► Reranker (optional cross-encoder)
                              └─► LLM (grounded prompt) → stream + citations

Separate ingest from query. On each ask, expand user groups, retrieve with filter principals ∩ user_groups nonempty, then generate with strict cite-or-abstain instructions.

Deep Dive: ACL-Safe Retrieval

Sync ACLs on every document update; bump acl_version and invalidate stale cached answers
Prefer pre-filter in the vector DB over post-filter (post-filter can return empty top-k under sparse grants)
Hybrid search: BM25 + dense; fuse with RRF before ACL-aware rerank
Abstain when max retrieval score < threshold or no ACL-visible hits

Cached answers

Semantic caches must key on (query embedding, user ACL fingerprint). Sharing cache entries across users is an ACL bug.

Bottlenecks at Scale

Full re-embed on model upgrade — shard by doc_id, dual-write embeddings, cut over
ACL fan-out for nested groups — cache group → members with short TTL; invalidate on IdP webhook
SharePoint crawl rate limits — prioritize changed pages via delta APIs
Reranker GPU saturation at peak — batch or skip rerank for FAQ cluster hits

Failure Modes & Monitoring

SLO	Target	Alert
p95 latency	< 4s	Page if > 6s for 10 min
Groundedness (sampled)	> 90%	Warn if weekly eval drops > 5 pts
ACL deny false-negatives	0 in red-team suite	Page on any fail
Ingest lag p95	< 15 min	Warn if > 1 hour

Dashboards: retrieval hit rate, abstain rate, citation click-through, cost/query
Canary new chunking/embed models on 5% traffic with side-by-side eval

AI & FDE Interview Room