Case study · FDE · 35 min

Secure Data Connector with ACL Propagation

Build a connector that syncs enterprise documents and faithfully propagates source ACLs into the retrieval layer.

FDEACLconnectors

Interview opener

Promised APIs vanish; nested AD ACLs with no API.

1.What do you tell the steering committee Thursday?
2.Smallest trustworthy cohort you can still ship?
3.How do you avoid becoming the permanent integration team?

Strong signals

Transparent risk register
Parallel allowlist path
Re-scoped milestone 0 for ACL

Hit 0/3

Weak signals (avoid)

Hide the blocker
Promise full ACL anyway

Requirements

Sources: SharePoint Online + Box; delta sync; delete propagation
ACL: users + nested groups; external sharing flags; break-glass deny
Lag: p95 change-to-searchable < 20 min
Compliance: customer-managed keys; audit every ACL decision

Back-of-envelope Estimation

Metric	Calculation	Result
Corpus	3M files	3M
Daily churn	2%	60K files/day
ACL principals/file avg	8	24M rows
Sync workers	60K / (8h×3600) × 2s each	~4.2 CPU
IdP group members	15K groups × 50	750K edges

API Design

Endpoint	Method	Description
POST /v1/connectors	POST	Register OAuth app + site list
POST /v1/connectors/{id}/sync	POST	Force delta sync
GET /v1/acl/check	GET	{ user, docId } → allow/deny + reason

Data Model

Entity	Key fields	Notes
connector_state	cursor, last_success, error	Per source
documents	doc_id, source_id, etag, deleted	Tombstones retained 30d
acl_entries	doc_id, principal, perm	Replace-on-write per doc
group_expand	group_id, member_id	Refreshed from IdP

High-level Design

Connector architecture

Source APIs → Auth vault → Delta crawler → Parse/chunk
                              └─► ACL extractor → acl_entries
IdP SCIM/webhooks → group_expand cache
Retriever → expand user groups → filter docs by acl_entries
Deletes → tombstone → remove vectors within 5 min

Deep Dive: Nested Groups

Expand groups offline into member sets; cap recursion depth
On IdP change, recompute affected users' ACL fingerprints
Deny by default if ACL sync stale beyond SLO
Red-team: user removed from group must lose access within 15 min

Bottlenecks at Scale

API throttling — adaptive backoff + priority for ACL-only updates
ACL write amplification — batch replaces; partition by doc_id
Group storms — debounce IdP webhooks 30s

Failure Modes & Monitoring

SLO	Target	Alert
Sync lag p95	< 20 min	Page if > 1h
ACL red-team	100% pass	Block release
Orphan vectors	0	Nightly reconcile
Auth token refresh	99.99%	Page on fail

AI & FDE Interview Room