CAP theorem: in a partition, a distributed system must choose Consistency (every read returns the latest write or errors) or Availability (every request gets a response, possibly stale). Partition tolerance is non-negotiable in real networks — so the tradeoff is really C vs A during failure.

PACELC extends CAP: else (normal operation, no partition), choose Latency vs Consistency. Dynamo-style systems pick EL (low latency, eventual consistency); Spanner picks EC (consistent but higher latency).

Consistency models define what guarantees readers observe about writes. Stronger guarantees simplify application logic but cost latency and availability. Weaker models scale better but require careful client handling.

• Leader-based replication enables strong consistency on primary
• Quorum reads/writes tune consistency vs latency in leaderless systems
• Version vectors and conflict-free replicated data types (CRDTs) for merge
• Session tokens route reads to replica that has seen user's writes

Message queues decouple producers from consumers, absorb traffic spikes, and isolate failures. Point-to-point queues deliver each message to one consumer (work distribution). Messages are deleted after acknowledgment.

API Server ──produce──▶ [ Queue ] ──consume──▶ Worker Pool
                              │
                         (buffers spikes;
                          retries on failure)

• Visibility timeout: message hidden while processing; extend if slow
• Dead-letter queue (DLQ) for messages that fail max retries
• Back-pressure: slow consumers cause queue depth to grow — monitor lag
• Ordering: single partition or FIFO queue when order matters

Publish/subscribe fan-out delivers one message to many subscribers. Log-based event streams (Kafka, Pulsar) retain ordered, replayable history in partitions — enabling event sourcing, stream processing, and multiple consumer groups at different offsets.

Producer → Topic (partitions P0, P1, P2)
              │
    ┌─────────┼─────────┐
    ▼         ▼         ▼
 Group A   Group B   Group C
(analytics)(indexer) (audit)
Each group reads all partitions independently

• Partition key determines ordering — same key → same partition
• Consumer group: one consumer per partition for parallel scale
• Replay: reset offset to reprocess history for new downstream
• Schema registry (Avro/Protobuf) for evolution without breaking consumers

Delivery semantics define whether messages can be lost or duplicated. Exactly-once is the holy grail but expensive; most production systems achieve 'effectively once' via at-least-once delivery plus idempotent consumers.

1. Producer acks: wait for leader (acks=1) or all replicas (acks=all)
2. Consumer commit offset after successful processing (not before)
3. Transactional producer + read-process-write in one Kafka transaction
4. Outbox pattern: write business row + event in same DB transaction

Idempotent operations produce the same result whether executed once or multiple times. Retries with exponential backoff and jitter are essential in distributed systems where timeouts don't distinguish failure from slow success.

• Exponential backoff: 1s, 2s, 4s, 8s… cap at max delay
• Full jitter: randomize delay to prevent synchronized retries
• Retry budget: limit total retries to avoid retry storms
• Retry only idempotent operations or those with idempotency keys

Rate limiting protects services from overload by throttling requests per user, IP, or API key. Circuit breakers stop calling a failing dependency, fail fast, and periodically probe for recovery — preventing cascade failures.

• Circuit states: Closed (normal) → Open (fail fast) → Half-open (probe)
• Open after N failures or error rate threshold in sliding window
• Bulkhead: isolate thread pools per dependency — one slow service can't starve others
• Return 429 Too Many Requests with Retry-After header

  CLOSED ──(failures > threshold)──▶ OPEN
    ▲                                  │
    │                                  │ (timeout expires)
    │                                  ▼
    └──(probe succeeds)──────── HALF-OPEN
         (probe fails → OPEN)

Consensus algorithms (Raft, Paxos) let distributed nodes agree on a single value or leader despite failures. Leader election ensures exactly one primary handles writes at a time, enabling strong consistency and coordinated decisions.

• Raft: follower timeout → candidate → request votes → leader appends log entries
• Quorum (majority) required for election and commit — tolerate f failures with 2f+1 nodes
• Split-brain prevented by requiring majority to hold leadership
• etcd/Consul provide distributed locks, service discovery, and config