System Design Study

• Gaurav Sen (YouTube) — best for fundamentals: sharding, consistent hashing, CAP, load balancing, caching. Start here.
• ByteByteGo (YouTube) — Alex Xu; best visual system breakdowns. 5–15 min animated videos.
• system-design-primer (GitHub) — the canonical free written reference; skim the "study guide" + the worked examples.
• Hello Interview — free written design walkthroughs structured the way interviews actually go.
• Book (if you want depth): Alex Xu, System Design Interview Vol 1 & 2.

• RAG: orq.ai — RAG Architecture Explained · n8n — RAG System Architecture (production) · Pinecone Learn (vector search + RAG fundamentals).
• LLM serving: Ubicloud — Life of an inference request (vLLM) (excellent end-to-end) · Nebius — Serving LLMs with vLLM.

• Functional ("what it does") vs non-functional ("how well"): scale/QPS, latency, availability, consistency, durability, cost. Pin these numbers before designing.
• Back-of-envelope: estimate QPS (DAU × actions ÷ 86,400), storage (records × size × retention), bandwidth, memory. Round aggressively; show the reasoning.
• Availability: "nines" — 99.9% ≈ 8.7h/yr down, 99.99% ≈ 52 min/yr. Latency: reason in p50/p95/p99, not averages.

• Vertical (bigger box, simple, capped) vs horizontal (more boxes, needs statelessness + LB). Default to horizontal for scale.
• Load balancer: L4 (transport, fast) vs L7 (application, content-aware routing). Algorithms: round-robin, least-connections, consistent hashing (minimizes reshuffling on node add/remove — key for caches & sharding).
• Stateless services scale trivially; push state to data stores / caches so any replica can serve any request.

• Where: client → CDN → API gateway → application (Redis/Memcached) → database. Cache closest to the reader that's still correct.
• Strategies: cache-aside (lazy, most common), write-through (consistent, slower writes), write-back (fast writes, durability risk).
• Eviction: LRU / LFU / TTL. Invalidation is the hard part — stale data vs thundering herd on expiry.
• "There are only two hard things: cache invalidation and naming things."

Indexing trades write speed + storage for read speed. Sharding/partitioning: split by a key (user_id, geo) — watch for hotspots from a bad key. Replication: leader-follower (read scaling), multi-leader / quorum (availability) — read replicas can serve stale data.

• CAP: under a network partition you choose Consistency or Availability. Most large web systems pick AP + eventual consistency; finance picks CP.
• Strong (every read sees the latest write) vs eventual (converges over time). Variants: read-after-write, monotonic reads.
• ACID (transactional DBs) vs BASE (Basically Available, Soft state, Eventual consistency — NoSQL).

• Message queue (SQS, RabbitMQ) / log (Kafka): decouple producers from consumers, absorb spikes, enable retries. Smooths bursty load.
• Pub/sub for fan-out to many consumers. Idempotency + dedup are mandatory (at-least-once delivery).
• Use async for anything slow or spiky: notifications, media processing, analytics, LLM batch jobs. Backpressure when consumers fall behind.

Two pipelines: an offline ingestion pipeline and an online query pipeline. Query flow: query → retrieve → assemble context → generate.

Ingestion

• Chunking (biggest lever on quality): size + overlap; semantic vs fixed-size; respect document structure. Bad chunking is the #1 cause of weak RAG.
• Embedding model choice; embed chunks → store vectors + metadata in a vector DB (HNSW / IVF index).

Retrieval

• Dense (semantic, embeddings) + sparse (keyword, BM25) → hybrid search for robustness.
• Re-ranking (cross-encoder) on the top-k candidates sharply improves precision.
• Context management: fit the window, order matters ("lost in the middle"), dedup, cite sources.

Eval & failure modes

• Measure retrieval (recall@k, precision) separately from generation (faithfulness/groundedness, answer relevance).
• Common failures: bad chunk boundaries, weak embeddings, retrieval misses, hallucination despite good context.
• RAG vs fine-tuning vs long-context: RAG for fresh/proprietary/citable knowledge; fine-tune for behavior/format/domain style; long-context when the corpus is small.

• Two metrics: TTFT (time-to-first-token, dominated by prefill) and TPOT/ITL (per-token, dominated by decode). Streaming hides TTFT from users.
• Latency vs throughput is the central tradeoff; batching trades one for the other.
• Continuous (in-flight) batching: add/remove requests every step instead of waiting for a fixed batch — big GPU-utilization win (≈5× throughput).
• KV cache + PagedAttention: cache attention keys/values to avoid recompute; page it like OS virtual memory to kill fragmentation (60–80% memory reclaimed).
• Prompt/prefix caching: reuse the KV for shared prompt prefixes (system prompts, few-shot) → big latency cut. (This is exactly the prompt-prefix caching your LRU/trie work modeled.)
• Quantization (INT8/INT4, GPTQ/AWQ): smaller, faster, cheaper, slight quality cost. Speculative decoding: a small draft model proposes tokens a big model verifies.
• Serving stacks: vLLM, TGI, TensorRT-LLM. Autoscaling on GPU is slow + expensive — scale-to-zero rarely fits low-latency.

• ReAct loop: reason → act (call a tool) → observe → repeat until done. Tool/function calling is the action interface.
• Planning (decompose a goal into steps), memory (short-term scratchpad + long-term store, often a vector DB).
• Control: step/iteration caps to avoid infinite loops, termination conditions, fallbacks. Guardrails on tool use.
• Multi-agent orchestration (planner + workers) when one agent's context/skills aren't enough. Eval is hard — measure task success + trajectory quality.

Listed for awareness; we'll rehearse this live in the mock rounds near June 1.

Clarify requirements (functional + non-functional, scale numbers) → estimate (QPS/storage) → API sketch → high-level architecture → data model → deep-dive one or two components → bottlenecks & tradeoffs → scale story (10K → millions) with cost/latency.