ML System Design: The Framework

ML system design questions (“design a feed ranking system”, “design spam detection”, “design product search”) test whether you can turn a business problem into a working ML product, not just whether you know models. Use this framework.

flowchart LR
  F[1. Frame the problem] --> D[2. Data & labels] --> FE[3. Features] --> M[4. Model] --> E[5. Evaluation<br/>offline + online] --> S[6. Serving] --> MO[7. Monitoring & iteration]
  MO -.-> D

1. Frame the problem

• Business goal: “increase orders”, “reduce fraud losses”, “more watch time without clickbait”.
• ML task: what exactly is predicted? (Probability a user clicks this item; probability a transaction is fraud; relevance score of a document for a query.) Classification, regression, ranking or generation?
• Constraints: latency (e.g. under 100 ms per request), scale (QPS, number of items), cost, interpretability and regulatory needs, freshness (how fast new items must be handled).
• Is ML even needed? Sometimes rules or heuristics are a better first version, and they also serve as the baseline.

2. Data and labels

• Sources: logs (impressions, clicks, purchases), user and item metadata, text and images.
• Labels: implicit (clicks, watch time, add-to-cart) or explicit (ratings, human annotation, chargebacks for fraud). Think about label noise, delay (fraud labels arrive weeks later) and bias (users can only click what was shown, i.e. position bias and exposure bias).
• Sampling: class imbalance, negative sampling for recommendations, deduplication.
• Splits: by time (train on the past, test on the future) to mimic production.
• Privacy and consent: what you may store and use.

3. Features

User features (history, demographics, device), item features (category, price, embeddings), context features (time, location, query), and cross features (user × item affinity). Discuss how each is computed and how fresh it must be (batch daily vs real-time streaming), and use a feature store so training and serving share definitions and avoid training–serving skew.

4. Model

Start with a simple baseline (popularity, logistic regression, gradient-boosted trees), then justify complexity. Common patterns:

• Tabular prediction: gradient-boosted trees.
• Large-scale ranking: a two-stage system: candidate generation (cheap retrieval of hundreds of items from millions, e.g. two-tower embeddings + ANN) followed by ranking (a heavier model scoring the candidates), and possibly re-ranking for diversity and business rules.
• Text and images: pretrained transformers or CNNs, fine-tuned.

State the loss (cross-entropy for click probability, pairwise/listwise losses for ranking) and how you’d handle imbalance.

5. Evaluation

• Offline: AUC/PR-AUC, log loss, NDCG@k, recall@k, calibrated for the business decision, and broken down by segment (new vs returning users, regions, languages).
• Online: A/B test on the true goal (orders, revenue, retention), with guardrail metrics (latency, complaints, unsubscribe rate). Decide the sample size and duration beforehand.
• Human review for quality, fairness and safety where relevant.

6. Serving

• Batch predictions (precompute nightly, e.g. recommendations emails) vs online (real time at request time).
• Latency budget: retrieval + feature lookup + model inference within, say, 100 ms. Use caching, model compression (distillation, quantisation), and GPUs only where needed.
• Deployment: shadow mode (run alongside the old system without serving), canary rollout, easy rollback, and model versioning.

7. Monitoring and iteration

• Data drift (input distributions shift) and concept drift (the relationship changes; fraudsters adapt).
• Prediction distribution, feature freshness and missing-value rates, latency and errors.
• Feedback loops: the model’s own outputs shape future training data (it shows popular items, which get more clicks, which makes them look more popular). Add exploration and randomisation.
• Retraining cadence (scheduled or triggered by drift) with automatic evaluation gates.