MASTERCLASS

Gold-standard deep dive

URL Shortener — Masterclass

Three additional artifacts a Staff/Principal candidate should be able to produce for this problem: an Architecture Decision Record, a business-driven design exercise, and a production incident scenario.

1. Architecture Decision Record

The format working architects use to document a decision so future teams understand context, options, and reversal conditions.

ADR 001

Architecture Decision Record

MySQL over DynamoDB for URL Shortener at L4/L5 (10K-100K RPS)

Context

We are building a URL shortener MVP that must handle 10K RPS at launch and scale to 100K RPS within 6 months. The team has 3 backend engineers, none with prior DynamoDB production experience. AWS is our cloud. Budget is <$10K/month for infrastructure. We must ship in 8 weeks.

Constraints

Team size: 3 backend engineers, none with DynamoDB experience
Timeline: MVP in 8 weeks
Budget: <$10K/month infra
Availability: 99.9% (~9 hours/year downtime)
Latency: p99 redirect < 100ms
Anticipated scale: 10K RPS launch → 100K RPS in 6 months
Analytics: click counts, no real-time requirement
Public URL shortener, so abuse prevention required

Options considered

MySQL (RDS Multi-AZ)

Chosen

Pros

Team has 5+ years combined experience
Ad-hoc SQL for analytics without engineering effort
Familiar migration and backup story
$170/mo for db.m6i.large Multi-AZ handles 10K RPS today, upgrades to db.r6i.4xlarge (~$1600/mo) at 100K
Adding read replicas is one API call
Rich ecosystem: ProxySQL, ProxySQL, Percona Toolkit, MySQL Shell.

Cons

Single-primary write bottleneck — needs sharding or Aurora at 500K+ RPS (not in scope)
Vertical scale ceiling ~1M QPS on read replicas

Cost: $170/mo (L4) → $1600/mo (L5) — well under budget

DynamoDB

Pros

Infinite horizontal scale — never touch a database again
Single-digit ms latency at any scale
Fully managed — zero ops burden
Multi-region replication built-in (Global Tables)

Cons

Team learning curve is 2-4 weeks minimum (composite keys, single-table design)
$$$ at low RPS: on-demand mode at 10K RPS = ~$1500/mo
No ad-hoc SQL — every analytics query needs to be pre-designed as GSI or streamed to Athena
Migration to different DB later is a multi-quarter project

Cost: $1500/mo (L4) → $8000/mo (L5) — 10x more expensive at MVP scale

MongoDB Atlas

Pros

Familiar-looking to some engineers
Auto-scaling
Managed

Cons

No team experience with production Mongo
Weaker transactional guarantees vs MySQL
Analytics with aggregation pipeline is a learning curve
Doesn't clearly solve any problem MySQL doesn't already solve

Cost: $500-2000/mo depending on tier

Redis + async MySQL write

Pros

Sub-ms redirect latency
No DB on hot path

Cons

Redis is not durable for source-of-truth data
Race conditions between cache and durable write
Complex operational story for an MVP

Cost: $400/mo Redis + $170/mo MySQL

Chosen solution

MySQL (RDS Multi-AZ)

Why

MySQL is the only option that satisfies ALL constraints: team can ship in 8 weeks (no learning curve), $170/mo fits budget with 10x headroom, 10K RPS is trivial for a well-tuned MySQL, and ad-hoc analytics via SQL is free. DynamoDB is 10x more expensive at MVP scale AND requires 2-4 weeks of team ramp-up we don't have. The 'infinite scale' argument doesn't apply — we don't need infinite scale for 100K RPS. MySQL will scale to 500K RPS with read replicas + ProxySQL.

Rejected alternatives (with reasons)

DynamoDB — 10x cost at MVP scale and team learning curve blocks the 8-week deadline
MongoDB — doesn't solve a problem MySQL doesn't; adds unfamiliar tech risk
Redis + async — introduces data-loss risk on primary write path; wrong tool for durability

Trade-offs accepted

Accept single-primary write bottleneck. Solve later with Vitess-like sharding if we exceed 500K writes/sec.
Accept manual failover coordination during Multi-AZ promotion (30-90s downtime once/year historical average)
Accept that migrating to distributed DB (Spanner, Cockroach) is a future project if truly global scale becomes necessary

Consequences

MVP ships on time on infrastructure the team can debug at 3am
Add read replicas at 100K RPS (~2 hours of work), Aurora migration at 500K RPS (~1 month of work)
Total cost through year 1: <$25K, well under $120K budget
Gained option: rich SQL analytics without any additional engineering effort

When would we reverse this decision?

Sustained write throughput exceeds 500K/sec — consider Aurora Serverless v2 or Vitess
Cross-region strong consistency requirement — consider Spanner
Team gains DynamoDB expertise AND workload pattern proves single-key access — consider migration

2. Business constraint exercise

Given real-world constraints (team size, budget, deadline), what architecture do you propose — and how do you push back when leadership asks for the wrong thing? This teaches engineering judgment.

Business constraint exercise

You've just joined as the tech lead for a URL shortener startup. The CEO wants to launch in 8 weeks to piggyback off a marketing campaign. You have 3 engineers, $10K/month burn on infrastructure, and the executive team is worried about 'what if we go viral.' You're picking the architecture.

Constraints

18-week deadline (blocked partnership deal on the marketing campaign date)
23 backend engineers total (2 with 5+ years, 1 junior)
3$10K/mo max burn on infrastructure
4SLA promised to partners: 99.9% availability
5The CEO read a blog post about 'the Netflix stack' and wants microservices from day 1
6You cannot hire more engineers for at least 6 months (runway constraint)

Your question

What architecture do you propose, and how do you push back on the microservices ask? Be specific about tech choices (LB, DB, cache, deployment).

3. Production incident scenario

You are on-call at 3:47am. p99 has spiked. Walk through the investigation, hypothesis, mitigation, and postmortem. This teaches real production reasoning — not just design.

INCIDENT

URL Shortener p99 latency spiked from 50ms to 8 seconds

PagerDuty alert at 3:47am. Redirect endpoint (/r/{code}) p99 latency has spiked from a baseline of 50ms to 8 seconds. Traffic is normal (1M req/hr, in line with the last week). Error rate is at 0% — nothing is failing, just slow. Users are complaining on Twitter. You are the on-call.

Metrics

app.redirect.latency.p99: 8.2s (was 50ms) — spike started 3:15am
app.redirect.latency.p50: 45ms (normal)
app.request_rate: 285 req/sec (normal for this hour)
app.error_rate: 0.02% (normal)
db.MySQL.cpu: 12% (normal)
db.MySQL.active_connections: 45/200 (normal)
db.MySQL.slow_queries: 2/sec (normal)
cache.redis.hit_rate: 42% (was 95%!) — spike started 3:15am
cache.redis.cpu: 89% (was 15%)
cache.redis.evictions_per_sec: 12000 (was ~0)
cache.redis.memory_used: 30GB / 32GB (was 22GB)

Logs

3:14am — normal traffic pattern, no errors
3:15am — deployment event: '`redis-config-v42`: memory limit lowered from 64GB to 32GB'
3:16am — Redis eviction rate jumps from 0 to 12K/sec
3:17am — Cache hit rate collapses from 95% to 42%
3:47am — PagerDuty alert fires (p99 threshold)

Traces

Trace of a slow request:
→ LB: 2ms
→ App server: check Redis (cache-aside pattern)
→ Redis GET user:abc123: MISS (7ms — normally 1ms)
→ App server: fetch from MySQL
→ MySQL SELECT: 65ms
→ App server: write to Redis SET: 15ms (normally 1ms)
→ Response: 82ms typical when cache misses
→ But queue at Redis is 200 requests deep, adding ~7 seconds wait

Dependency health

MySQL primary: healthy (CPU 12%)
MySQL read replica: healthy (CPU 8%)
Redis: DEGRADED (CPU 89%, eviction spike)
ALB: healthy
External DNS: healthy

Your investigation

You look at the metrics dashboard. Which single metric is telling you the most useful thing right now?

Hint: Look at everything that changed at 3:15am. What's a huge change vs normal?

You correlate the cache hit rate drop with the deployment event at 3:15am — the Redis config change lowered memory from 64GB to 32GB. What's your working hypothesis?

Hint: Redis was using 22GB. New limit is 32GB. Where's the eviction storm coming from?

What do you do RIGHT NOW to mitigate? You need to decide in 60 seconds — user pain is climbing.

Hint: You have two levers: fix Redis, or reduce the load on it. Which is faster to execute at 4am?

After rollback, cache recovers in 5 minutes. What's your postmortem action item to prevent this?

Hint: The proximate cause was the config change. The root cause was...?

You get out of the incident. In the retro, someone says 'we should have caught this in staging.' Would staging have caught it? Why or why not?

Hint: Staging usually has 10-100x less traffic than production. What does that mean for cache behavior?

Knowledge graph

Learn these first

URL Shortener L4 design (foundation)
Cache-aside pattern
Postgres + Redis operational basics

Where this appears in the curriculum

This is the Gold Standard.

Every other system will eventually have a masterclass tab like this one. The pattern proven here — ADR + business exercise + incident scenario — scales to all 50+ problems on the platform.