Queues, databases and scale stories
Kafka docs, storage engines and p99 thinking for the P2 weeks, plus one scale story per week.
This page covers weeks 8 to 19, when you build Logsmith and start Shiplane. Read each post before the build week it is tagged to. The first eight are the core; the storage engine sources carry you to week 19.
What to do
Open the post for this week before you touch the code. Read it with the milestone in front of you, so the design question in the post is the question you are about to answer.
After each post, add one line to your project notes: the claim, a number, and what it changed in your design. That line is what you will say out loud in a design round.
The P2 stack
| # | Week | Post | Why it matters |
|---|---|---|---|
| 1 | 8 | Uber Go Style Guide | The readability bar your P2 code must survive. |
| 2 | 8 | Confluent: Kafka and the File System | The authors explain why Kafka is fast: sequential I/O and the page cache. |
| 3 | 9 | Apache Kafka design documentation | The log, partitions and consumer groups in the authors' own words. |
| 4 | 10 | RabbitMQ vs Kafka series, introduction (Jack Vanlightly) | Two architectures for the same idea. Sharpens your design defence. |
| 5 | 11 | Confluent: Kafka Replication and Committed Messages | What "committed" actually means, before the replication milestone. |
| 6 | 11 | AutoMQ: page cache and zero-copy in Kafka | Pairs with the file system essay for the sendfile milestone. |
| 7 | 12 | ByteByteGo: Scale from Zero to Millions of Users | The escalation ladder interviewers paraphrase all the time. |
| 8 | 13 | Pieter Levels of Scale: ScyllaDB, Database Performance at Scale (condensed) | p99 and latency arithmetic your benchmark tables must respect. |
Storage engines, weeks 14 to 19
Read one per week while you build Shiplane. These are the internals behind the databases your projects sit on.
| Source | What you get | When |
|---|---|---|
| PingCAP blog (TiDB) | Key-value storage internals, rotating. | 14 to 19 |
| PlanetScale blog (MySQL and Vitess) | Sharding and scale patterns from a real managed fleet. | 14 to 19 |
| Cockroach Labs blog | Distributed SQL internals, clearly written. | 14 to 19 |
| Flipkart: MySQL High Availability | India-scale database operations. Good before a DB round. | 16 |
| Why Flipkart chose TiDB to replace its large MySQL fleet (PingCAP) | The "why migrate" story behind a large Indian fleet. | 18 |
Rotating scale stories
- Uber Engineering: giant-scale stories every week. The curated Uber picks by topic are the collection-day shortcut.
- Netflix TechBlog: streaming scale and chaos culture. The origin post is on the observability page.
- Cloudflare blog: edge and networking internals, with a tag for engineering. The best free deep-dive feed here.
- AWS Architecture Blog: architecture patterns at cloud scale. One pick per week is enough.
- Dan Luu: contrarian, numbers-first essays.
- Marc Brooker's blog: short, precise notes on randomness and reliability from an AWS principal.
The habit that carries the level
The p99 habit
From week 13, end every reading note with one line: what would this cost at p99? The benchmark tables you ship live or die on that framing.
Common mistakes
- Reading Kafka docs once and assuming the design is settled. Re-read the replication page the week you implement it.
- Copying the ScyllaDB numbers into your own tables. Use the arithmetic, not the result.
- Collecting scale stories with no project attached. One story per week, applied to Shiplane.