Concurrency
Goroutines, races, locks, channels, cancellation and the race detector, with the Logsmith and Ledgerline examples.
Concurrency is how one program does many things at once without corrupting its own data. Interviewers ask because every backend you build is concurrent, and a vague answer is easy to spot. Here it shows up in Ledgerline's outbox worker and Logsmith's consumer groups.
Priority for SDE-1
Learn races, mutexes, channels, cancellation, worker pools and go test -race cold. Scheduler internals and the memory model can wait until after the offer.
SDE-1 scope vs SDE-2 scope
| Area | SDE-1 must | SDE-2 later |
|---|---|---|
| Race conditions | Spot one in a snippet, explain the fix | Reason from happens-before in the memory model |
| Sync tools | Mutex vs channel, and when each is right | Lock-free code built on compare-and-swap |
| Goroutines | Start, stop, and avoid leaks | Scheduler internals, G-M-P, work stealing |
| Cancellation | Carry context.Context through every layer | Propagation cost across many services |
| Pools | Worker pool with graceful shutdown | Contention profiling with pprof |
| Testing | go test -race in CI, plus its limits | Stress and fuzz harnesses for timing bugs |
Roadmap
Stage 1: how the pieces fit (week 5, before the drill)
Read the model in one sitting, then run the code. A process owns its memory, a thread shares the process heap, and a goroutine is a function the Go runtime schedules onto a thread.
Answer in that order, then name the race that shared memory creates. Everything else in this subject removes it.
Stage 2: interview core (weeks 5 to 7)
Write three small programs: fan-out and fan-in, a worker pool with graceful shutdown, and a rate-limited HTTP client. Each one is 40 lines and each one is a question you will be asked. The week 5 drill then tests all of it out loud.
Then break them on purpose. Removing a mutex and watching -race fire teaches more than reading about it.
Stage 3: SDE-2 depth (after the offer)
Read the memory model, the scheduler series and the contention profiles. This is the layer where you explain why 200 goroutines per request still fit in 50 MB.
Core topics, in order
| Topic | What you must be able to say | Level |
|---|---|---|
| Process, thread, goroutine | What each shares, and what a goroutine costs | SDE-1 must |
| Shared memory and data races | Two goroutines, one write, no sync | SDE-1 must |
| Mutexes and lock ordering | A critical section, and why ordering kills deadlock | SDE-1 must |
| Atomics | One counter, no invariant across fields | SDE-1 nice |
| Channels and CSP | Send a value, transfer ownership; CSP sends messages, not shared memory | SDE-1 must |
| The Go scheduler | G, M, P and GOMAXPROCS in one breath | SDE-2 later |
| Context cancellation | The first argument to a blocking call | SDE-1 must |
| Worker pools and backpressure | Bounded queue, N workers, a way to stop | SDE-1 must |
| Fan-out and fan-in | Many workers, one results channel, order lost | SDE-1 must |
| Deadlock, livelock, starvation | The four conditions, and why retries stall | SDE-1 must |
go test -race | What it proves and what it cannot | SDE-1 must |
| Async I/O | Blocking code in Go, an event loop in Python | SDE-1 nice |
What a race looks like (Ledgerline)
Two transfer requests read the same balance, both add 100, both write back. One update is lost, and the ledger no longer sums to zero.
balance := readBalance(id)
balance += amount
writeBalance(id, balance) // read, modify, write, no lockA data race is two goroutines touching the same memory, at least one writing, with no synchronization between them. In Ledgerline the fix is the transaction and the row lock you already use: SELECT ... FOR UPDATE on the account row, acquired in account ID order.
Say the failure in money terms: two transfers lost one update and the journal stopped balancing.
Mutex or channel
Use a mutex to protect state that several goroutines read and write, such as a map or a balance in memory. Use a channel to pass ownership of a value or to signal an event, such as a finished job or a shutdown.
The Go proverb is "share memory by communicating", and it is advice, not a law. A counter behind a sync.Mutex is easier to read than the same counter behind a channel. Logsmith's in-memory offset table is a mutex; the handoff to its consumer goroutines is a channel.
sync/atomic covers a single word, such as a request counter or a byte total. It does not make two fields agree, so a balance with a matching audit row still needs a mutex.
Cancellation and goroutine leaks
Every request carries a context.Context. It holds a deadline, a cancellation signal and request-scoped values, and it is the first argument in Go's standard style.
ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
defer cancel()A goroutine leak is a goroutine blocked forever on a send or receive that nobody will complete. It keeps its stack and its captured variables, so a leak grows with traffic until the process dies.
Every goroutine you start needs an exit: select on ctx.Done(), a closed input channel, or a finite loop. In Logsmith's consumer, shutdown cancels the context, drains the current batch, commits the last offset, then returns.
Worker pools and backpressure
A worker pool is N goroutines reading jobs from one channel. N is a decision, not a default: pick it from the bottleneck, which is usually the database, not the CPU.
jobs := make(chan Job, 100) // bounded on purpose
for i := 0; i < workers; i++ {
go func() { for j := range jobs { process(ctx, j) } }()
}Backpressure is what happens when the queue is full and the producer waits. A bounded channel gives you that for free. An unbounded queue turns a slow consumer into an out-of-memory kill.
In Ledgerline the pool is in Postgres: the outbox worker claims rows with FOR UPDATE SKIP LOCKED, so ten workers take ten different rows and none blocks. That pool survives a restart, which the in-memory one does not.
Fan-out, fan-in, and order
Fan-out sends one input to many workers. Fan-in merges their results into one channel, with a sync.WaitGroup that closes the output when the last worker finishes.
Fan-in loses order, because the fastest worker wins. If order matters, carry a sequence number and sort at the end. Logsmith has the same problem in another shape: a partition is ordered, a consumer group across partitions is not, and the offset is the sequence number.
Deadlock and livelock
Deadlock needs four conditions at once: mutual exclusion, hold and wait, no preemption, and circular wait. Break any one and it cannot happen. Ledgerline breaks circular wait by locking account IDs in ascending order.
Go prints fatal error: all goroutines are asleep - deadlock! when every goroutine is blocked. It covers channel deadlocks, not lock deadlocks, so a locked-up service can hang silently.
Livelock is different: every goroutine runs, but none makes progress, because each retries and yields to the other. Two workers retrying the same conflicting row is the common case. Jittered backoff and SKIP LOCKED fix it.
Testing with -race
The race detector instruments memory access at runtime and reports a race when two accesses actually conflict. Run it on the whole module.
go test -race ./...It only sees paths your tests execute, so coverage is the limit. It cannot prove a program race-free, and it does not find deadlocks or leaks.
For Ledgerline, add one test that fires the same idempotency key from ten goroutines and asserts one ledger row. For Logsmith, one test that commits offsets from two consumers at once.
Async I/O and the scheduler
In Go you write blocking code. The runtime's network poller parks a goroutine while its socket waits and reuses the OS thread, so a blocked read costs a goroutine, not a thread.
The Go scheduler has three parts. A G is a goroutine, an M is an OS thread, and a P is the right to run Go code, with a local run queue. GOMAXPROCS sets the number of Ps, usually the core count, and a goroutine blocked in a syscall gives its P to another M.
In Python, asyncio runs one thread and an event loop, and one blocking call inside a coroutine stalls the whole loop. That is why Python async stays a glue tool here. See Python reading.
Resources
Learn
- A Tour of Go: concurrency: goroutines and channels, runnable in the browser, 45 minutes. SDE-1 must.
- Go Concurrency Patterns: pipelines and cancellation: the fan-in and fan-out code for the drills. SDE-1 must.
- Go Concurrency Patterns: context: cancellation and request scope, from the authors. SDE-1 must.
- context package: the reference for
WithCancel,WithTimeoutandDone. SDE-1 must.
Practice
- Data Race Detector: how
-raceworks, what it costs, and what it cannot see. Read it before week 5. SDE-1 must. - Codewalk: share memory by communicating: one program, rewritten from mutexes to channels. SDE-1 nice.
Long-form
- The Go Memory Model: the happens-before rules behind every correct answer. SDE-2 later.
- Under the Hood: the G-M-P model: scheduler internals with diagrams. SDE-2 later.
- Scheduling in Go, part II: the long version of the scheduler paragraph. SDE-2 later.
Questions to answer out loud
- What does a goroutine share with the goroutine next to it, and what does it not share?
- Show a data race in six lines, then fix it two different ways.
- Mutex or channel for a shared counter? Defend the choice.
- What is a goroutine leak, and how does cancellation prevent one?
- Walk a worker pool from start to graceful shutdown, naming every channel close.
- What is backpressure, and what does an unbounded queue do instead?
- What does
go test -raceprove, and what does it miss?
SDE-2 questions
- Explain G-M-P, and say what happens when a goroutine blocks on a syscall.
- State two happens-before rules from the Go memory model.
- How would you profile lock contention in a hot path?
How this shows up in your projects
- Ledgerline's outbox worker is a worker pool over
FOR UPDATE SKIP LOCKED, with backoff and a bounded claim size. - Ledgerline's lock ordering by account ID is a deadlock answer you can point at in an ADR.
- Logsmith's consumer groups are fan-out with a commit rule:
commit-then-processis at-least-once,process-then-commitis at-most-once. - Logsmith's offset commits from several consumers are the race you test with
-racefirst.
Common mistakes
- Answer in this order: process, thread, goroutine, shared memory, race, fix.
- Give every goroutine an exit, and say what it is.
- Put a number on the pool size and name the bottleneck.