Logsmith: an append-only event log
A small Go event log with durable appends, recovery, consumer offsets, and measured performance.
Build a small append-only event log in Go. Learn how sequential writes, durable commits, and consumer offsets behave when a process stops unexpectedly. This project is a learning system, not a Kafka replacement.
Focus: file I/O, recovery, concurrency, and delivery semantics. Plan about 55 to 65 hours across global weeks 8 to 13. The milestones below use project weeks 1 to 6.
Use the project reading in the calendar
Read the linked sources in the calendar reading slots. Do not add a separate reading block before the build.
- Apache Kafka: Design documentation: the log, partitions, consumer groups and replication semantics in the authors' own words. Your spec is a re-derivation of this.
- Confluent: Kafka and the File System: the original essay on sequential I/O and page cache. Answer "why is Kafka fast" from here, not from a LinkedIn carousel.
- Confluent: Kafka Replication and Committed Messages: what "committed" means, before you attempt the stretch replication milestone.
- Page cache and zero-copy deep dive (AutoMQ): a clean article for your sendfile milestone. Pairs with the Kafka file-system essay.
The flow
Build spec
Scope
- Core: append records, recover after restart, read records, and persist consumer offsets.
- If time allows: retention, a small Go client, and benchmark comparisons.
- Cut first: compaction, zero-copy reads, and replication. These features add substantial edge cases and are not needed to explain the core design.
Start with one process and one disk. Describe how you would partition and replicate it in the design note; do not imply that the single-node demo provides those guarantees.
Log format
- Topic leads to partitions, which lead to segment files (
0000001.logplus a sparse.indexevery N bytes). - CRC32 checksum per record, so you can tell a torn write from a good one.
Writes and recovery
- Append path is write-ahead, with the fsync policy selectable from the CLI (
always | every_100ms | never). - Crash recovery: truncate the partial tail record, rebuild the index on boot, and log what was repaired.
- Optional design exercise: describe how leader epochs and fencing could prevent an old leader from accepting writes. Implement replication only if the core is complete.
Reads
- Reads use ordinary file reads first. Treat
sendfileas an optional comparison, and measure whether it helps this workload.
Consumer groups
- Consumer groups: assign partitions to consumers and persist committed offsets.
- Keep assignment deterministic in the first version. Add rebalance behavior only after restart recovery works.
Delivery semantics
- Toggle between
commit-then-process(at-least-once) andprocess-then-commit(at-most-once). - Tests demonstrate the exact failure of each mode.
Retention
- Optional retention by size or age.
- Compaction is a stretch feature. State its recovery and ordering limits if you implement it.
Client and benchmarks
- Thin Go client library plus CLI (
logsmith produce/consume/admin). - Benchmark harness: produce MB/s and msgs/s per fsync mode, p99 append latency, consume throughput. Document the disk type.
Milestones
Artifacts to publish
- README with the record format, recovery steps, and measured benchmark results.
- A short restart demonstration using a reproducible input.
- Optional post explaining the design trade-offs and test results.
Interview yield
Why Kafka is fast (sequential I/O, page cache, zero-copy) · partition ordering · at-least and at-most-once first-hand · retention and compaction · consumer rebalance trade-offs.