5. Encoding and Evolution
5. Encoding and Evolution
Chapter 5 of Designing Data-Intensive Applications — 15 sections.
"Everything changes and nothing stands still." — Heraclitus
The premise: applications change. A change to features usually requires a change to the data they store. But in a large application, code changes cannot happen instantaneously:
- Server-side: you want a rolling upgrade (staged rollout) — deploy to a few nodes at a time, monitor, work through all nodes. No downtime → more frequent releases → better evolvability.
- Client-side: you're at the mercy of the user, who may not install the update for a long time.
Therefore old and new versions of code, and old and new data formats, coexist in the system at the same time.
Sections
- 5.02The two compatibilities — memorize this
- 5.11Encoding and decodingPrograms work with data in (at least) two representations:
- 5.2Language-specific formats — and why not to use themjava.io.Serializable, Python pickle, Ruby Marshal, Kryo for Java.
- 5.32JSON, XML, CSV, and binary variantsWidely adopted: in OpenAPI specs, in schema registries (Confluent Schema Registry, Red Hat Apicurio), and in databases (PostgreSQL's pg_jsonschema, MongoDB's $jsonSchema validator…
- 5.42Protocol BuffersBinary encoding library from Google; similar to Apache Thrift (originally Facebook) — most of this section applies to Thrift too.
- 5.54AvroStarted 2009 as a Hadoop subproject, because Protocol Buffers was not a good fit for Hadoop's use cases.
- 5.6The merits of schemasProtocol Buffers' and Avro's schema languages are much simpler than XML Schema or JSON Schema, which support detailed validation rules ("must match this regex", "must be between 0…
- 5.75Modes of DataflowClients connect to servers; the API exposed by the server is a service.
- 5.8Deep dives2Technology deep divesgRPC adds: HTTP/2 transport (multiplexed streams, header compression), four call types (unary, server-streaming, client-streaming, bidirectional), deadlines propagated through the…
- 5.9Failure catalogProduction failure catalog for this chapter
- 5.10Decision sheetDecision cheat sheetAsk both directions separately. Can new code read old data? (backward) Can old code read new data? (forward) A change is only safe if the answer to both is yes for the duration of…
- 5.11Worked examplesWorked examplesJSON 81 B → MessagePack 66 B (−19%) → Protobuf 33 B (−59%) → Avro 32 B (−60%).
- 5.12Self-testSelf-test
- 5.13TerminologyTerminology introduced here
- 5.14Forward linksForward links