5.5 Avro

Started 2009 as a Hadoop subproject, because Protocol Buffers was not a good fit for Hadoop's use cases.

Two schema languages: Avro IDL (for human editing) and a JSON-based one (machine-readable). Like protobuf, they specify only fields and types — no complex validation rules.

record Person {
    string               userName;
    union { null, long } favoriteNumber = null;
    array<string>        interests;
}

{ "type": "record", "name": "Person",
  "fields": [
    {"name": "userName",       "type": "string"},
    {"name": "favoriteNumber", "type": ["null", "long"], "default": null},
    {"name": "interests",      "type": {"type": "array", "items": "string"}}
  ] }

5.1 The encoding — 32 bytes, and NOTHING self-describing

Notice: the schema has NO TAG NUMBERS.

0c  M a r t i n        length 6 (varint), then UTF-8 bytes
02  f2 14              union branch 1 (long), then varint 1337
04  16 daydreaming  0e hacking  00    array: count, items…, terminator

TOTAL: 32 bytes — the most compact of all the encodings.

Nothing in the encoded data says the first field is a string — it could just as well be an integer. That is only workable because the reader always has a schema.

The encoding is simply VALUES CONCATENATED TOGETHER. To parse it, you go through the fields in the order they appear in the schema and use the schema to determine each field's datatype.

⇒ The binary data can be decoded correctly ONLY IF the reading code uses the EXACT SAME SCHEMA as the writing code. Any mismatch means incorrectly decoded data.

So how does Avro evolve at all? This is the clever part.

5.2 Writer's schema and reader's schema

Schema resolution — how differences are reconciled:

5.3 Evolution rules

THE RULE: you may add or remove only a field THAT HAS A DEFAULT VALUE.

• Add a field with a default → a reader on the new schema reading old data fills in the default. ✔
• Add a field without a default → new readers can't read data written by old writers → breaks BACKWARD compatibility. ✗
• Remove a field without a default → old readers can't read data written by new writers → breaks FORWARD compatibility. ✗

Nulls are explicit. In some languages null is an acceptable default for any variable — not in Avro. To allow null you must use a union type: union { null, long, string } field;. You can use null as a default only if it is the FIRST branch of the union. More verbose than nullable-by-default, but it helps prevent bugs by being explicit about what can and cannot be null.

Two asymmetric cases worth remembering:

5.4 But how does the reader know the writer's schema?

You can't include the whole schema with every record — it would likely be much bigger than the encoded data, negating all the space savings. Three answers depending on context:

A database of schema versions is useful in any case: it acts as documentation and gives you a chance to CHECK SCHEMA COMPATIBILITY. Version number = a simple incrementing integer or a hash of the schema.

5.5 Why "no tag numbers" matters: dynamically generated schemas

The scenario: dump a relational database's contents to a file in a binary format.