Avro schema versioning and evolution

avro-versioning.md

Avro schema evolution and management

See schema evolution and compatibility types for more details on different Avro compatibility types.

• Proposed to

  • use FORWARD compatibility by default

    • should cover 80%+ of changes

    • allows producer to add new required fields and start producing messages in the new format right away

    • optional fields can be added or deleted

    • consumers using previous schema can consume new messages w/out having access to added fields

    • consumers eventually migrate to use new schema on their own time line

    • message replay and retention policy

      • with a FORWARD compatible change (i.e. new required field added), a topic will contain messages with new and previous avro schema versions up to the retention period

      • once a consumer updates their schema version to v2 (with new required field added) they won't be able to "replay" old messages published with version v1 (w/out new required field)

      • to guarantee an opportunity to replay old messages for a consumer,  its v2 schema deployment should be done after old messages expire (retention period ends)

  • use BACKWARD compatibility periodically for schema cleanup

    • should be used less frequently (<20%) for removing obsolete fields that are no longer in use

    • optional fields can be added or deleted

    • requires all consumers to use new schema with required fields removed before producer can start publishing messages in new format

    • once all consumers migrated to use new schema, producer can start publishing messages in new format

    • field rename / type change should result in two changes:

      • add new field with a new name (FORWARD compatibility change)

      • delete old field (BACKWARD compatibility change)

      • there seems to be an alternative using BACKWARD compatibility mode

        • add a new field with a new type and a default value - v2

        • delete original field - v3

        • add original field back with new type and a default value - v4

        • delete a new field - v5

        • remove default value from original field - v6

    • consider using versioned / different topic for "very significant" changes

      • should producer continue to produce the same message in the old topic and new topic for a period of time?

• Outstanding questions

  • consumers that use specific Kafka connectors

    • How can they handle topic change?

    • How can they handle forward -> backward -> forward compatibility transition?