kibana_migrations.md

Migrations

Plugins

• Expose their mappings so that they are programmatically consumable and easily associated w/ the right plugin
• Expose a list of migrations, which are one of:
  • Transform: Migrate a document from one version to the next
  • Seed: Create a new document
• Old data which is imported will be run through the migrations before being persisted
  • This means exported data needs to contain information about the migration state of that data
• A plugin's migrations must not conflict with another plugin's migrations
  • Several plugins can can affect the same document
  • A plugin should only affect properties which they own now and indefinitely
  • Property ownership should never transfer from one plugin to another (see scenario property ownership)

Disabling a plugin

• Doesn't require a migration
  • The plugin's mappings still exist
  • The plugin's data still exists
  • Checksum / migration health needs to be intelligent enough to handle this
• The current migration-state of any disabled plugin's data is persisted across migrations
• Imports that have data which is associated with a disabled plugin
  • Will fail if that data is not the same version as the index's migration state
  • Can be done if the user specifies that the plugin's data can be dropped
  • To drop fields,
    • Exported doc will have basic mapping metadata which will indicate what properties were owned by what plugins at that time
    • Run docs through the migrations
    • Prior to save, drop any properties that are owned by the disabled plugin(s)

Removing a plugin

• The index is migrated
• The plugin's mappings no longer exist in the new index
• All of the plugin's data is removed in the new index

Not supported

• Splitting a document into many
• Merging multiple documents into one / fewer
• Guaranteed transfer of responsibility for data from one plugin to another, e.g.
  • PluginA and pluginB decide to merge functionality
  • PluginA decides to hand off some of its data to pluginB and no longer manage it
  • These scenarios can be done with the current migration system, if the system has both pluginA and pluginB
    • If a system only has pluginA or pluginB and not both, data loss will happen

Running migrations

• For the Kibana index, migrations will automatically run when Kibana starts
• Duplicate runs of a migration will be avoided using optimistic concurrency in setting an indexe's migration state to 'migrating'
• The existing index will
  • Have it's migration status set to 'migrating'
  • Be set to read-only
  • Be converted into an alias, if it is not already
• Migration will create a new index, and leave the old index's documents / state untouched (with the exceptions listed previously)

Scenarios

Scenario: unpredictable migration order

Migrations within a plugin are guaranteed to be applied sequentially, however they are not guaranteed to have a predictable global order.

For example:

• pluginA has migrations [100, 102, 104]
• pluginB has migrations [101, 103, 105]

System1

• System1 has only pluginA installed
• Migrations [100, 102, 104] are applied
• System1 later installs pluginB
• Migrations [101, 103, 105] are applied
• Global migration order is [100, 102, 104, 101, 103, 105]

System2

• System1 has only an old version of pluginB installed with migrations [101, 103]
• Migrations [101, 103] are applied
• System1 later installs an old version of pluginA with [100]
• Migrations [100] are applied
• System1 upgrades to the latest version of plugins A and B
• Migrations [102, 104, 105] are applied
• Global migration order is [101, 103, 100, 102, 104, 105]

This is why migrations between plugins must be guaranteed to never conflict. This guarantee is made by specifying that a plugin indefinitely owns the properties that it modifies.