Fleet packages specify mappings that are used to create index component templates. When a Fleet package is updated the component template is replaced, and a new backing index is created by rolling over the data stream. Field mapping changes are implemented this way because it is generally not possible to change the field mappings of an existing index after data has been indexed. So this results in a single data stream that is composed to backing indices that have different data types for the same field.
When a query is executed on the data stream it may span multiple backing indices, therefore we should avoid having mixed data types for a field that result in incompatibilities at query-time.
Examples of ways queries can break due to conflicting types
- Term query where the value is not a valid IP, but one of the backing indices uses
type: ip, (mixed ip and keyword types) - Term query where a boolean field where the value is not a valid boolean (true, false). (mixed boolean and keyword)
- Numeric aggregation on a field when some indices contain a non-numeric data type (e..g. keyword).
- Any existing query should execute without errors when both the old and new indices are included.
- Adding new fields, including multi-fields on an existing field, is safe.
- Kibana data views may show conflicts, but these should be benign warnings.
- Removing a field is breaking change.
- Replacing an existing field with an alias is safe assuming it points to a compatible type.
This is a strict view of the safe data types combinations. Any other combinations should be considered incompatible and therefore a breaking change. If the specific queries used with the indices are known, then it may be possible to demonstrate those data types are safe.
Data types within a field families are safe although the performance characteristics may differ:
- keyword family -
keyword,constant_keyword,wildcard - text family
- numeric -
byte,short,integer,long,float,scaled_float,double(not an official field family type)