Skip to content

Instantly share code, notes, and snippets.

@netanel246
Last active February 22, 2018 20:35
Show Gist options
  • Save netanel246/80e77507a12ee4a28d174dc4767271ea to your computer and use it in GitHub Desktop.
Save netanel246/80e77507a12ee4a28d174dc4767271ea to your computer and use it in GitHub Desktop.
elasticsearch shard sizing

elasticsearch shard sizing

Shard Size: Up to 50GB, and for use-cases with time-based data, it is common to see shards between 20GB and 40GB in size.

Number of Shards: Between 20 to 25 per GB heap. For example:
A node with a 30GB heap should therefore have a maximum of 600-750 shards, but the further below this limit you can keep it the better

Rollover index API: If you have time-based, immutable data where volumes can vary significantly over time, consider using the rollover index API to achieve an optimal target shard size by dynamically varying the time-period each index covers. This gives great flexibility and can help avoid having too large or too small shards when volumes are unpredictable.

Shrink index API: If you need to have each index cover a specific time period but still want to be able to spread indexing out across a large number of nodes, consider using the shrink API to reduce the number of primary shards once the index is no longer indexed into. This API can also be used to reduce the number of shards in case you have initially configured too many shards.

Based on this blog

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment