Skip to content

Instantly share code, notes, and snippets.

@lawrencejones

lawrencejones/README.md

Created March 25, 2020 13:57
Show Gist options
  • Save lawrencejones/e8b5f724939d27566bacc9d86b05742d to your computer and use it in GitHub Desktop.
Save lawrencejones/e8b5f724939d27566bacc9d86b05742d to your computer and use it in GitHub Desktop.
Download ZIP
Find indices that require forcemerging
Raw
README.md

Force-merge analysis

Find indicies that still require force-merging

Raw
main.rb
#!/usr/bin/env ruby
require "rubygems"
require "excon"
require "json"
require "pry"
resp = Excon.get("http://localhost:9200/_all/_segments")
segments = JSON.parse(resp.body)
rows = segments["indices"].map do |name, index|
next unless name.start_with?("logstash")
[
name,
index["shards"].values.flat_map(&:count).sum,
index["shards"].
flat_map { |shard_id, replicas| replicas }.
map { |replica| replica["segments"].count }.
max,
]
end
puts("index\tshard_count\tmax_segment_count")
puts(rows.compact.sort.map { |row| row.join("\t") }.join("\n"))
Raw
results.csv
We can make this file beautiful and searchable if this error is corrected: No commas found in this CSV file in line 0.
index shard_count max_segment_count
logstash-platform-2020.01.21 2 1
logstash-platform-2020.01.22 2 1
logstash-platform-2020.01.23 2 1
logstash-platform-2020.01.24 2 1
logstash-platform-2020.01.25 2 1
logstash-platform-2020.01.26 2 1
logstash-platform-2020.01.27 2 1
logstash-platform-2020.01.28 2 1
logstash-platform-2020.01.29 2 1
logstash-platform-2020.01.30 2 1
logstash-platform-2020.01.31 2 1
logstash-platform-2020.02.01 2 1
logstash-platform-2020.02.02 2 1
logstash-platform-2020.02.03 2 1
logstash-platform-2020.02.04 2 1
logstash-platform-2020.02.05 2 1
logstash-platform-2020.02.06 2 1
logstash-platform-2020.02.07 2 1
logstash-platform-2020.02.08 2 1
logstash-platform-2020.02.09 2 1
logstash-platform-2020.02.10 2 1
logstash-platform-2020.02.11 2 1
logstash-platform-2020.02.12 2 1
logstash-platform-2020.02.13 2 1
logstash-platform-2020.02.14 2 1
logstash-platform-2020.02.15 2 1
logstash-platform-2020.02.16 2 1
logstash-platform-2020.02.17 2 1
logstash-platform-2020.02.18 2 1
logstash-platform-2020.02.19 2 1
logstash-platform-2020.02.20 2 1
logstash-platform-2020.02.21 2 1
logstash-platform-2020.02.22 2 30
logstash-platform-2020.02.23 2 1
logstash-platform-2020.02.24 2 1
logstash-platform-2020.02.25 2 1
logstash-platform-2020.02.26 2 1
logstash-platform-2020.02.27 2 1
logstash-platform-2020.02.28 2 1
logstash-platform-2020.02.29 2 1
logstash-platform-2020.03.01 2 1
logstash-platform-2020.03.02 2 1
logstash-platform-2020.03.03 2 1
logstash-platform-2020.03.04 2 1
logstash-platform-2020.03.05 2 1
logstash-platform-2020.03.06 2 1
logstash-platform-2020.03.07 2 1
logstash-platform-2020.03.08 2 1
logstash-platform-2020.03.09 2 1
logstash-platform-2020.03.10 2 1
logstash-platform-2020.03.11 4 1
logstash-platform-2020.03.12 2 1
logstash-platform-2020.03.13 2 1
logstash-platform-2020.03.14 2 1
logstash-platform-2020.03.15 2 1
logstash-platform-2020.03.16 2 1
logstash-platform-2020.03.17 2 1
logstash-platform-2020.03.18 2 1
logstash-platform-2020.03.19 2 1
logstash-platform-2020.03.20 2 1
logstash-platform-2020.03.21 2 1
logstash-platform-2020.03.22 2 1
logstash-platform-2020.03.23 2 1
logstash-platform-2020.03.24 2 1
logstash-platform-2020.03.25 2 38
logstash-production-2020.01.21 10 35
logstash-production-2020.01.23 10 33
logstash-production-2020.01.24 10 35
logstash-production-2020.01.25 10 31
logstash-production-2020.01.26 10 29
logstash-production-2020.01.27 4 79
logstash-production-2020.01.28 10 36
logstash-production-2020.01.29 4 49
logstash-production-2020.01.30 4 85
logstash-production-2020.01.31 10 35
logstash-production-2020.02.01 10 1
logstash-production-2020.02.02 4 39
logstash-production-2020.02.03 10 37
logstash-production-2020.02.04 10 37
logstash-production-2020.02.05 10 33
logstash-production-2020.02.06 10 37
logstash-production-2020.02.07 10 37
logstash-production-2020.02.08 4 7
logstash-production-2020.02.09 4 1
logstash-production-2020.02.10 10 29
logstash-production-2020.02.11 10 35
logstash-production-2020.02.12 10 6
logstash-production-2020.02.13 10 1
logstash-production-2020.02.14 10 30
logstash-production-2020.02.15 10 31
logstash-production-2020.02.16 10 1
logstash-production-2020.02.17 4 18
logstash-production-2020.02.18 10 36
logstash-production-2020.02.19 10 37
logstash-production-2020.02.20 10 31
logstash-production-2020.02.21 10 1
logstash-production-2020.02.22 10 28
logstash-production-2020.02.23 10 1
logstash-production-2020.02.24 10 34
logstash-production-2020.02.25 10 32
logstash-production-2020.02.26 10 37
logstash-production-2020.02.27 4 1
logstash-production-2020.02.28 10 34
logstash-production-2020.02.29 4 36
logstash-production-2020.03.01 4 1
logstash-production-2020.03.02 10 1
logstash-production-2020.03.03 10 1
logstash-production-2020.03.04 10 24
logstash-production-2020.03.05 4 1
logstash-production-2020.03.06 4 1
logstash-production-2020.03.07 4 31
logstash-production-2020.03.08 10 1
logstash-production-2020.03.09 10 1
logstash-production-2020.03.10 10 31
logstash-production-2020.03.11 4 1
logstash-production-2020.03.12 4 1
logstash-production-2020.03.13 4 1
logstash-production-2020.03.14 4 1
logstash-production-2020.03.15 4 1
logstash-production-2020.03.16 4 1
logstash-production-2020.03.17 4 1
logstash-production-2020.03.18 4 1
logstash-production-2020.03.19 4 1
logstash-production-2020.03.20 4 34
logstash-production-2020.03.21 4 33
logstash-production-2020.03.22 4 32
logstash-production-2020.03.23 4 38
logstash-production-2020.03.24 4 36
logstash-production-2020.03.25 4 52
logstash-stackdriver-2020.01.22 2 1
logstash-stackdriver-2020.01.23 2 1
logstash-stackdriver-2020.01.24 2 1
logstash-stackdriver-2020.01.25 2 1
logstash-stackdriver-2020.01.26 2 1
logstash-stackdriver-2020.01.27 2 26
logstash-stackdriver-2020.01.28 2 1
logstash-stackdriver-2020.01.29 2 1
logstash-stackdriver-2020.01.30 2 1
logstash-stackdriver-2020.01.31 2 1
logstash-stackdriver-2020.02.01 2 1
logstash-stackdriver-2020.02.02 2 1
logstash-stackdriver-2020.02.03 2 1
logstash-stackdriver-2020.02.04 2 1
logstash-stackdriver-2020.02.05 2 1
logstash-stackdriver-2020.02.06 2 1
logstash-stackdriver-2020.02.07 2 1
logstash-stackdriver-2020.02.08 2 1
logstash-stackdriver-2020.02.09 2 1
logstash-stackdriver-2020.02.10 2 1
logstash-stackdriver-2020.02.11 2 1
logstash-stackdriver-2020.02.12 2 1
logstash-stackdriver-2020.02.13 2 1
logstash-stackdriver-2020.02.14 2 1
logstash-stackdriver-2020.02.15 2 1
logstash-stackdriver-2020.02.16 2 1
logstash-stackdriver-2020.02.17 2 1
logstash-stackdriver-2020.02.18 2 1
logstash-stackdriver-2020.02.19 2 1
logstash-stackdriver-2020.02.20 2 1
logstash-stackdriver-2020.02.21 2 20
logstash-stackdriver-2020.02.22 2 1
logstash-stackdriver-2020.02.23 2 1
logstash-stackdriver-2020.02.24 2 1
logstash-stackdriver-2020.02.25 2 1
logstash-stackdriver-2020.02.26 2 32
logstash-stackdriver-2020.02.27 2 1
logstash-stackdriver-2020.02.28 2 1
logstash-stackdriver-2020.02.29 2 1
logstash-stackdriver-2020.03.01 2 1
logstash-stackdriver-2020.03.02 2 1
logstash-stackdriver-2020.03.03 2 1
logstash-stackdriver-2020.03.04 2 1
logstash-stackdriver-2020.03.05 2 1
logstash-stackdriver-2020.03.06 2 1
logstash-stackdriver-2020.03.07 2 1
logstash-stackdriver-2020.03.08 2 1
logstash-stackdriver-2020.03.09 2 1
logstash-stackdriver-2020.03.10 2 1
logstash-stackdriver-2020.03.11 2 1
logstash-stackdriver-2020.03.12 2 1
logstash-stackdriver-2020.03.13 2 1
logstash-stackdriver-2020.03.14 2 1
logstash-stackdriver-2020.03.15 2 1
logstash-stackdriver-2020.03.16 2 28
logstash-stackdriver-2020.03.17 2 1
logstash-stackdriver-2020.03.18 2 1
logstash-stackdriver-2020.03.19 2 1
logstash-stackdriver-2020.03.20 2 1
logstash-stackdriver-2020.03.21 2 1
logstash-stackdriver-2020.03.22 2 1
logstash-stackdriver-2020.03.23 2 1
logstash-stackdriver-2020.03.24 2 23
logstash-stackdriver-2020.03.25 2 38
logstash-staging-2020.01.22 2 1
logstash-staging-2020.01.23 2 1
logstash-staging-2020.01.24 2 1
logstash-staging-2020.01.25 2 1
logstash-staging-2020.01.26 2 27
logstash-staging-2020.01.27 2 1
logstash-staging-2020.01.28 2 1
logstash-staging-2020.01.29 2 1
logstash-staging-2020.01.30 2 1
logstash-staging-2020.01.31 2 1
logstash-staging-2020.02.01 2 1
logstash-staging-2020.02.02 2 1
logstash-staging-2020.02.03 2 1
logstash-staging-2020.02.04 2 27
logstash-staging-2020.02.05 2 27
logstash-staging-2020.02.06 2 1
logstash-staging-2020.02.07 2 1
logstash-staging-2020.02.08 2 1
logstash-staging-2020.02.09 2 1
logstash-staging-2020.02.10 2 20
logstash-staging-2020.02.11 2 1
logstash-staging-2020.02.12 2 1
logstash-staging-2020.02.13 2 1
logstash-staging-2020.02.14 2 1
logstash-staging-2020.02.15 2 1
logstash-staging-2020.02.16 2 1
logstash-staging-2020.02.17 2 1
logstash-staging-2020.02.18 2 1
logstash-staging-2020.02.19 2 1
logstash-staging-2020.02.20 2 1
logstash-staging-2020.02.21 2 1
logstash-staging-2020.02.22 2 1
logstash-staging-2020.02.23 2 1
logstash-staging-2020.02.24 2 30
logstash-staging-2020.02.25 2 1
logstash-staging-2020.02.26 2 1
logstash-staging-2020.02.27 2 1
logstash-staging-2020.02.28 2 1
logstash-staging-2020.02.29 2 1
logstash-staging-2020.03.01 2 39
logstash-staging-2020.03.02 2 1
logstash-staging-2020.03.03 2 1
logstash-staging-2020.03.04 2 1
logstash-staging-2020.03.05 2 1
logstash-staging-2020.03.06 2 1
logstash-staging-2020.03.07 2 1
logstash-staging-2020.03.08 2 1
logstash-staging-2020.03.09 2 1
logstash-staging-2020.03.10 2 1
logstash-staging-2020.03.11 4 1
logstash-staging-2020.03.13 2 1
logstash-staging-2020.03.14 2 1
logstash-staging-2020.03.15 2 1
logstash-staging-2020.03.16 2 1
logstash-staging-2020.03.17 2 1
logstash-staging-2020.03.18 2 1
logstash-staging-2020.03.19 2 1
logstash-staging-2020.03.20 2 27
logstash-staging-2020.03.21 2 1
logstash-staging-2020.03.22 2 1
logstash-staging-2020.03.23 2 1
logstash-staging-2020.03.24 2 22
logstash-staging-2020.03.25 2 42
logstash-vault-2020.01.21 2 28
logstash-vault-2020.01.22 2 31
logstash-vault-2020.01.23 2 30
logstash-vault-2020.01.24 2 29
logstash-vault-2020.01.25 2 33
logstash-vault-2020.01.26 2 26
logstash-vault-2020.01.27 2 30
logstash-vault-2020.01.28 2 25
logstash-vault-2020.01.29 2 26
logstash-vault-2020.01.30 2 27
logstash-vault-2020.01.31 2 26
logstash-vault-2020.02.01 2 28
logstash-vault-2020.02.02 2 26
logstash-vault-2020.02.03 2 29
logstash-vault-2020.02.04 2 28
logstash-vault-2020.02.05 2 31
logstash-vault-2020.02.06 2 24
logstash-vault-2020.02.07 2 29
logstash-vault-2020.02.08 2 28
logstash-vault-2020.02.09 2 29
logstash-vault-2020.02.10 2 29
logstash-vault-2020.02.11 2 30
logstash-vault-2020.02.12 2 25
logstash-vault-2020.02.13 2 30
logstash-vault-2020.02.14 2 29
logstash-vault-2020.02.15 2 26
logstash-vault-2020.02.16 2 25
logstash-vault-2020.02.17 2 17
logstash-vault-2020.02.18 2 21
logstash-vault-2020.02.19 2 27
logstash-vault-2020.02.20 2 29
logstash-vault-2020.02.21 2 27
logstash-vault-2020.02.22 2 27
logstash-vault-2020.02.23 2 25
logstash-vault-2020.02.24 2 32
logstash-vault-2020.02.25 2 24
logstash-vault-2020.02.26 2 25
logstash-vault-2020.02.27 2 28
logstash-vault-2020.02.28 2 30
logstash-vault-2020.02.29 2 28
logstash-vault-2020.03.01 2 18
logstash-vault-2020.03.02 2 28
logstash-vault-2020.03.03 2 32
logstash-vault-2020.03.04 2 22
logstash-vault-2020.03.05 2 28
logstash-vault-2020.03.06 2 26
logstash-vault-2020.03.07 2 27
logstash-vault-2020.03.08 2 28
logstash-vault-2020.03.09 2 27
logstash-vault-2020.03.10 2 30
logstash-vault-2020.03.11 2 1
logstash-vault-2020.03.12 2 1
logstash-vault-2020.03.13 2 1
logstash-vault-2020.03.14 2 1
logstash-vault-2020.03.15 2 1
logstash-vault-2020.03.16 2 1
logstash-vault-2020.03.17 2 1
logstash-vault-2020.03.18 2 23
logstash-vault-2020.03.19 2 23
logstash-vault-2020.03.20 2 26
logstash-vault-2020.03.21 2 25
logstash-vault-2020.03.22 2 29
logstash-vault-2020.03.23 2 25
logstash-vault-2020.03.24 2 21
logstash-vault-2020.03.25 2 39
@lawrencejones
Copy link
Author

Remaining indices that aren't forcemerged:

cat <<EOF | xargs -n1 -I IDX -P20 gtimeout 1 curl -X POST --silent "localhost:9200/IDX/_forcemerge?max_num_segments=1"
logstash-staging-2020.02.10
logstash-staging-2020.03.24
logstash-staging-2020.02.05
logstash-staging-2020.03.20
logstash-staging-2020.02.24
logstash-staging-2020.03.01
logstash-staging-2020.03.25
logstash-production-2020.02.02
logstash-production-2020.02.03
logstash-production-2020.02.04
logstash-production-2020.02.05
logstash-production-2020.02.06
logstash-production-2020.02.07
logstash-production-2020.02.08
logstash-production-2020.02.10
logstash-production-2020.02.11
logstash-production-2020.02.12
logstash-production-2020.02.14
logstash-production-2020.02.15
logstash-production-2020.02.17
logstash-production-2020.02.18
logstash-production-2020.02.19
logstash-production-2020.02.20
logstash-production-2020.02.22
logstash-production-2020.02.24
logstash-production-2020.02.25
logstash-production-2020.02.26
logstash-production-2020.02.28
logstash-production-2020.02.29
logstash-production-2020.03.04
logstash-production-2020.03.07
logstash-production-2020.03.10
logstash-production-2020.03.20
logstash-production-2020.03.21
logstash-production-2020.03.22
logstash-production-2020.03.23
logstash-production-2020.03.24
EOF

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment