Simple awk script to calculate totals # of logs and total size of logs for "project" and "operations" indices from a /_cat/indices list retrieved from an OpenShift cluster with Aggregated Logging installed.

sum-es-indices.awk

#!/usr/bin/awk -f

BEGIN {
    if (length(DAY) > 0) {
        gensub(/\./, "\\\\.", DAY)
        dayfmt = " (" DAY ")"
    } else {
        dayfmt = ""
    }
    projects = "project\\..+\\." DAY
    operations = "\\.operations\\." DAY
}

# Total the number of all logs
{ all_logs += $6 }
# Total # of logs for Apps
$3 ~ projects { app_logs += $6}
# Total # of logs for Ops:
$3 ~ operations { ops_logs += $6}

# Total size of logs (unreplicated):
{ all_logs_size += $9 }
# Total size of logs for Apps (unreplicated):
$3 ~ projects { app_logs_size += $9 }
# Total size of logs for Ops (unreplicated):
$3 ~ operations { ops_logs_size += $9 }

END {
    if (length(SIMPLE) > 0) {
        printf "%d %s %d %.2f %d %.2f %d %s %d %.2f %d %.2f\n", all_logs, dayfmt, app_logs, 100*(app_logs/all_logs), ops_logs, 100*(ops_logs/all_logs), all_logs_size, dayfmt, app_logs_size, 100*(app_logs_size/all_logs_size), ops_logs_size, 100*(ops_logs_size/all_logs_size)
    } else {
        printf "Total count of logs: %d\nTotal count of application logs%s: %d (%.2f)\nTotal count of operational logs%s: %d (%.2f)\nTotal size of all logs: %d\nTotal size of application logs%s: %d (%.2f)\nTotal size of operational logs%s: %d (%.2f)\n", all_logs, dayfmt, app_logs, 100*(app_logs/all_logs), dayfmt, ops_logs, 100*(ops_logs/all_logs), all_logs_size, dayfmt, app_logs_size, 100*(app_logs_size/all_logs_size), dayfmt, ops_logs_size, 100*(ops_logs_size/all_logs_size)
    }
}

Here is an example of the output:

# oc exec -n logging -c elasticsearch logging-es-data-master-mopnq4m2-10-q5x6r -- indices | ./sum-es-indices.awk
Total count of logs: 514304981
Total count of application logs: 424040843 (82.45)
Total count of operational logs: 84099039 (16.35)
Total size of all logs: 659573
Total size of application logs: 574691 (87.13)
Total size of operational logs: 77299 (11.72)

Here is an example of the output specifying a specific day to consider for indices:

# oc exec -n logging -c elasticsearch logging-es-data-master-mopnq4m2-10-q5x6r -- indices | ./sum-es-indices.awk -v DAY="2018.06.02"
Total count of logs: 517292122
Total count of application logs: 37544606 (7.26)
Total count of operational logs: 24339114 (4.71)
Total size of all logs: 663457
Total size of application logs: 43222 (6.51)
Total size of operational logs: 22357 (3.37)

Here is an example using it as a command line argument with a file containing the data:

# ./sum-es-indices.awk -v DAY=2018.06.03 indices.2018-06-05.lis
Total count of logs: 486312696
Total count of application logs (2018.06.03): 33101698 (6.81)
Total count of operational logs (2018.06.03): 24223805 (4.98)
Total size of all logs: 629301
Total size of application logs (2018.06.03): 39001 (6.20)
Total size of operational logs (2018.06.03): 22209 (3.53)

Here is an example of printing out the last 14 days of indices individually sum'd:

# for range in {0..13}; do cat indices.2018-06-05.lis | awk -f ./sum-es-indices.awk -v DAY=$(date +"%Y.%m.%d" --date="$range days ago"); done
Total count of logs: 486312696
Total count of application logs (2018.06.05): 3947748 (0.81)
Total count of operational logs (2018.06.05): 901799 (0.19)
Total size of all logs: 629301
Total size of application logs (2018.06.05): 5141 (0.82)
Total size of operational logs (2018.06.05): 875 (0.14)
Total count of logs: 486312696
Total count of application logs (2018.06.04): 50921760 (10.47)
Total count of operational logs (2018.06.04): 22129898 (4.55)
Total size of all logs: 629301
Total size of application logs (2018.06.04): 65176 (10.36)
Total size of operational logs (2018.06.04): 20259 (3.22)
Total count of logs: 486312696
Total count of application logs (2018.06.03): 33101698 (6.81)
Total count of operational logs (2018.06.03): 24223805 (4.98)
Total size of all logs: 629301
Total size of application logs (2018.06.03): 39001 (6.20)
Total size of operational logs (2018.06.03): 22209 (3.53)
Total count of logs: 486312696
Total count of application logs (2018.06.02): 37544606 (7.72)
Total count of operational logs (2018.06.02): 24339114 (5.00)
Total size of all logs: 629301
Total size of application logs (2018.06.02): 43222 (6.87)
Total size of operational logs (2018.06.02): 22357 (3.55)
Total count of logs: 486312696
Total count of application logs (2018.06.01): 42845727 (8.81)
Total count of operational logs (2018.06.01): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.06.01): 59629 (9.48)
Total size of operational logs (2018.06.01): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.31): 44602768 (9.17)
Total count of operational logs (2018.05.31): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.31): 56134 (8.92)
Total size of operational logs (2018.05.31): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.30): 29409114 (6.05)
Total count of operational logs (2018.05.30): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.30): 45486 (7.23)
Total size of operational logs (2018.05.30): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.29): 26810819 (5.51)
Total count of operational logs (2018.05.29): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.29): 41097 (6.53)
Total size of operational logs (2018.05.29): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.28): 29119217 (5.99)
Total count of operational logs (2018.05.28): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.28): 45590 (7.24)
Total size of operational logs (2018.05.28): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.27): 23088189 (4.75)
Total count of operational logs (2018.05.27): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.27): 33639 (5.35)
Total size of operational logs (2018.05.27): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.26): 22224467 (4.57)
Total count of operational logs (2018.05.26): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.26): 32284 (5.13)
Total size of operational logs (2018.05.26): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.25): 23640292 (4.86)
Total count of operational logs (2018.05.25): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.25): 34310 (5.45)
Total size of operational logs (2018.05.25): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.24): 20788471 (4.27)
Total count of operational logs (2018.05.24): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.24): 28217 (4.48)
Total size of operational logs (2018.05.24): 0 (0.00)
Total count of logs: 486312696
Total count of application logs (2018.05.23): 20508092 (4.22)
Total count of operational logs (2018.05.23): 0 (0.00)
Total size of all logs: 629301
Total size of application logs (2018.05.23): 27092 (4.31)
Total size of operational logs (2018.05.23): 0 (0.00)

Use SIMPLE=1 to get a one line summary output:

# ./sum-es-indices.awk -v DAY=2018.06.03 -v SIMPLE=1 indices.2018-06-05.lis
486312696  (2018.06.03) 33101698 6.81 24223805 4.98 629301  (2018.06.03) 39001 6.20 22209 3.53