Whisper Aggregation Backfill

outline.md

whisper-aggregation-backfill

Assumptions:

• All metrics and aggregations have matching retentions
• A proper metric aggregator is already running
• The resulting metric file will then be used in a whisper-fill operation against the aggregator's actual metric files.

Procedure

1. Aggregate all source metric files to a common location
2. Create one virtual archive per destination archive
3. Aggregate data points from each source archive into the virtual destination.
   • sum: $virt[$timestamp] += $value
   • avg: $virt[$timestamp]['value'] += $value; ++virt[$timestamp]['count']
4. Pare virtual archive points down to fit in their respective retentions.
5. Resolve complex aggregations, eg: avg
6. Write final aggregation file.

Concerns

1. Memory Usage for Virtual Archives
   • Actually not that bad, depending on how far back the aggregations go and assuming 12 bytes per point.
   • Worst case scenario of an archive with secondsPerPoint of 1 and data going back one year, the virtual archive would still fit within 361MB of memory.
   • Additional configuration could be implemented to discard points older than given thresholds.