To understand the proper memory limits for Prometheus, we first need to observe the memory usage behavior of the deployment. After the increase in memory, we can clearly see the reported usage doesn't decrease but oscillate around new level. This is an unexpected behavior, and why the RSS memory is not released after the increase needs to be investigated. Keep in mind that this pattern was initially observed in only one region.
Current configuration:
- Version 2.18.1
- GoVersion go1.14.2
- Prometheus Helm Chart: v11.5.0