Additional log lines to show the bug in action.

instant-metric-query-bug-incorrect-results

(first-time)

Debug!! val {} => 598 @[1710835200000] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[1h]))
Debug!! val {} => 598 @[1710835200000] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>>)

Debug!! val {} => 1803 @[1710831600000] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[1h]))
Debug!! val {} => 1803 @[1710831600000] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>>)


Debug!! val {} => 18 @[1710835690422] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[8m10s422ms]))
Debug!! val {} => 18 @[1710835690422] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[8m10s422ms])), shard=<nil>>)

Debug!! val {} => 659 @[1710828000000] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[51m49s577ms]))
Debug!! val {} => 659 @[1710828000000] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[51m49s577ms])), shard=<nil>>)

Debug!! val {} => 0.285 @[1710835690422] err <nil> expr sum((sum without ()(downstream<sum(count_over_time({job="varlogs"} != "cache17"[51m49s577ms])), shard=<nil>> ++ downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>> ++ downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>> ++ downstream<sum(count_over_time({job="varlogs"} != "cache17"[8m10s422ms])), shard=<nil>>) / 10800))
Debug!!! split_by_range result-value [{} => 0.285 @[1710835690.422]]


(second-time) <- Buggy!!!!!

Debug!! val {} => 598 @[1710835200000] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[1h]))
Debug!! val {} => 598 @[1710835200000] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>>)

Debug!! val {} => 55 @[1710835764723] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[9m24s723ms]))
Debug!! val {} => 55 @[1710835764723] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[9m24s723ms])), shard=<nil>>)

Debug!! val {} => 659 @[1710828000000] err <nil> expr sum(count_over_time({job="varlogs"} != "cache17"[50m35s276ms]))
Debug!! val {} => 659 @[1710828000000] err <nil> expr sum(downstream<sum(count_over_time({job="varlogs"} != "cache17"[50m35s276ms])), shard=<nil>>)

Debug!! val {} => 0.12148148148148148 @[1710835764723] err <nil> expr sum((sum without ()(downstream<sum(count_over_time({job="varlogs"} != "cache17"[50m35s276ms])), shard=<nil>> ++ downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>> ++ downstream<sum(count_over_time({job="varlogs"} != "cache17"[1h])), shard=<nil>> ++ downstream<sum(count_over_time({job="varlogs"} != "cache17"[9m24s723ms])), shard=<nil>>) / 10800))
Debug!!! split_by_range result-value [{} => 0.12148148148148148 @[1710835764.723]]

explanation:

1. In first time, correct results, no cache involved. Result = (598 + 1083 + 18 + 659)/10800 = 0.265
2. In second time, in-correct results, one of the subquery is never fetched from the cache (or not used in the aggregation) = (598 + 55 + 659)/10800 = 0.12148. Note the value 1083 for one specific subquery was never used in calculating final result