Example for computing various running statistics with Lua in Redis backed by a hash

Redis_Stats.md

Running statistics with Redis and Lua

This is an example for computing running statistics with Lua backed by a hash in Redis. We support counting, average (with and without exponential smoothing), stddev, variance, min, max, sum of observed values. An example for approximating a running median can be found here: https://gist.github.com/thomasdarimont/fff68191d45a001b2d84

Data structure

We use a hash for storing various statistic value under the key "stats_value" in redis. Note: If you need a specific alpha value for smoothing the average, then set the desired alpha -> e.g. alpha 0.7. If alpha is 0.0 then no smoothing is applied.

HMSET stats_value 
  current 0  -- current / last seen value
  min 9223372036854775807  -- the min value seen 
  max -9223372036854775808 -- the max value seen
  mean 0.0 -- the observed mean / average value
  stddev 0.0  -- the observed standard deviation
  variance 0.0  -- the observed variance
  sumOfSquares 0.0 -- the observed sum of squared values  
  sum 0.0 -- the total sum of values observed
  count 0 -- the count of values observed
  alpha 0.0 -- the alpha value used for a smoothing average

The lua script

Note that the lua script needs to be in one line to be loaded properly.

-- script load "
local key, value = KEYS[1], tonumber(ARGV[1]);

local values = redis.call('HMGET', key, 'min', 'max', 'mean', 'count', 'sumOfSquares', 'sum', 'alpha');

local min = math.min(value, tonumber(values[1]));
local max = math.max(value, tonumber(values[2]));
local mean = tonumber(values[3]);
local count = tonumber(values[4]) + 1;
local sumOfSquares = tonumber(values[5]) + value * value;
local sum = tonumber(values[6]) + value;
local alpha = tonumber(values[7]);
local stddev = 0.0;
local variance = 0.0;

if(count > 1) then

  if(alpha == 0.0) then
    mean = mean + (value  - mean) / count;
  else 
    mean = (alpha * value) + (1.0 - alpha) * mean;
  end;
  
  stddev = math.sqrt((count * sumOfSquares - sum * sum) / (count * (count -1)));
  variance = stddev * stddev;
else
  mean = value;
end;

redis.call('HMSET', key, 'min', min, 'max', max, 'current', value, 'mean', mean, 'variance', variance, 'stddev', stddev, 'count', count, 'sum', sum, 'sumOfSquares', sumOfSquares);

if(ARGV[2]=='get_stats') then
  return {'current', value, 'min', min, 'max', max, 'mean', mean, 'stddev', stddev, 'variance', variance, 'sum', sum, 'count', count, 'alpha', alpha};
end;
--      "

Load script with Redis

You can import the script with the script load command which returns the sha value of the script that we use with the evalsha command later on. Paste the following snippet into redis-cli:

script load "local key, value = KEYS[1], tonumber(ARGV[1]); local values = redis.call('HMGET', key, 'min', 'max', 'mean', 'count', 'sumOfSquares', 'sum', 'alpha'); local min = math.min(value, tonumber(values[1])); local max = math.max(value, tonumber(values[2])); local mean = tonumber(values[3]); local count = tonumber(values[4]) + 1; local sumOfSquares = tonumber(values[5]) + value * value; local sum = tonumber(values[6]) + value; local alpha = tonumber(values[7]); local stddev = 0.0; local variance = 0.0; if(count > 1) then if(alpha == 0.0) then mean = mean + (value  - mean) / count; else mean = (alpha * value) + (1.0 - alpha) * mean; end; stddev = math.sqrt((count * sumOfSquares - sum * sum) / (count * (count -1))); variance = stddev * stddev; else mean = value; end; redis.call('HMSET', key, 'min', min, 'max', max, 'current', value, 'mean', mean, 'variance', variance, 'stddev', stddev, 'count', count, 'sum', sum, 'sumOfSquares', sumOfSquares); if(ARGV[2]=='get_stats') then return {'current', value, 'min', min, 'max', max, 'mean', mean, 'stddev', stddev, 'variance', variance, 'sum', sum, 'count', count, 'alpha', alpha}; end;"

Prepare a key for statistics collection

Just define a hash structure for statistics for the key "stats_value" based on the following definition. Note: If you need a specific alpha value for smoothing just initialize the hash with the desired alpha (between 0...1) -> e.g. alpha 0.7. A alpha value of 0.0 disables smoothing.

HMSET stats_value current 0 min 9223372036854775807 max -9223372036854775808 mean 0.0 stddev 0.0 variance 0.0 sumOfSquares 0.0 sum 0.0 count 0 alpha 0.0

List all stored statistic values

List all values for key "stats_value" in redis

hgetall stats_value

Update statistics for a key

We support 2 usage modes:

• Just update via evalsha THE_SCRIPT_SHA 1 THE_STATS_VALUE_KEY NEW_VALUE
• Update and return values AFTER update (by adding the get_stats as a second argument) evalsha THE_SCRIPT_SHA 1 THE_STATS_VALUE_KEY NEW_VALUE get_stats

Example

Note the sha 40ac074530b90a340a4d5013052b0a40e3c4aa7f is the result of the script load command above.

Eval loaded script with args:

evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 10
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 255 get_stats
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 32
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value -4  get_stats
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value -23
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 13
hgetall stats_value
evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value 3 get_stats
hgetall stats_value

Example Output

127.0.0.1:6379> hgetall stats_value
 1) "min"
 2) "-23"
 3) "max"
 4) "255"
 5) "mean"
 6) "40.8571428571428577"
 7) "stddev"
 8) "95.905211140008049"
 9) "variance"
10) "9197.8095238095248"
11) "sumOfSquares"
12) "66872"
13) "sum"
14) "286"
15) "count"
16) "7"
17) "current"
18) "3"
19) "alpha"
20) "0.0"

Example with exponential smoothing

HMSET stats_value_with_smoothing current 0 min 9223372036854775807 max -9223372036854775808 mean 0.0 stddev 0.0 variance 0.0 sumOfSquares 0.0 sum 0.0 count 0 alpha 0.7

127.0.0.1:6379> evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value_with_smoothing 10
(nil)
127.0.0.1:6379> hgetall stats_value_with_smoothing
 1) "min"
 2) "10"
 3) "max"
 4) "10"
 5) "mean"
 6) "10"
 7) "stddev"
 8) "0"
 9) "variance"
10) "0"
11) "sumOfSquares"
12) "100"
13) "sum"
14) "10"
15) "count"
16) "1"
17) "current"
18) "10"
19) "alpha"
20) "0.7"
127.0.0.1:6379> evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value_with_smoothing 5
(nil)
127.0.0.1:6379> hgetall stats_value_with_smoothing
 1) "min"
 2) "5"
 3) "max"
 4) "10"
 5) "mean"
 6) "6.5"
 7) "stddev"
 8) "3.5355339059327378"
 9) "variance"
10) "12.500000000000002"
11) "sumOfSquares"
12) "125"
13) "sum"
14) "15"
15) "count"
16) "2"
17) "current"
18) "5"
19) "alpha"
20) "0.7"
127.0.0.1:6379> evalsha "40ac074530b90a340a4d5013052b0a40e3c4aa7f" 1 stats_value_with_smoothing 15
(nil)
127.0.0.1:6379> hgetall stats_value_with_smoothing
 1) "min"
 2) "5"
 3) "max"
 4) "15"
 5) "mean"
 6) "12.449999999999999"
 7) "stddev"
 8) "5"
 9) "variance"
10) "25"
11) "sumOfSquares"
12) "350"
13) "sum"
14) "30"
15) "count"
16) "3"
17) "current"
18) "15"
19) "alpha"
20) "0.7"
127.0.0.1:6379>

Thank for this great script.

As a suggestion, it could initialise the key if it doesn't exist on the script itself as follows:

local key, value = KEYS[1], tonumber(ARGV[1]);

local values = redis.call('HMGET', key, 'min', 'max', 'mean', 'count', 'sumOfSquares', 'sum', 'alpha');

local min;
local max;
(...)

if(tonumber(values[1]) == nil) then
  min = 9223372036854775807;
  max = -9223372036854775808;
  (...) // Initialise other variables
else
  min = math.min(value, tonumber(values[1]));
  max = math.max(value, tonumber(values[2]));
  (...) // Calculate other variables
end;

// The rest of the script