replay

latency to 

histogram_quantile(
  0.99, sum by (le) (
    cortex_request_duration_seconds_bucket{cluster=~"$cluster", job=~"($namespace)/(store-gateway|cortex$)", route=~"/gatewaypb.StoreGateway/.*"}
  )
) * 1e3

---

replay

#!/usr/bin/env python3

import fileinput

data_points_by_id = {}
metric_names_by_id = {}

# that's expecting the output of
# MT_KAFKA_MDM_IN_OFFSET=newest time mt-kafka-mdm-sniff -format-md '{{.OrgId}} {{.Id}} {{.Name}} {{.Tags}} {{.Time}}' -format-point '{.MKey}} {{.Time}}'

replay

* Spinning up MT cluster which consists of 4 shard groups, 2 MTs per shard.
  Each read MT has 10G Mem and 2 CPU cores on GKE n1-highmem-2 nodes.
* Feeding it with a fakemetrics generator. In total it generates 5 million metrics,
  each metric has 12 tags:
  8 of them have the same tags and values for all metrics.
  3 of them have a unique value for each metric
  1 is a tag of the format "host=[0-9]+"
    in total there are 30k possible values for this tag then they start to repeat,
    so each possible value such as "host=123" has 5M/30k = 166 metrics associated.
* 30k Meta records get generated. Each of the records is associated with one "host=X"

replay

#!/usr/bin/env python

import os
import time
import re
import sys

pattern = re.compile(os.getenv('METRIC_PATTERN', False) or '.*')

replay

import (
	"context"
	"sync"
	"sync/atomic"
	"testing"

	"golang.org/x/sync/errgroup"
)

replay

This example assumes we only keep 3 hours of data in Kafka, to illustrate the problem with 6h chunks

Time --->
| 05:00 | 06:00 | 07:00 | 08:00 | 09:00 | 10:00 | 11:00 | 12:00 |
|---------------------------------------------------------------|
        ^                                               ^ Times when 6h of data get flushed into the backend store
                                                    ^ MT  restarts shortly before reaching an hour when it was supposed to flush 6 hours
                            |-----------------------| Time range that MT can replay from Kafka      

replay

I0121 22:47:45.819471       1 main.go:61] hosted-metrics-cli (built with go1.11, git hash watch_endpoint18)
I0121 22:47:45.821422       1 config.go:108] reloading default configs from configmap hm-api-config
I0121 22:47:45.830057       1 config.go:146] default configs loaded.
I0121 22:47:45.905048       1 manager.go:524] cassandra keyspace created for 1-replaytest1
I0121 22:47:45.905074       1 manager.go:959] DeployInstance: Configuring kafka topics for 1-replaytest1
I0121 22:47:48.640802       1 manager.go:569] Kafka topic m-1-M-replaytest1 created successfully. Error while executing topic command : Topic 'm-1-M-replaytest1' already exists.
I0121 22:47:51.042218       1 manager.go:569] Kafka topic p-1-M-replaytest1 created successfully. Error while executing topic command : Topic 'p-1-M-replaytest1' already exists.
I0121 22:47:51.042308       1 manager.go:965] DeployInstance: Configuring carbon-relay-ng service for 1-replaytest1
I0121 22:47:51.053158       1 manager.go:970] DeployInstance: Configuring metri

replay

#!/bin/bash

set -x

OLD_BROKER_ID=$1
NEW_BROKER_ID=$2

if [ -z "$OLD_BROKER_ID" ]; then
  echo "usage: $0 <oldBroker> <newBroker>"
  exit 1

replay

#!/usr/bin/env python                                                                                                                    
                                                                                                                                     
import math                                                                                                                          
from prettytable import PrettyTable                                                                                                  
                                                                                                                                     
def get_hours(start, end):                                                                                                           
    if start == end:                                                                                                                 
        return str(start)                                                         

replay

#!/bin/bash

# Make sure only root can run our script
if [ "$(id -u)" != "0" ]; then
   echo "You need to be 'root' dude." 1>&2
   exit 1
fi

clear