Skip to content

Instantly share code, notes, and snippets.

@thepaul
thepaul / testspannerxor_short.sql
Created November 15, 2024 14:30
query to calculate an xor over the sha1 digests of all fields in all rows of a spanner table
WITH as_bytes AS (
SELECT
stream_id,
CAST(CAST(position AS STRING) AS BYTES) AS position,
CAST(format_timestamp('%E4Y-%m-%d %H:%M:%E6S', created_at, 'UTC') AS BYTES) AS created_at,
CAST(COALESCE(format_timestamp('%E4Y-%m-%d %H:%M:%E6S', repaired_at, 'UTC'), '(null)') AS BYTES) AS repaired_at,
CAST(COALESCE(format_timestamp('%E4Y-%m-%d %H:%M:%E6S', expires_at, 'UTC'), '(null)') AS BYTES) AS expires_at,
root_piece_id,
encrypted_key_nonce,
encrypted_key,
@thepaul
thepaul / encode_ulid_from_timestamp.sql
Created January 6, 2024 17:42
Generate a ULID with a specific TIMESTAMPTZ embedded in it
SELECT
overlay(
overlay(
gen_random_ulid()::STRING
PLACING
overlay(
'0' -- caution: prepending '0' is only appropriate until June of 2527
||
to_hex(
(
@thepaul
thepaul / decode_timestamp_from_ulid.sql
Created January 6, 2024 17:41
Decode the timestamp embedded in a ULID into a TIMESTAMPTZ
SELECT
'epoch'::TIMESTAMPTZ
+
(
'x'
||
lpad(
substr(
replace(
the_uuid::STRING,
@thepaul
thepaul / writecache_sizecalc.go
Created June 14, 2022 23:03
Calculating memory usage of reputation writecache
package main
import (
"sync"
"time"
"unsafe"
"golang.org/x/text/language"
"golang.org/x/text/message"
)
data loss lambda dq thresh grace rounds dq fraction mean rounds to dq rounds
0.01 0.950 0.90 30 0.193 473.36 1000
0.02 0.950 0.90 30 0.724 399.98 1000
0.03 0.950 0.90 30 0.975 271.07 1000
0.04 0.950 0.90 30 0.998 153.60 1000
0.05 0.950 0.90 30 1.000 99.79 1000
0.06 0.950 0.90 30 1.000 74.29 1000
0.01 0.980 0.90 30 0.001 50.00 1000
0.02 0.980 0.90 30 0.032 145.78 1000
0.03 0.980 0.90 30 0.202 291.65 1000
Data Lost Nodes Lambda Init-alpha Init-beta Succ-adj Fail-adj DQ thresh Weight Audits % DQ'd Mean audits to DQ Stddev audits to DQ
0.01 3000 0.95 1 0 1 -1 0.8 1 10000 4.27 97 819
0.02 3000 0.95 1 0 1 -1 0.8 1 10000 11.03 944 2276
0.03 3000 0.95 1 0 1 -1 0.8 1 10000 27.73 2563 3248
0.04 3000 0.95 1 0 1 -1 0.8 1 10000 61.62 3098 3139
0.05 3000 0.95 1 0 1 -1 0.8 1 10000 93.24 2328 2565
0.06 3000 0.95 1 0 1 -1 0.8 1 10000 99.77 1192 1544
0.07 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 571 802
0.08 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 311 457
0.09 3000 0.95 1 0 1 -1 0.8 1 10000 100.00 189 281
@thepaul
thepaul / data_loss_vs_dq_chance_sim.py
Created February 4, 2022 19:50
Simulating chance of disqualification for a given % of data loss
#!/usr/bin/env python3
#
# This is meant to act as a double-check of math used elsewhere to calculate the chance
# of disqualification given a level of data loss on a storage node. Instead of calculating
# the chance algorithmically, this performs a simulation using the specified parameters
# and reports how many nodes under the given circumstances would actually be disqualified.
import argparse
import csv
import math
@thepaul
thepaul / tuning reputation further.ipynb
Created February 4, 2022 19:43
Tuning reputation calculation parameters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
// Copyright (C) 2021 Storj, Inc.
// See LICENSE for copying information.
package main
import (
"context"
"crypto/tls"
"fmt"
"log"
@thepaul
thepaul / git-get-remote-head.sh
Created December 18, 2020 17:22
Get the name of the remote HEAD (the default branch on that remote)
#!/bin/bash -e
REMOTE=${1:-origin}
ref=$(git rev-parse --symbolic-full-name "$REMOTE/HEAD" 2>/dev/null) || {
git remote set-head "$REMOTE" -a >/dev/null
ref=$(git rev-parse --symbolic-full-name "$REMOTE/HEAD")
}
echo "${ref#refs/remotes/$REMOTE/}"