Skip to content

Instantly share code, notes, and snippets.

View amosbird's full-sized avatar
🎯
Focusing

Amos Bird amosbird

🎯
Focusing
View GitHub Profile
@amosbird
amosbird / gist:beda0facb616d5cc24a182902ba08000
Last active June 5, 2026 18:22
RFC: Per-Part Static Type Narrowing for JSON Dynamic Paths

Summary

When all values of a JSON dynamic path within a single data part share the same type (e.g., all Int64), store that path as a plain typed column (1 .bin + 1 .cmrk2) instead of a full Dynamic column (5 substream pairs = 10 files). This eliminates the variant_discr, dynamic_structure, SharedVariant, and SharedVariant.size substreams that are redundant when type is homogeneous.

Motivation

The JSON type stores each dynamic path using the Dynamic (Variant) type, which always produces 5 substream pairs per path:

payload.field.dynamic_structure.{bin,cmrk2} -- which type is this row
@amosbird
amosbird / selective_replication_rfc.md
Created March 1, 2026 12:43
RFC: Selective Replication for ReplicatedMergeTree

RFC: Selective Replication for ReplicatedMergeTree

Status: Draft Based on: #45766, PR #58132

Summary

Extend ReplicatedMergeTree so that each replica is responsible for an automatically selected subset of partitions. Replicas and shards are unified — no Distributed table needed. Partition key doubles as sharding key.

Motivation

```
0x89996d0
inline_memcpy(void*, void const*, unsigned long)
./build_docker/./base/glibc-compatibility/memcpy/memcpy.h:188
memcpy
./build_docker/./base/glibc-compatibility/memcpy/memcpy.cpp:5
0x13c74c1a
void DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 63ul, 64ul>::push_back<unsigned long const&>(unsigned long const&)
./build_docker/./src/Common/PODArray.h:436
DB::ColumnString::insert(DB::Field const&)
static void initRowsBeforeLimit(IOutputFormat * output_format)
{
RowsBeforeLimitCounterPtr rows_before_limit_at_least;
/// TODO: add setRowsBeforeLimitCounter as virtual method to IProcessor.
std::vector<LimitTransform *> limits;
std::vector<PartialSortingTransform *> partial_sortings;
std::vector<RemoteSource *> remote_sources;
std::unordered_set<IProcessor *> visited;
@amosbird
amosbird / amos.pac
Last active December 14, 2022 02:19
pac
var direct = 'DIRECT';
var http_proxy = 'PROXY 127.0.0.1:12639';
var gfwed_list = [
"oa.com",
"woa.com",
"oa.tencent.com",
];
var gfwed = {};
@amosbird
amosbird / gist:113c8c06cb04ac2557af85350fe00c00
Created January 21, 2022 07:20
Minimal CentOS 6 image to compile Apache Doris
FROM centos:6
RUN curl https://www.getpagespeed.com/files/centos6-eol.repo --output /etc/yum.repos.d/CentOS-Base.repo
RUN yum update -y && \
yum install -y \
byacc \
I have an vpn tunnel which generates this device
22: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1390 qdisc fq_codel state UNKNOWN group default qlen 500
link/none
inet 172.19.60.64/32 scope global tun0
valid_lft forever preferred_lft forever
@amosbird
amosbird / diskcheck.sh
Last active August 17, 2017 05:48
check disk failures using megacli
#!/usr/bin/env bash
while [[ 1 ]]; do
nd=$(sudo /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL | grep "Firmware state: Online, Spun Up" | wc -l)
if [[ nd -ne <YOU DISK NUM> ]];
then
echo "check it using sudo /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL" | mail -s "$HOSTNAME has disk failure!!" <YOU EMAIL ADDRESS>
sleep 3600
fi
@amosbird
amosbird / ahk.ahk
Created August 16, 2017 13:46
better windows experience
#SingleInstance force
Loop, %0% ; For each parameter:
{
param := %A_Index% ; Fetch the contents of the variable whose name is contained in A_Index.
params .= A_ralt . param
}
ShellExecute := A_IsUnicode ? "shell32\ShellExecute":"shell32\ShellExecuteA"
if not A_IsAdmin
❯ xinput --list-props 10 (53s 637ms)
Device 'Lite-On Technology Corp. ThinkPad USB Keyboard with TrackPoint':
Device Enabled (138): 1
Coordinate Transformation Matrix (140): 1.000000, 0.000000, 0.000000, 0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000
libinput Accel Speed (276): 1.000000
libinput Accel Speed Default (277): 0.000000
libinput Accel Profiles Available (278): 1, 1
libinput Accel Profile Enabled (279): 1, 0
libinput Accel Profile Enabled Default (280): 1, 0
libinput Natural Scrolling Enabled (281): 0