Skip to content

Instantly share code, notes, and snippets.

View redsquare's full-sized avatar
📞
OnCall

steve flitcroft redsquare

📞
OnCall
View GitHub Profile
@redsquare
redsquare / gist:486f87acc0b9956c2d036b8b8b694643
Created September 12, 2024 14:08
clickhouse example server settings
<!--
NOTE: User and query level settings are set up in "users.xml" file.
If you have accidentally specified user-level settings here, server won't start.
You can either move the settings to the right place inside "users.xml" file
or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> here.
-->
<clickhouse>
<logger>
<!-- Possible levels [1]:
@redsquare
redsquare / meegemap.sql
Created September 4, 2024 07:15
merge maps take latest clickhouse
Handle the map as an array and then via ARRAY JOIN bring it into multiple rows, so you can use more complex SQL features. with argMax you could then get the latest value per key.
Then you need to regroup it again:
SELECT source, integration_id, transaction_id, min(min_event_time), groupArray((field_key, field_value))::Map(String, String)
FROM (
SELECT source, integration_id, transaction_id, min(event_time) AS min_event_time, field.1 AS field_key, argMax(field.2, event_time) AS field_value
FROM events
ARRAY JOIN fields AS field
GROUP BY source, integration_id, transaction_id, field.1
)
@redsquare
redsquare / clickhouse_parts.sql
Last active March 15, 2024 13:24
clickhouse parts query
SELECT
table,
partition_id,
count(name) AS parts,
formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE (table = 'qw_3_requests_quotes') AND active AND (toUInt32(partition_id) < 20231001)
GROUP BY
table,
partition_id
SELECT
table,
partition_id,
count(name) AS parts,
formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE (table = 'qw_3_requests_quotes') AND active AND (toUInt32(partition_id) < 20231001)
GROUP BY
table,
partition_id
@redsquare
redsquare / dump_restore_clickhouse.sql
Last active March 14, 2024 16:11
Dump Clickhouse Table Schemas
#dump table ddl's
clickhouse-client --host your_clickhouse_host -q "SHOW TABLES FROM db_name" | while read -r table_name; do
clickhouse-client --host your_clickhouse_host -q "SHOW CREATE TABLE db_name.\`$table_name\`" >> db_structure.sql
done
#restore
clickhouse-client --host target_clickhouse_host < db_structure.sql
#also in /var/lib/clickhouse/metadata/
:) CREATE TABLE ranges (id UInt64, date_start Date, date_end Date) ENGINE = TinyLog;
:) INSERT INTO ranges SELECT rowNumberInAllBlocks() AS id, least(dates[1], dates[2]) AS date_start, greatest(dates[1], dates[2]) AS date_end FROM (SELECT arrayJoin(arrayMap(d -> [today() - d - rand64(d) % 10, yesterday() - d - rand(d) % 10], range(10))) AS dates);
:) CREATE TABLE dates (id UInt64, date Date) ENGINE = TinyLog;
:) INSERT INTO dates SELECT rowNumberInAllBlocks() AS id, date FROM (SELECT arrayJoin(arrayMap(d -> today() - rand64(d) % 10, range(10))) AS date);
@redsquare
redsquare / gist:69394ee2c13dc1a541f84d88a5b12665
Created November 3, 2022 10:24
example prefect deploy - deploys new/changed flows
name: Register & Deploy Prefect flows
on:
push:
workflow_dispatch:
inputs:
pattern:
description: "Project Name or prohect name Pattern? (defaults to all)"
required: false
default: ".*"

Keybase proof

I hereby claim:

  • I am redsquare on github.
  • I am redsquare (https://keybase.io/redsquare) on keybase.
  • I have a public key ASBXSLaKvqYBgD34X1I2FSTC5hTdJqkr6lKhzu4__ZYE8Qo

To claim this, I am signing this object:

@redsquare
redsquare / program.cs
Created March 1, 2019 14:22
SSL stack redis issue
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Security;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using StackExchange.Redis;
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8"/>
<title>t1</title>
<script src="https://cdnjs.cloudflare.com/ajax/libs/benchmark/1.0.0/benchmark.min.js"></script>
<script src="./suite.js"></script>
</head>
<body>
<h1>Open the console to view the results</h1>