Skip to content

Instantly share code, notes, and snippets.

@xoelop

xoelop/Getting help with a ClickHouse problem — issues & support tickets.md

Last active July 24, 2026 14:20
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save xoelop/fc6266377443286ab6394cce0a757219 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save xoelop/fc6266377443286ab6394cce0a757219 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
clickhouse-fiddle.md

ClickHouse Fiddle — run SQL against any CH version

fiddle.clickhouse.com is a free web playground that runs arbitrary SQL against pinned ClickHouse versions. Use it to reproduce bugs against specific versions, compare query results across versions (e.g. "does this fail in 26.2 but work in 26.3?"), and attach a permalink to GitHub issues.

API (no auth, no cookie needed)

curl -sS 'https://fiddle.clickhouse.com/api/runs' \
  -H 'content-type: application/json' \
  --data-raw '{
    "query": "SELECT version()",
    "version": "26.3",
    "settings": {"clickhouse": {"output_format": "TabSeparated"}}
  }'

Returns JSON:

{
  "result": {
    "query_run_id": "3d467784-e68e-47a0-87af-7a553a6c1769",
    "output": "26.3.10.60\n",
    "time_elapsed": "1.457s"
  }
}

On error:

{ "error": { "message": "Code: 49. DB::Exception: …", "code": 500 } }

The query_run_id is a permalink shareable as https://fiddle.clickhouse.com/<query_run_id>.

Version pinning

The version field accepts major.minor only, not full tags:

  • "latest" — currently runs the newest stable
  • "26.4" — latest patch of the 26.4 line
  • "26.3" — latest patch of the 26.3 LTS line
  • "26.2", "26.1", "25.10", "25.8", … — older lines (whatever patch they keep available)

v26.3.5.12-lts and similar full tags return {"error":{"message":"unknown version","code":400}}.

To find the exact patch for a major.minor, query SELECT version() once.

Multi-statement scripts

Pass --multiquery semantics implicitly — just put ;-separated statements in the query string:

curl -sS 'https://fiddle.clickhouse.com/api/runs' \
  -H 'content-type: application/json' \
  --data-raw '{"query": "CREATE TABLE t (id UInt64) ENGINE = MergeTree ORDER BY id; INSERT INTO t VALUES (1),(2),(3); SELECT count() FROM t", "version": "latest"}'

Comparing versions for a bug repro

Pattern: same query, two API calls with different version. If one errors and the other returns a result, you have a clean version bisect → paste both query_run_id permalinks into the GitHub issue:

REPRO_SQL='<minimal repro here>'
for v in 26.2 26.3 26.4; do
  curl -sS 'https://fiddle.clickhouse.com/api/runs' \
    -H 'content-type: application/json' \
    --data-raw "$(jq -nc --arg q "$REPRO_SQL" --arg v "$v" '{query: $q, version: $v}')" \
    | jq --arg v "$v" '{version: $v, output: .result.output, error: .error.message, id: .result.query_run_id}'
done

Limits / caveats

  • Sandbox: no network, no file(), no s3() to private buckets, no MergeTree settings that require Keeper coordination beyond the single-node sandbox.
  • Tables don't persist between API calls — each query is one isolated script. Inline your CREATE TABLE + INSERT + the failing SELECT in one payload.
  • time_elapsed is wall time of the sandbox run, not query CPU — don't use it for benchmarking, use a real CH for that.

When to use

  • Pre-issue triage: reproduce a CH error against multiple versions before filing on ClickHouse/ClickHouse. Include the permalink in the issue body — the maintainers can re-run it instantly.
  • Pre-upgrade validation: confirm that a known-broken query starts passing on the target version before bumping CH.
  • Quick syntax / function lookup: faster than spinning up a local CH for one-off "does this work in 26.2 too?" checks.

Not for: benchmarking realistic workloads, anything stateful, anything that needs >1 node.

Raw
Getting help with a ClickHouse problem — issues & support tickets.md

Getting help with a ClickHouse problem — issues & support tickets

When we hit a ClickHouse problem we can't solve on our own — almost always a bug or something that doesn't behave as the docs describe, occasionally a genuine question — this is the playbook for asking for help in a way that actually gets it solved fast.

The whole strategy rests on one idea: make it trivially easy for someone who has never seen our data to help us. The more of our production context they'd need, the slower (or more impossible) the help.

The golden rule: a minimal, reproducible example

Always try to ship a self-contained, minimal reproduction. If someone can paste it and see the bug themselves, anyone — ClickHouse staff and the wider community — can jump in. If instead they'd need our exact data, our schema, or access to our database, help gets much harder to get.

Note on access: ClickHouse Cloud staff tell us they cannot get into our database. In practice we could always create a user for them (or they may have some break-glass path), but we should operate as if they can't. So the burden is on us to hand them a repro they can run anywhere.

Isolate and shrink. Cut everything that isn't needed to trigger the problem. Remove columns, rows, settings, and steps until removing one more makes the bug disappear. Noise makes it harder for whoever picks it up to understand what's going on — the minimization is part of explaining the problem, and often reveals the cause (or our own mistake) along the way.

Never use our production schema

The repro must not use our real table or column names. Two reasons:

  1. We don't want to expose our schema.
  2. A generic, familiar schema is easier for a stranger to read. Use a textbook domain that explains itself:
    • authors / books / publishers / comments
    • movies / actors / directors
    • documents / …

Anyone can reason about authors and books instantly; nobody has to decode what fanout_feed_v3 is.

Where the data comes from (in order of preference)

The ideal repro needs no external data at all. Reach for the cheapest option that works:

  1. Data generated on the fly — numbers(), generateRandom(), a handful of INSERT ... VALUES. Self-contained inside the SQL, nothing to download. This is the target.
  2. Public data ClickHouse already uses — the public S3 datasets from their own docs/tutorials, or any other already-public source. No upload needed.
  3. A small CSV on GitHub — if you truly need real-shaped data and it's small, attach it to the issue or drop it in a GitHub Gist.
  4. Our public R2 / S3 bucket — last resort. If we've had to upload our own data, it usually means we didn't reduce the problem enough. Treat needing this as a smell, not a solution.

Always attach a ClickHouse Fiddle

Every issue and every support ticket should include a ClickHouse Fiddle permalink alongside the repro SQL. ClickHouse people live in Fiddle: it's a one-click "run this exact SQL against this exact version" — no copy-pasting into a client, and it makes it trivial to try the same code across ClickHouse versions (does it break in 26.4 but work in 26.3?).

How we build and share Fiddle links (the API, version pinning, cross-version bisects): clickhouse-fiddle.md.

So a good bug report is: a generic self-contained repro in the body + a Fiddle permalink that runs it.

Issue in the ClickHouse repo and a support ticket

When it's a reproducible bug, do both:

  • Open an issue on ClickHouse/ClickHouse. More eyes — ClickHouse engineers and the community. Often someone replies quickly with "you're holding it wrong" and the fix, which is the fastest possible outcome.
  • Also open a ClickHouse Cloud support ticket, linking the GitHub issue. The support ticket tends to get it looked at faster.

When you cannot build any reproduction, there's little value in a public GitHub issue (nobody can act on it without our data) — go straight to a support ticket, since it's tied to our account and our cluster's live state.

Either way, put the Fiddle link in whatever you file.

Support ticket severity: pick the lowest that fits. Unless something is genuinely on fire — production is down, data loss, a customer-facing outage — open the ticket at the minimum severity. High-severity tickets page people and burn goodwill; save them for real emergencies so that when we do raise one, it's taken seriously.

Anatomy of a good bug report (worked example)

#110275 — Wrong result with filter push-down, ANY JOIN, CTE, and NOT IN is the template to copy. It's worth reading in full; here's why it works, section by section:

  • Summary — one paragraph: the version, the exact symptom, and the one knob that flips it. "On 26.4.1.1960, the query returns 0 rows with query_plan_filter_push_down = 1, but the expected 25 rows with = 0." A reader knows the whole bug before scrolling.
  • Repro SQL — a single self-contained script: DROP + CREATE + INSERT ... VALUES + the failing SELECT, twice, differing only in the one setting. No external files, generic authors / books / publishers / blocked_authors schema, tiny data (25 authors, 32 books, 1 publisher, 1 blocked author). Paste-and-run anywhere. The self-SELECT version(), getSetting(...) and a 'sanity' count line make it obvious what ran.
  • Actual Result vs Expected Result — both printed literally, side by side, so there's zero ambiguity about what "wrong" means.
  • Notes From Reduction — the part most reports skip and the reason this one is easy to act on. It states that the original query had ranking/window/limit logic that turned out to be irrelevant to the bug, why we can't just disable the setting globally (perf), and — crucially — a list of control reductions that did not reproduce (remove the NOT IN → 25 rows; filter only by genre → 25 rows; …). That last list hands the maintainer a pre-bisected boundary of exactly which ingredient triggers it.

The lesson: don't just paste a repro — show your reduction. Listing what you removed and what still failed is doing the maintainer's first hour of work for them.

Pair this exact body with a Fiddle permalink that runs the same script and the report is essentially self-serve.

Examples from our own history

Real issues filed from xoelop, as models to copy.

Reproducible (the majority — aim for these)

Not reproducible (question / live-cluster state / feature request)

These couldn't be reduced to a portable repro — they depend on our hardware, workload, or a stuck live state — so they lean on a support ticket and/or read as questions:

Checklist before filing

  • Problem isolated and reduced to the minimum that still triggers it.
  • Generic schema (authors/books/movies/…), no production table or column names.
  • Data is generated on the fly or from a public source; nothing uploaded to our R2 unless unavoidable.
  • A ClickHouse Fiddle permalink is included.
  • If it's a reproducible bug: GitHub issue + support ticket (cross-linked).
  • If it's not reproducible: support ticket directly.
  • Support ticket opened at the lowest severity that fits (high severity only for production-down / data-loss).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment