shashi

ClickHouse: Correct MV Aggregation Over CDC Data

Background

This document describes patterns for building materialized views over tables that use Change Data Capture (CDC) via ReplacingMergeTree. The core challenge is that MVs in ClickHouse are insert triggers, not recomputed views — so they see raw CDC rows before deduplication, and naive aggregations will be wrong.

CDC primer

shashi

Extensible Symbolic Programming in Julia

       Vancouver Julia Meetup

            04/20/2021
           Shashi Gowda

         PhD candidate in

Computational Science and Engineering

shashi

using Tullio
using Dagger
using Dagger: DArray, chunks, Thunk, ArrayDomain

struct Zero end
Base.zero(::Type{Thunk}) = delayed(()->Zero())()
Base.zero(::Type{Union{Zero, ArrayDomain}}) = Zero()
Base.:(+)(z::Zero, x) = x
Base.:(+)(x, z::Zero) = x
Base.:(+)(z::Zero, x::Zero) = z

shashi

shashi

shashi

immutable IRef{T}
    value::T
    IRef() = new()
    IRef(value) = new(value)
end

immutable Result{T,S}
    issuccess::Bool
    value::IRef{T}
    error::IRef{S}

shashi

using Lazy

function methoddoc(name, sign, doc)
   vbox(
    fontweight(bold, code(string(name) * "(" * join(map(string, sign.types), ", ") *")" )),
    doc
   )
end

function showdoc(fn::Function, fndoc)

shashi

shashi

(function() {
  function Sneaky (node) {
    var lightNode = Polymer.dom(node)
    this.lightNode = lightNode
    this.node = lightNode.node
  }

  Sneaky.prototype = Polymer.dom()

  Object.defineProperties(Sneaky.prototype, {

shashi

# An infinite number of mathematicians walk into a bar...

using Color

colors = distinguishable_colors(9)

box(w, h, n) =
    empty |> fillcolor(colors[n % 9 + 1]) |> size(w, h)