Yaroslav Tkachenko sap1ens
sap1ens
/ Sink.scala
Created
March 3, 2022 22:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val records: DataStream[Record] = … | |
| val fileSink: SinkFunction[Record] = … | |
| records.keyBy(_.partition).addSink(fileSink) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val dedupColumn = "..." // column name to use as a key for deduplication | |
| val ttl = Some(Time.minutes(60)) // state TTL | |
| stream | |
| .keyBy(row => row.getField(dedupColumn)) | |
| .flatMap(new RichFlatMapFunction[Row, Row] { | |
| @transient | |
| private var seen: ValueState[Boolean] = _ | |
| override def open(parameters: Configuration): Unit = { |
sap1ens
/ compile-perf.sh
Last active
December 31, 2024 01:11
Compile perf from scratch with a specific kernel version
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apt-get update | |
| apt-get install -y git make gcc flex bison \ | |
| libunwind8-dev libdwarf-dev libelf-dev libdw-dev systemtap-sdt-dev \ | |
| libssl-dev libslang2-dev binutils-dev libzstd-dev libbabeltrace-dev \ | |
| libiberty-dev libnuma-dev libcap-dev | |
| git clone -b v5.10.205 --single-branch -n --depth=1 --filter=tree:0 \ | |
| git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git | |
| cd linux-stable | |
| git sparse-checkout set --no-cone tools scripts | |
| git checkout |
sap1ens
/ IcebergDemo.java
Created
March 21, 2025 23:35
Flink / Iceberg connecting to Confluent Tableflow via REST Catalog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.example.demo; | |
| import org.apache.flink.streaming.api.datastream.DataStream; | |
| import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; | |
| import org.apache.flink.table.data.RowData; | |
| import org.apache.hadoop.conf.Configuration; | |
| import org.apache.iceberg.catalog.Catalog; | |
| import org.apache.iceberg.catalog.Namespace; | |
| import org.apache.iceberg.catalog.TableIdentifier; | |
| import org.apache.iceberg.flink.CatalogLoader; |
OlderNewer