Skip to content

Instantly share code, notes, and snippets.

View devlatte's full-sized avatar

Eunseon Song devlatte

View GitHub Profile
@devlatte
devlatte / kinesis-firehose-lambda-newline-arrivedts.js
Last active December 29, 2020 14:10
kinesis firehose transformation, origin: json log, add kinesis approximateArrivalTimestamp to log, append newline string for s3 json line
exports.handler = async (event, context) => {
const output = event.records.map((record) => {
const arrived_ts = record.approximateArrivalTimestamp;
const buff = Buffer.from(record.data, 'base64');
const obj = JSON.parse(buff.toString());
obj.arrived_ts = arrived_ts;
record.data = Buffer.from(JSON.stringify(obj) + "\n").toString('base64');
record.result = 'Ok';
return record;
@devlatte
devlatte / thousand separator.sql
Created July 29, 2020 04:11
presto add thousand separator w/ regexp, regular expression
select regexp_replace('100000', '(\d{1,3})(?=(\d{3})+(?!\d))', '$1,');
@devlatte
devlatte / ganglia_timezone_config.php
Created November 23, 2017 06:52
Ganglia web timezone setting
# when 'browser timezone' is not working or not supported ganglia version
# ganglia/conf.php
# for graph(rddtool) timezone
$conf['rrdtool'] = "env TZ='Asia/Seoul' /usr/bin/rrdtool";
# ganglia/header.php
# for header (query) time
<?php
@devlatte
devlatte / logstash_rename_empty_key.conf
Last active April 28, 2017 08:10
logstash rename json empty key field using ruby filter
filter{
ruby {
code => "h = event.get('[record][param]').to_hash; if h.has_key?('') then h['_LS_EMPTY_KEY']=h['']; h.delete(''); event.set('[record][param]', h) end"
}
}
# rename empty key field using ruby filter
# solution for error msg "name cannot be empty string" in elasticsearch output plugin
@devlatte
devlatte / logstash_split_regexp.conf
Created April 13, 2017 06:11
logstash filter string split with control unicode character(\001, start of heading), or regexp
filter{
ruby {
code => "event.set('message', event.get('message').split(/\001\s+/, -1)); event.set('@timestamp', LogStash::Timestamp.now)"
add_field => {
"time" => "%{[message][0]}"
"level" => "%{[message][1]}"
"thread" => "%{[message][2]}"
"msg" => "%{[message][3]}"
"location" => "%{[message][4]}"
"stacktrace" => "%{[message][5]}"
@devlatte
devlatte / logstash5_1_1_plugin_install.sh
Last active February 14, 2018 10:17
logstash 5.1.1 install plugin with proxy (fixed at 5.2+)
//from pure package
export http_proxy="proxy.host:3128"
export https_proxy=$http_proxy
export JRUBY_OPTS="-J-Dhttp.proxyHost=proxy.host -J-Dhttp.proxyPort=3128"
DEBUG=1 JARS_SKIP='true' bin/logstash-plugin install logstash-filter-json_encode