Skip to content

Instantly share code, notes, and snippets.

View myouju's full-sized avatar
💭
i am data engineer

Yuki Maeno myouju

💭
i am data engineer
6 followers · 0 following
View GitHub Profile
@myouju
myouju / setup_classpath_tez.sh
Created January 22, 2016 05:35
emrでhive on tezをするときにclasspathをちゃんと設定する方法. bootstrapで実行する
#!/bin/bash
export HADOOP_USER_CLASSPATH_FIRST=true;
echo "
export TEZ_CONF_DIR=/etc/tez/conf
export TEZ_HOME=/home/hadoop/tez
export TEZ_JARS=/apps/tez/tez-0.4.1/*:/apps/tez/tez-0.4.1/lib/*
export TEZ_LOG_DIR=/home/hadoop/tez/log
export HADOOP_CLASSPATH=\${TEZ_CONF_DIR}:\${TEZ_JARS}/*:\${TEZ_JARS}/lib/*:\${HADOOP_CLASSPATH}" >> /home/hadoop/conf/hadoop-user-env.sh
@myouju
myouju / .drone.yml
Last active October 7, 2016 06:42
drone.ioでhiveをunit testする
image: localhost/maeno/hadoop
script:
# execute unittest
- export PATH=$HADOOP_PREFIX/bin:$HIVE_HOME/bin:$PATH
- UNIT_TEST_LIST=`find ./test -name "setup.hql"|xargs -Idir dirname dir|sort -u`
- for u in `echo $UNIT_TEST_LIST`;do sh /tmp/run-unittest.sh ${u}; done
- if [ ${DRONE_BRANCH} != "master" ]; then exit 0; fi
- mkdir -p ~/.aws/
- echo -e "[profile release]\nregion = ap-northeast-1\naws_access_key_id = $$AWS_KEY\naws_secret_access_key = $$AWS_SECRET" | tee ~/.aws/config
- aws --profile release s3 cp src/ s3://<s3 bucket>/hiveql/ --recursive --exclude "*" --include "*.q"
@myouju
myouju / Dockerfile
Last active February 26, 2016 05:14
僕が考えた最強のCUI twitterクライアント(白目
FROM dahlb/alpine-node
RUN npm install twitter user-stream coffee-script
ADD stream stream
ENTRYPOINT ./node_modules/coffee-script/bin/coffee stream
@myouju
myouju / Dockerfile
Created February 29, 2016 10:27
ip listからgeoipを使って国名を取得する
FROM ubuntu-debootstrap:14.04
MAINTAINER yuki-maeno
RUN echo "Asia/Tokyo" > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata \
&& apt-get update \
&& apt-get install --no-install-recommends -y git python python-pip liblua5.1-dev libssl-dev python-dev libevent-dev libgeoip-dev wget build-essential \
&& apt-get clean \
&& rm -fr /var/cache/apt/archives/* /var/lib/apt/lists/*
RUN wget https://pypi.python.org/packages/source/G/GeoIP/GeoIP-1.3.2.tar.gz \
&& tar zxvf GeoIP-1.3.2.tar.gz \
@myouju
myouju / enc_dec.go
Last active April 29, 2024 09:42
encryption - encrypt / decrypt by ruby, python, and golang with AES-256-CFB
package main
import (
"fmt"
"io"
"encoding/base64"
"crypto/rand"
"crypto/cipher"
"crypto/aes"
"crypto/md5"
@myouju
myouju / gist:1ebfc4691eb781c50ed0939c39336b2f
Last active October 4, 2016 09:36
#!/usr/bin/env bash
SCRIPT=`basename $0`
function usage () {
echo ""
echo "$SCRIPT -s [security group id] -a [aws cli path]"
echo ""
}
@myouju
myouju / a.md
Last active October 26, 2016 02:40

ビッグデータのリアルタイム処理技術勉強会

ストリーム処理を支えるキューイングシステムの選び方

  • Kafka、Redis、Nats、ZeroMQ、RabbitMQとかのキューイングシステムの紹介
  • ブローカーの有無
  • メッセージのシャーディングとか

周辺クライアント

  • 標準プロトコルお実装あactiveMQとRabbitMQだけ
  • 独自相成りベース Kafka
@myouju
myouju / polling.go
Last active June 8, 2017 06:33
func polling(interval, maxTime int) error {
ticker := time.NewTicker(time.Duration(interval) * time.Second)
timeout := time.NewTimer(time.Duration(maxTime) * time.Second)
defer timeout.Stop()
defer ticker.Stop()
for {
select {
case <-timeout.C:
return errors.New("timeout")
case <-ticker.C:
@myouju
myouju / diff_test_out_s3.rb.md
Created September 12, 2017 02:59 
$ git diff
diff --git a/test/test_out_s3.rb b/test/test_out_s3.rb
index 4e8438c..c7be37f 100644
--- a/test/test_out_s3.rb
+++ b/test/test_out_s3.rb
@@ -187,6 +187,22 @@ class S3OutputTest < Test::Unit::TestCase
     assert_equal(expected, d.formatted)
   end
@myouju
myouju / .gitignore
Last active December 24, 2019 04:18
A Go dependency graph visualization tool (golangのpackage依存を可視化)
mo
*.html