Skip to content

Instantly share code, notes, and snippets.

View tamtam180's full-sized avatar

tamtam180 tamtam180

31 followers · 3 following
View GitHub Profile
@tamtam180
tamtam180 / gist:3493811
Created August 28, 2012 00:31
# coding: utf-8
require 'rubygems'
require 'optparse'
require 'json'
require 'time'
require 'msgpack'
require 'zlib'
require 'fileutils'
@tamtam180
tamtam180 / in_tail.py
Created October 19, 2012 07:04
fluenpyをwindowsで動かすためのin_tail (後で消します)
# coding: utf-8
"""
fluenpy.plugins.in_tail
~~~~~~~~~~~~~~~~~~~~~~~~
tail from rotated log.
:copyright: (c) 2012 by INADA Naoki
:license: Apache v2
"""
@tamtam180
tamtam180 / gist:3947586
Created October 24, 2012 17:41
create table hoge(
id bigint unsigned not null,
primary key pk_hoge (id)
)engine=InnoDB;
pager(slider)
< 1 2 3 ... 100 >
例えば100万件ある場合、
50を押すと50万番目から表示する。
70を押すと70万番目から表示する。
@tamtam180
tamtam180 / topology.pl
Created December 9, 2012 08:04
Hadoop rack_awarenessのperlスクリプト
#!/usr/bin/perl
use warnings;
use strict;
use Socket;
our $TOPOLOGY_FILE = "/etc/hadoop/conf/rack.txt";
our $DEFAULT_RACK = "/default-rack";
my %topologies;
@tamtam180
tamtam180 / gist:4325800
Created December 18, 2012 07:23
<match game.**>
type copy
<store>
type stdout
</store>
<store>
type forest
subtype file_alternative
remove_prefix game
<template>
@tamtam180
tamtam180 / HADOOP-xxxx.diff
Created December 20, 2012 07:21
diff --git hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp
index 9a16f23..f153ee6 100644
--- hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp
+++ hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp
@@ -30,6 +30,7 @@
<!DOCTYPE html>
<html>
<head>
+<link rel="stylesheet" type="text/css" href="/static/hadoop.css">
<%JspHelper.createTitle(out, request, request.getParameter("filename")); %>
@tamtam180
tamtam180 / HDFS-4344.1.patch
Last active December 10, 2015 06:18
Hadoopの不具合をJIRAに書く前の下書き・校正メモ。
diff --git hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
index 93bf5ff..154fce5 100644
--- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
+++ hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
@@ -967,7 +967,7 @@ private DatanodeID parseDNFromHostsEntry(String hostLine) {
port = DFSConfigKeys.DFS_DATANODE_DEFAULT_PORT;
} else {
hostStr = hostLine.substring(0, idx);
- port = Integer.valueOf(hostLine.substring(idx));
+ port = Integer.valueOf(hostLine.substring(idx + 1));
@tamtam180
tamtam180 / _aaa.rb
Last active December 12, 2015 23:08
msgpackがセグっためも。
# coding: utf-8
require 'msgpack'
test_records = Array.new
open("abc.txt") do |f|
f.each_line do | line |
line.chomp!
test_records << line
end
@tamtam180
tamtam180 / seg.txt
Last active December 13, 2015 17:39
*** glibc detected *** ruby: munmap_chunk(): invalid pointer: 0x0000000002674c50 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75916)[0x7f75c06e9916]
/usr/local/rvm/gems/ruby-1.9.3-p327/gems/msgpack-0.5.2/lib/msgpack/msgpack.so(_msgpack_buffer_shift_chunk+0x5c)[0x7f75b9f749cc]
/usr/local/rvm/gems/ruby-1.9.3-p327/gems/msgpack-0.5.2/lib/msgpack/msgpack.so(msgpack_buffer_clear+0x10)[0x7f75b9f74c30]
/usr/local/rvm/gems/ruby-1.9.3-p327/gems/msgpack-0.5.2/lib/msgpack/msgpack.so(MessagePack_pack+0x117)[0x7f75b9f77747]
/usr/local/rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9(+0x16a44f)[0x7f75c165544f]
/usr/local/rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9(+0x16bc9d)[0x7f75c1656c9d]
/usr/local/rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9(+0x171219)[0x7f75c165c219]
/usr/local/rvm/rubies/ruby-1.9.3-p327/lib/libruby.so.1.9(+0x174f89)[0x7f75c165ff89]
@tamtam180
tamtam180 / gist:4983766
Created February 19, 2013 07:26
in_status を stdoutに出力したら...
2013-02-19 16:22:25 +0900: failed to configure/start sub output webhdfs: can't add a new key into hash during iteration2013-02-19 16:22:24 +0900 status:
{"queue_size":0,"emit_count":0}
2013-02-19 16:22:24 +0900 status: {"queue_size":0,"emit_count":0}
2013-02-19 16:22:25 +0900: /usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluentd-0.10.31/lib/fluent/status.rb:28:in `[]='
/usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluentd-0.10.31/lib/fluent/status.rb:28:in `register'
/usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluentd-0.10.31/lib/fluent/output.rb:203:in `configure'
/usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluentd-0.10.31/lib/fluent/output.rb:449:in `configure'
/usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluent-mixin-config-placeholders-0.2.0/lib/fluent/mixin/config_placeholders.rb:93:in `configure'
/usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluent-mixin-plaintextformatter-0.2.1/lib/fluent/mixin/plaintextformatter.rb:18:in `configure'
/usr/local/rvm/gems/ruby-1.9.3-p385/gems/fluent-plugin-webhdfs-0.1.1/lib/fluent/plugin/out_we