Lars George larsgeorge
larsgeorge
/ gist:822679
Created
February 11, 2011 17:09
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hbase(main):014:0> add_peer '1', 'localhost:2181:/hbase-2' | |
| 0 row(s) in 0.0580 seconds | |
| hbase(main):015:0> start_replication | |
| 2011-02-11 18:04:58,347 INFO org.apache.hadoop.hbase.replication.ReplicationZookeeper: Replication is now started | |
| 0 row(s) in 0.0500 seconds | |
| hbase(main):016:0> put 'test', '2011-02-11 18:05:22,003 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [10.0.0.57,60020,1297437319991] | |
| 2011-02-11 18:05:22,016 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs for 10.0.0.57,60020,1297437319991 | |
| 2011-02-11 18:05:22,016 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=10.0.0.57,60020,1297437319991 to dead servers, submitted shutdown handler to be executed, root=true, meta=true |
larsgeorge
/ gist:822865
Created
February 11, 2011 19:35
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2011-02-11 19:49:11,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10 | |
| 2011-02-11 19:49:22,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10 | |
| 2011-02-11 19:49:33,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10 | |
| 2011-02-11 19:49:44,747 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10 | |
| 2011-02-11 19:49:54,964 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB, free=197.54 MB, max=199.18 MB, blocks=4, accesses=55, hits=51, hitRatio=92.72%%, cachingAccesses=55, cachingHits=51, cachingHitsRatio=92.72%%, evictions=0, evicted=0, evictedPerRun=NaN | |
| 2011-02-11 19:49:55,748 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: No log to process, sleeping 1000 times 10 | |
| ==> logs/hbase-larsgeorge-master-d |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| HBase 1: | |
| 506 wget http://apache.easy-webs.de//hbase/hbase-0.90.0/hbase-0.90.0.tar.gz | |
| 507 tar -zxvf hbase-0.90.0.tar.gz | |
| 508 cd hbase-0.90.0 | |
| 522 cp -pR conf conf.2 | |
| 523 vim conf/hbase-site.xml | |
| 524 cp conf/hbase-site.xml conf.2/ | |
| 525 vim conf.2/hbase-site.xml |
larsgeorge
/ Hadoop log4j.properties
Created
August 9, 2011 12:58
Use syslog (on MacOS) with Hadoop's userlog/syslog
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #... | |
| #Default values | |
| hadoop.tasklog.taskid=null | |
| hadoop.tasklog.iscleanup=false | |
| hadoop.tasklog.noKeepSplits=4 | |
| hadoop.tasklog.totalLogFileSize=100 | |
| hadoop.tasklog.purgeLogSplits=true | |
| hadoop.tasklog.logsRetainHours=12 | |
| log4j.appender.TLA=org.apache.log4j.net.SyslogAppender |
larsgeorge
/ hdfs-timing.txt
Created
January 8, 2014 09:22
Measure time it takes to create and read a file in HDFS
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ time dd if=/dev/zero bs=1G count=1 | hadoop fs -put - 1g.bin | |
| 1+0 records in | |
| 1+0 records out | |
| 1073741824 bytes (1.1 GB) copied, 64.4632 s, 16.7 MB/s | |
| real 1m5.239s | |
| user 0m9.560s | |
| sys 0m3.394s | |
| $ time hadoop fs -get 1g.bin - > /dev/null |
larsgeorge
/ gist:3f7daee9f96200d2f229
Last active
January 10, 2017 11:11
HBase Configuration Differences (based on the hbase-default.xml file in each release)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /projects/private/hadoop-config-differ [git:master]$ sh target/bin/run-differ /projects/opensource/hbase/hbase-r1130916/src/main/resources/hbase-default.xml r1130916 /projects/opensource/hbase/hbase-0.92-rw/src/main/resources/hbase-default.xml 0.92 /projects/opensource/hbase/hbase-0.94-rw/src/main/resources/hbase-default.xml 0.94 /projects/opensource/hbase/hbase-0.96-rw/hbase-common/src/main/resources/hbase-default.xml 0.96 /projects/opensource/hbase/hbase-0.98-rw/hbase-common/src/main/resources/hbase-default.xml 0.98 /projects/opensource/hbase/hbase-trunk-rw-git/hbase-common/src/main/resources/hbase-default.xml 0.99 | |
| ========================================================= | |
| Start | |
| ========================================================= | |
| Checking differences across versions... | |
| Added or renamed keys in 0.92: | |
| added: Property{key='dfs.support.append', value='true', description='Does HDFS allow appends to files? This is an hdfs config. set in here so the hdfs client will do append support. You must ensure tha |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| IMPORTANT DISCLAIMER: Many of the found "properties" aren't really useful or mistakenly listed, | |
| like those starting with "org.apache.hadoop.*". The tool simply greps a pattern in the matching | |
| files, and some of them are not used as properties. On the other hand, all named properties from | |
| the given configuration file are removed, so quite a few are valid. Again, some of those are still | |
| hidden on purpose, since it is not useful to tinker with them unless in extreme situations, or | |
| during test runs. Be VERY CAREFUL of what you do. I warned ya! | |
| $ sh target/bin/run-finder -d /projects/opensource/hbase/hbase-trunk-rw-git/ -u -s \ | |
| --exclude **/generated --exclude **/generated/** --exclude **/target \ | |
| --exclude **/test/** -c hbase-common/src/main/resources/hbase-default.xml -o /dev/null |
larsgeorge
/ gist:e5797433a32edd6487886069f150e9c7
Created
June 10, 2016 12:46
Metric Sequence Diagram
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # See: https://www.websequencediagrams.com/ | |
| title HBase Metrics Call Sequence Diagramm | |
| MetricsSystem->MetricsSystem: onTimerEvent() | |
| activate MetricsSystem | |
| MetricsSystem->MetricsSystem: publishMetrics() | |
| MetricsSystem->MetricsSystem: sampleMetrics() | |
| MetricsSystem->+MetricsBufferBuilder: constructor | |
| MetricsBufferBuilder->-MetricsSystem: MetricsBufferBuilder |
larsgeorge
/ presto-histogramm-20161125.log
Created
November 25, 2016 13:08
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2016-11-25T12:39:20.420+0100: 177.558: [SoftReference, 0 refs, 0.0000547 secs]2016-11-25T12:39:20.420+0100: 177.558: [WeakReference, 2766 refs, 0.0004900 secs]2016-11-25T12:39:20.421+0100: 177.558: [FinalReference, 589 refs, 0.0007352 secs]2016-11-25T12:39:20.422+0100: 177.559: [PhantomReference, 0 refs, 182 refs, 0.0000369 secs]2016-11-25T12:39:20.422+0100: 177.559: [JNI Weak Reference, 0.0000666 secs] 177.562: [G1Ergonomics (Concurrent Cycles) do not request concurrent cycle initiation, reason: still doing mixed collections, occupancy: 8925478912 bytes, allocation request: 0 bytes, threshold: 7730941095 bytes (45.00 %), source: end of GC] | |
| 177.562: [G1Ergonomics (Mixed GCs) do not continue mixed GCs, reason: reclaimable percentage not over threshold, candidate old regions: 94 regions, reclaimable: 850530200 bytes (4.95 %), threshold: 5.00 %] | |
| , 0.0895976 secs] | |
| [Parallel Time: 82.9 ms, GC Workers: 33] | |
| [GC Worker Start (ms): Min: 177473.8, Avg: 177474.2, Max: 177474.6, Diff: 0.8] | |
| [Ext Root Scann |
larsgeorge
/ gist:9577d2cef0c8ff39c7c8c6ca2e7b570b
Last active
June 12, 2020 17:13
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -- cleanup previous demo | |
| -- drop database and permissions | |
| drop database if exists marketing cascade INCLUDING PERMISSIONS; | |
| drop database if exists demo cascade INCLUDING PERMISSIONS; | |
| -- ensure roles are all created cleanly | |
| drop role if exists analyst_role; | |
| create role if not exists analyst_role; | |
| grant role analyst_role to group analyst_group; | |
| -- create US role for old view row filter demo. | |
| -- TODO remove once we are only using 2.1 to demo |
OlderNewer