bugcy013
/ notes.txt
Created
February 21, 2013 17:35
— forked from nipra/notes.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Installing CDH4 on a Single Linux Node in Pseudo-distributed Mode | |
| # https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode | |
| # Installing CDH4 with MRv1 on a Single Linux Node in Pseudo-distributed mode | |
| # On Ubuntu and other Debian systems | |
| nipra@lambda:Downloads$ wget -cv http://archive.cloudera.com/cdh4/one-click-install/precise/amd64/cdh4-repository_1.0_all.deb | |
| nipra@lambda:Downloads$ sudo dpkg -i cdh4-repository_1.0_all.deb # Adds /etc/apt/sources.list.d/cloudera-cdh4.list ?? | |
| nipra@lambda:Downloads$ dpkg -L cdh4-repository # To view the files on Ubuntu systems | |
| # Install CDH4 |
bugcy013
/ gist:5028652
Last active
December 14, 2015 04:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 0.check-selinux.log | |
| 1.install-repo-pkg.log | |
| 2.refresh-repo.log | |
| 3.install-oracle-j2sdk1.6.log | |
| 4.install deameons | |
| sudo ufw enable | |
| root@dhana:/etc/apt/sources.list.d# cat cloudera-cdh4.list |
bugcy013
/ listHadoopJSP.py
Created
March 1, 2013 09:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import glob | |
| import xml.dom.minidom | |
| for f in glob.glob('*/WEB-INF/web.xml'): | |
| print f | |
| dom = xml.dom.minidom.parse(f) | |
| mappings = dom.getElementsByTagName('servlet-mapping') | |
| for m in mappings: | |
| urlp = m.getElementsByTagName('url-pattern')[0].firstChild.data | |
| sname = m.getElementsByTagName('servlet-name')[0].firstChild.data | |
| print " %s (%s)" %(urlp,sname) |
bugcy013
/ check_hdfs.py
Created
March 1, 2013 09:24
— forked from taicki/check_hdfs.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from itertools import islice | |
| import os | |
| import sys | |
| import subprocess | |
| CMD = "sudo -u hdfs /usr/lib/hadoop/bin/hadoop dfsadmin -report" | |
| def parse(resultstr): | |
| dic = {} | |
| for line in islice(resultstr, 0, 8): |
bugcy013
/ set-hadoop-heap.sh
Created
March 1, 2013 09:28
— forked from darkseed/set-hadoop-heap.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| if [ $# -lt 1 ]; then | |
| SIZE="2048" | |
| else | |
| SIZE=$1 | |
| fi | |
| echo "HADOOP_HEAPSIZE=${SIZE}" >> /home/hadoop/conf/hadoop-user-env.sh |
bugcy013
/ find_bad_file_by_block_hdfs.sh
Created
March 1, 2013 09:31
— forked from alexanderlz/find_bad_file_by_block_hdfs.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| sudo -u hdfs hadoop fsck / -files -blocks | grep BAD_BLOCK_ID -B 5 |
bugcy013
/ hdfs_list_running_jobs.sh
Created
March 1, 2013 09:32
— forked from alexanderlz/hdfs_list_running_jobs.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| hadoop job -list | grep job_ | awk 'BEGIN{FS="\t";OFS=","};{print $1,strftime("%H:%M:%S", (systime()-int($3/1000)),1),"\""$4"\"","\""$6"\""}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Task tracker | |
| http://172.16.30.128:50060 | |
| Task tracker metrics info | |
| http://172.16.30.128:50060/metrics | |
| Data node block scanner details | |
| http://172.16.30.128:50075/blockScannerReport | |
| Secondary datanode status |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| research@dvcliftonhera227:/usr/lib/hadoop-0.20-mapreduce$ hadoop jar hadoop-test.jar TestDFSIO -read -nrFiles 10 -fileSize 1000 | |
| 13/03/06 03:27:49 INFO fs.TestDFSIO: TestDFSIO.0.0.6 | |
| 13/03/06 03:27:49 INFO fs.TestDFSIO: nrFiles = 10 | |
| 13/03/06 03:27:49 INFO fs.TestDFSIO: fileSize (MB) = 1000.0 | |
| 13/03/06 03:27:49 INFO fs.TestDFSIO: bufferSize = 1000000 | |
| 13/03/06 03:27:49 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO | |
| SLF4J: Class path contains multiple SLF4J bindings. | |
| SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: Found binding in [jar:file:/app_cloe/cloe_hadoop/cloe/deepvalue/slf4j-simple/1.5.8/slf4j-simple-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. |
bugcy013
/ gist:5106453
Created
March 7, 2013 08:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| root@Datanode-192-168-70-97:~# service cloudera-scm-server-db initdb | |
| root@Datanode-192-168-70-97:~# /etc/init.d/cloudera-scm-server restart | |
| cloudera-scm-server is already stopped | |
| root@Datanode-192-168-70-97:~# service cloudera-scm-server restart | |
| cloudera-scm-server is already stopped | |
| root@Datanode-192-168-70-97:~# service cloudera-scm-server start | |
| Starting cloudera-scm-server: * cloudera-scm-server started | |
| root@Datanode-192-168-70-97:~# cd /var/log/cloudera-scm-server/ | |
| root@Datanode-192-168-70-97:/var/log/cloudera-scm-server# ls -ltr |