Skip to content

Instantly share code, notes, and snippets.

@ankyhe
ankyhe / SparkWithHadoop.md
Created November 26, 2019 02:55 — forked from afranzi/SparkWithHadoop.md
Spark 2.4.0 with Hadoop 2.8.5

Setup Environmnet variables for Hadoop.

export HADOOP_VERSION=2.8.5
export HADOOP_HOME=${HOME}/hadoop-$HADOOP_VERSION
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=${HADOOP_HOME}/bin:$PATH

Download Hadoop files.

@ankyhe
ankyhe / linux_performance_monitoring
Last active March 12, 2016 03:56
Linux process performance monitoring
Zsh only.
TIMEFMT='%J %U user %S system %P cpu %*E total'$'\n'\
'avg shared (code): %X KB'$'\n'\
'avg unshared (data/stack): %D KB'$'\n'\
'total (sum): %K KB'$'\n'\
'max memory: %M MB'$'\n'\
'page faults from disk: %F'$'\n'\
'other page faults: %R'
time ls
@ankyhe
ankyhe / git_tips
Last active August 29, 2015 13:58
git tips
-- Diffmerge as Diff in OSX --
1. Download diffmerge (https://sourcegear.com/diffmerge/), I suggest donwload dmg file
2. Copy diffmerge/extra/diffmerge into /usr/local/bin
3. sudo vim /usr/local/bin/dmdiff, which content is as below:
#!/bin/sh
/usr/local/bin/diffmerge "$2" "$5" >& /dev/null
4. vim ~/.gitconfig:
[diff]
external = dmdiff
tool = diffmerge