Skip to content

Instantly share code, notes, and snippets.

View xman1980's full-sized avatar

Vitali xman1980

2 followers · 3 following
View GitHub Profile
@xman1980
xman1980 / gist:5e892475dc5872212bc093f5e777cb0c
Created June 9, 2017 08:37
kill-all-yarn-jobs
for app in $(yarn application -list | awk '$6 == "RUNNING" { print $1 }')
do
echo "killing $app ...";
yarn application -kill "$app";
done
@xman1980
xman1980 / gist:c45061f5dcc8d04de3750ea454ff57c5
Created July 7, 2017 07:17
azkaban-workers-safe-restart
> also we found a safe way to restart executor
> We can call "curl http://localhost:12321/executor?action=shutdown"
> It puts executor to some shutown sate, but wait till evey job finishes
> But you need to check with curl http://localhost:12321/executor?action=ping if executor is still alive,
@xman1980
xman1980 / gist:ffa8f8056852a46b1aaf950e8116606f
Created August 30, 2017 08:23
clean_old_hdfs_search
hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-job.jar org.apache.solr.hadoop.HdfsFindTool -find /user/spark/applicationHistory -mtime +7 | xargs hdfs dfs -rm -r
@xman1980
xman1980 / gist:d2e56189b36345d8b9382c383b815bca
Created October 9, 2017 07:49
spark example test
/opt/hadoop/spark/bin/run-example SparkPi
@xman1980
xman1980 / gist:89e5373e3348399d4469c8af41d47510
Last active October 10, 2017 14:16
parted aditional hdd linux #mkpart
parted -s /dev/sdf mklabel gpt -- mkpart primary ext4 1 -1
@xman1980
xman1980 / gist:c04a6d4ceb9506571f805efc078f7317
Created January 9, 2018 11:47
scan jars for class
for i in *.jar; do jar -tvf "$i" | grep -Hsi ShellBasedUnixGroupsMapping && echo "$i"; done
@xman1980
xman1980 / gist:508abd7e0e7d93d20cf25bb6ae6341ab
Created January 19, 2018 09:07
parted additional disk
parted -s -a optimal /dev/sdm mklabel gpt -- mkpart dn12 ext4 1 -1 && mkfs.ext4 /dev/sdm1
@xman1980
xman1980 / gist:b47b5b0789681fc3c988cd1926e9245a
Created January 19, 2018 10:02
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded hadoop client fix
export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx5g"
@xman1980
xman1980 / gist:7e62e7202d39ea8866d5c6cb0fd38782
Created February 15, 2018 17:01
certbot certificat ask
certbot certonly --webroot --agree-tos --email email@admin.aol -w /var/www/html/ -d domain.com -d www.domain.com
@xman1980
xman1980 / gist:6871e05bc76cd0ef57aa0aba6fb27b82
Created February 15, 2018 17:02
certbot http location if needed
server {
listen 80;
server_name mysite.ru www.mysite.ru;
root /var/www/mysite.ru;
location /.well-known/acme-challenge/ {
access_log off;
default_type "text/plain";
}
location / {
rewrite ^(.*)$ https://mysite.ru permanent;