Skip to content

Instantly share code, notes, and snippets.

View bigorn0's full-sized avatar
💭
Enthousiast

Ugo Bechameil bigorn0

💭
Enthousiast
10 followers · 14 following
  • UB1
  • Brussels
View GitHub Profile
@bigorn0
bigorn0 / Endpoint response time analysis with curl
Created May 18, 2016 07:55
bash-4.3# curl -4 \
--insecure \
-sSf \
-w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nAppCon time:\t%{time_appconnect}\nRedirect time:\t%{time_redirect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' \
-o /dev/null
-XPUT https://my-endpoint
@bigorn0
bigorn0 / tools.md
Last active February 20, 2017 11:29
@bigorn0
bigorn0 / git archive
Created July 22, 2016 10:00
PRO TIP: Transfer git repo without `.git` folder using tar.gz | ssh `git archive HEAD | gzip -c | ssh <host> 'tar -zxvf -'
@bigorn0
bigorn0 / go-tools.md
Last active February 14, 2017 14:25
@bigorn0
bigorn0 / spark_logging-log4j.md
Last active July 28, 2016 22:32

How to split/redirect part of the logs based on level

apache-spark-stderr-and-stdout

Try this in log4j.properties passed to Spark (or modify default configuration under Spark/conf)

# Log to stdout and stderr
log4j.rootLogger=INFO, stdout, stderr
@bigorn0
bigorn0 / docker-security.md
Created August 24, 2016 21:05
@bigorn0
bigorn0 / Run_spark_submit_from_scala.md
Last active August 24, 2016 21:08
Run_spark_submit_from_scala.md

Running spark jogs from scala code

http://henningpetersen.com/post/22/running-apache-spark-jobs-from-applications

Example out of spark test classes

// NOTE: This is an expensive operation in terms of time (10 seconds+). Use sparingly.
  private def runSparkSubmit(args: Seq[String]): Unit = {
    val sparkHome = sys.props.getOrElse("spark.test.home", fail("spark.test.home is not set!"))
    val process = Utils.executeCommand(
 Seq("./bin/spark-submit") ++ args,
@bigorn0
bigorn0 / dood_cleanup.sh
Created August 25, 2016 22:25
Cleanup Dood
#!/bin/sh
str_in_list() {
str="$1"
shift
list="$@"
if test "${list#*$str}" != "$list"
then
return 0 # $str is in $list
else
@bigorn0
bigorn0 / Second approach
Last active November 28, 2016 21:38
import java.sql._
dataframe.coalesce("NUMBER OF WORKERS").mapPartitions((d) => Iterator(d)).foreach { batch =>
val dbc: Connection = DriverManager.getConnection("JDBCURL")
val st: PreparedStatement = dbc.prepareStatement("YOUR PREPARED STATEMENT")
batch.grouped("# Of Rows you want per batch").foreach { session =>
session.foreach { x =>
st.setDouble(1, x.getDouble(1))
st.addBatch()
}
@bigorn0
bigorn0 / 1-restify-server-cheatsheet.js
Created November 28, 2016 21:27 — forked from LeCoupa/1-restify-server-cheatsheet.js
Restify CheatSheet. More: http://journal.gentlenode.com/node-3-restify-cheatsheet/
// Restify Server CheatSheet.
// More about the API: http://mcavage.me/node-restify/#server-api
// Install restify with npm install restify
// 1.1. Creating a Server.
// http://mcavage.me/node-restify/#Creating-a-Server
var restify = require('restify');