Skip to content

Instantly share code, notes, and snippets.

View cloudcalvin's full-sized avatar

Calvin M cloudcalvin

  • Stellenbosch South Africa
View GitHub Profile
@cloudcalvin
cloudcalvin / build.sbt
Created August 25, 2015 20:07
SBT Assembly build settings to allow Spark App to work in jar. Just add to your current settings
assemblyMergeStrategy in assembly := {
case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
case PathList("org", "apache", xs @ _*) => MergeStrategy.last
case PathList("com", "google", xs @ _*) => MergeStrategy.last
case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
case "about.html" => MergeStrategy.rename
case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
@cloudcalvin
cloudcalvin / createJsonFromHeaders.scala
Created August 17, 2015 17:37
Builld a JSON style mapping in scala using sequence of headers as keys and cvs.split(",") as key_values
//fiels are the array of data values
//column map is the headers
val columnMap = Headers.column_name.zipWithIndex
val jsonMapping = (ListMap[String, Any]()/: columnMap) { (a, f) =>
a + ( "\"" + f._1 + "\"" -> fields(f._2))
fields.foreach(_ match => {case empty : null})
@cloudcalvin
cloudcalvin / JSONSparkSQLCassandra.scala
Last active August 29, 2015 14:27 — forked from helena/JSONSparkSQLCassandra.scala
JSON Integration with Spark SQL and Cassandra
import com.datastax.spark.connector.cql.CassandraConnector
import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.sql.{Row, SQLContext}
/** Spark SQL: Txt, Parquet, JSON Support with the Spark Cassandra Connector */
object SampleJson extends App {
import com.datastax.spark.connector._
import GitHubEvents._
val conf = new SparkConf(true)
@cloudcalvin
cloudcalvin / CaseClassToString.scala
Created August 16, 2015 19:06
Print Scala class members fields with types
object Implicits {
implicit class CaseClassToString(c: AnyRef) {
def toStringWithFields: String = {
val fields = (Map[String, Any]() /: c.getClass.getDeclaredFields) { (a, f) =>
f.setAccessible(true)
a + (f.getName -> f.get(c))
}
s"${c.getClass.getName}(${fields.mkString(", ")})"
@cloudcalvin
cloudcalvin / rayTest.cpp
Last active August 29, 2015 14:26 — forked from sloriot/gist:5291656
Nanoflann vs CGAL KD-tree implementation of photon ray tracer for speed comparison
#include <CGAL/Timer.h>
/// Nanoflann code
#include <nanoflann.hpp>
#include <cstdlib>
#include <iostream>
#include <fstream>
#include <vector>
@cloudcalvin
cloudcalvin / git.sh
Created July 23, 2015 13:28
Git wrapper script to add rsa key specification capability with the -i flag.
#!/bin/bash
# The MIT License (MIT)
# Copyright (c) 2013 Alvin Abad
# https://alvinabad.wordpress.com/2013/03/23/how-to-specify-an-ssh-key-file-with-the-git-command
if [ $# -eq 0 ]; then
echo "Git wrapper script that can specify an ssh-key file
Usage:
git.sh -i ssh-key-file git-command
@cloudcalvin
cloudcalvin / FileRecover.md
Created May 5, 2015 10:43
File Recovery Commands

#After pdfs and docs have been carved from filsystem, the folowing commands can be used to search for specific documents ##Search for PDF containing specific text: find . -name '.pdf' -exec sh -c 'pdftotext -q "{}" - | grep --with-filename --label="{}" --color "Search Query"' ; ##Search for .doc or .DOC files containing specific text: find -name '.doc' | while read -r file; do
catdoc "$file" | grep -H --label="$file" "Search Query" done

Happy Time Go Libraries.

Another curated list like awesome-go.
Not complete. Not authoritative. Not cupcake.
Send suggestions: @squarism :)
☆ = Github stars (in September 2014)


Adapters and Drivers

@cloudcalvin
cloudcalvin / etcd.service-unit
Last active August 29, 2015 14:12
etcd2.0 setup
- name: etcd.service
command: start
content: |
[Unit]
Description=etcd
Requires=setup-network-environment.service
After=setup-network-environment.service
[Service]
EnvironmentFile=/etc/environment
User=etcd
@cloudcalvin
cloudcalvin / network-environment-setup.sh
Last active August 29, 2015 14:12
Coreos bare metal network environment setup for IPv4 variable access
#!/bin/bash
ENV="/etc/environment"
HOSTS="/etc/hosts"
# Test for RW access to $1
touch $HOSTS
touch $ENV
if [ $? -ne 0 ]; then
echo exiting, unable to modify: $ENV