Skip to content

Instantly share code, notes, and snippets.

View mlehman's full-sized avatar

Matt Lehman mlehman

43 followers · 29 following
View GitHub Profile
@mlehman
mlehman / hgrep.sh
Created June 22, 2016 18:00
history grep
# utility function for your profile to search history
hgrep(){
history | grep $1 | sort -k 2 | uniq -c -f 1 | sort
}
@mlehman
mlehman / ejson.sh
Last active December 14, 2015 18:22
MongoDB to Extended JSON with SED
sed -e 's/NumberLong("*\(-*[[:digit:]]*\)"*)/{ "$numberLong" : "\1" }/' -e 's/ObjectId("*\([[:alnum:]]*\)"*)/{ "$oid" : "\1
" }/'
@mlehman
mlehman / gist:69dc9eaa1e254080833c
Created August 2, 2014 15:05
Save Case Class as TSV On Spark
implicit class ProductRDD[T <: Product](rdd: RDD[T]) {
/* Saves a RDD of Tuples into a TSV.
* Ex: Employee(emp_id = 123, Name(first="Bob",last="Smith")) => "123\tBob\tSmith"
*/
def saveAsTsv(path: String) {
rdd.map(p => p.productIterator.flatMap {
case a: Product => a.productIterator //flattens nested case classes
case b => Seq(b)
}.mkString("\t"))
@mlehman
mlehman / MultipleOutputsExample.scala
Last active April 11, 2022 06:54
Hadoop MultipleOutputs on Spark Example
/* Example using MultipleOutputs to write a Spark RDD to multiples files.
Based on saveAsNewAPIHadoopFile implemented in org.apache.spark.rdd.PairRDDFunctions, org.apache.hadoop.mapreduce.SparkHadoopMapReduceUtil.
val values = sc.parallelize(List(
("fruit/items", "apple"),
("vegetable/items", "broccoli"),
("fruit/items", "pear"),
("fruit/items", "peach"),
("vegetable/items", "celery"),
("vegetable/items", "spinach")
@mlehman
mlehman / gist:8772354
Last active August 29, 2015 13:55
Secondary Sort in Avro using Pair
import org.apache.avro.Schema;
import org.apache.avro.io.BinaryData;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapred.Pair;
import org.apache.avro.mapreduce.AvroJob;
import org.apache.avro.reflect.ReflectData;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.RawComparator;
@mlehman
mlehman / solrclj-example.clj
Created December 29, 2010 00:03
;; Embedded Solr Example
(def server (create-solr-server
{:type :embedded :core "mycore" :dir "/home-path"}))
;; Http Solr Example
(def server (create-solr-server
{:type :http :host "localhost"}))
;; Adding Documents
(add server {:title "Don Quixote"
@mlehman
mlehman / clj-freckle-example.clj
Created December 28, 2010 17:56
(def my-credentials
(struct credentials "my-token" "my-client-name"))
(get-users my-credentials)
(get-projects my-credentials)
(get-tags my-credentials)
(get-entries my-credentials)
(get-entries my-credentials
{:to "iso-date-string" :from "iso-date-string"})
@mlehman
mlehman / Fluent.Toc.Receive.cs
Created June 2, 2010 16:44
TocClient tc = new TocClient();
tc.Message += new MessageEventHandler(OnMessage);
tc.SignOn("myscreenname","password");
tc.StartListening();
...
protected void OnMessage(object sender, MessageEventArgs e){
WriteLine("{0}: {1}",e.From, e.Message);
}
@mlehman
mlehman / Fluent.Toc.Send.cs
Created June 2, 2010 16:43
TocClient tc = new TocClient();
tc.SignOn("myscreenname","password");
tc.Send("mybuddy","Hello!");
tc.SignOff();
@mlehman
mlehman / Fluent.LogicTemplates.aspx
Created April 22, 2010 22:59
Fluent.LogicTemplates Example
<%@ Register TagPrefix="fluent" Namespace="Fluent.LogicTemplates" Assembly="Fluent.LogicTemplates" %>
<html>
<head>
<script language="C#" runat="server">
private void Page_Load(object sender, System.EventArgs e) {
if(!IsPostBack){
BindDataList();
}
}