Skip to content

Instantly share code, notes, and snippets.

@destan
destan / ParseRSAKeys.java
Last active October 28, 2024 07:07
Parse RSA public and private key pair from string in Java
import java.io.IOException;
import java.net.URISyntaxException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.KeyFactory;
import java.security.NoSuchAlgorithmException;
import java.security.PrivateKey;
import java.security.interfaces.RSAPublicKey;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.PKCS8EncodedKeySpec;
@snowindy
snowindy / spark-create-rdd-from-s3-parallel.scala
Last active September 24, 2020 12:25
This code allows parallel loading of data from S3 to Spark RDD. Support multiple paths to load from. Based on http://tech.kinja.com/how-not-to-pull-from-s3-using-apache-spark-1704509219
val s3Paths = "s3://yourbucket/path/to/file1.txt,s3://yourbucket/path/to/directory"
val pageLength = 100
val key = "YOURKEY"
val secret = "YOUR_SECRET"
import com.amazonaws.services.s3._, model._
import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.services.s3.model.ObjectListing
import scala.collection.JavaConverters._
import scala.io.Source
@andershammar
andershammar / install-apache-zeppelin-on-amazon-emr.sh
Last active October 9, 2018 03:31
Bootstrap script for installing Apache Zeppelin on an Amazon EMR Cluster. Verfied on Amazon EMR release 4.x.
#!/bin/bash -ex
if [ "$(cat /mnt/var/lib/info/instance.json | jq -r .isMaster)" == "true" ]; then
# Install Git
sudo yum -y install git
# Install Maven
wget -P /tmp http://apache.mirrors.spacedump.net/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz
sudo mkdir /opt/apache-maven
sudo tar -xvzf /tmp/apache-maven-3.3.3-bin.tar.gz -C /opt/apache-maven
@bigsnarfdude
bigsnarfdude / gist:d9c0ceba1aa8c1cfa4e5
Last active November 7, 2016 03:38
Experimenting with the Spark Shell and S3 read write
// in the spark shell here i load the file from S3
val myFile = sc.textFile("s3://some-s3-bucket/us-constitution.txt")
// Classic wordcount
val counts = myFile.flatMap(line => line.toLowerCase().replace(".", " ").replace(",", " ").split(" ")).map(word => (word, 1L)).reduceByKey(_ + _)
// create tuples for the words
val sorted_counts = counts.collect().sortBy(wc => -wc._2)
// print out a sample of 10 to see results
@sebsto
sebsto / gist:19b99f1fa1f32cae5d00
Created August 8, 2014 15:53
Install Maven with Yum on Amazon Linux
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven
mvn --version
@keyle
keyle / HDD.groovy
Created August 4, 2014 10:34
Groovy. Write objects to file (json) and read them back
package utils
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
class HDD {
static save(Object content, String filePath) {
new File(filePath).write(new JsonBuilder(content).toPrettyString())
}
@neolitec
neolitec / BasicAuthenticationFilter.java
Created February 12, 2014 11:09
HTTP Basic authentication Java filter
package com.neolitec.examples;
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;