Skip to content

Instantly share code, notes, and snippets.

View softberries's full-sized avatar

Kris softberries

24 followers · 1 following
View GitHub Profile
@softberries
softberries / aws-rekognition-with-iot
Created June 8, 2017 09:16
Lambda funtion for comparing photos and publishing back to IoT
from __future__ import print_function
import boto3
from decimal import Decimal
import json
import urllib
print('Loading function')
rekognition = boto3.client('rekognition')
@softberries
softberries / read-rfid-on-rpi.py
Last active June 11, 2019 09:25
RFID reading on raspberry pi
import sys
hid = { 4: 'a', 5: 'b', 6: 'c', 7: 'd', 8: 'e', 9: 'f', 10: 'g', 11: 'h', 12: 'i', 13: 'j', 14: 'k', 15: 'l', 16: 'm', 17: 'n', 18: 'o', 19: 'p', 20: 'q', 21: 'r', 22: 's', 23: 't', 24: 'u', 25: 'v', 26: 'w', 27: 'x', 28: 'y', 29: 'z', 30: '1', 31: '2', 32: '3', 33: '4', 34: '5', 35: '6', 36: '7', 37: '8', 38: '9', 39: '0', 44: ' ', 45: '-', 46: '=', 47: '[', 48: ']', 49: '\\', 51: ';' , 52: '\'', 53: '~', 54: ',', 55: '.', 56: '/' }
hid2 = { 4: 'A', 5: 'B', 6: 'C', 7: 'D', 8: 'E', 9: 'F', 10: 'G', 11: 'H', 12: 'I', 13: 'J', 14: 'K', 15: 'L', 16: 'M', 17: 'N', 18: 'O', 19: 'P', 20: 'Q', 21: 'R', 22: 'S', 23: 'T', 24: 'U', 25: 'V', 26: 'W', 27: 'X', 28: 'Y', 29: 'Z', 30: '!', 31: '@', 32: '#', 33: '$', 34: '%', 35: '^', 36: '&', 37: '*', 38: '(', 39: ')', 44: ' ', 45: '_', 46: '+', 47: '{', 48: '}', 49: '|', 51: ':' , 52: '"', 53: '~', 54: '<', 55: '>', 56: '?' }
fp = open('/dev/hidraw0', 'rb')
ss = ""
shift = False
done = False
while not done:
@softberries
softberries / rpi_rfid_rekogntion.py
Created June 9, 2017 12:57
Access Control System with Raspberry Pi, RFiD and AWS Rekogintion
from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient
import sys
import logging
import time
import getopt
from datetime import datetime
import picamera
import os
import tinys3
import json
@softberries
softberries / sites.txt
Created October 23, 2017 12:01
example sites to be processed
3dprinting.meta.stackexchange.com
3dprinting.stackexchange.com
academia.meta.stackexchange.com
academia.stackexchange.com
ai.meta.stackexchange.com
ai.stackexchange.com
android.meta.stackexchange.com
android.stackexchange.com
anime.meta.stackexchange.com
anime.stackexchange.com
@softberries
softberries / prepare-data.sh
Created October 23, 2017 12:08
prep data bash script
#!/usr/bin/env bash
S3_DESTINATION_FOLDER="grajo001log/users"
SITES_FILE=sites.txt
IFS=$'\n'
for site in `cat $SITES_FILE`
do
echo "$site"
wget https://archive.org/download/stackexchange/$site.7z
@softberries
softberries / README.md
Created October 23, 2017 12:10
Command to get the size of the bucket

aws s3api list-objects --bucket YOUR_BUCKET_NAME --output json --query "[sum(Contents[].Size), length(Contents[])]" | awk 'NR!=2 {print $0;next} NR==2 {print $0/1024/1024/1024" GB"}'

@softberries
softberries / AgeCounterJob.scala
Created October 23, 2017 12:12
Scalding Job Definition
class AgeCounterJob(args: Args) extends Job(args) {
val lines: TypedPipe[String] = TypedPipe.from(TextLine(args("input")))
val tokens: TypedPipe[Token] = lines.flatMap(f => xmlToToken(f))
val byAge = tokens.groupBy(_.age)
byAge.size
.write(TypedTsv[(Int, Long)](args("output")))
@softberries
softberries / README.md
Created October 23, 2017 12:14
run the job locally

hadoop jar target/scala-2.11/emr-scalding-tutorial-assembly-0.1.jar com.softwaremill.AgeCounterJob — hdfs — input “data/*” — output data-output

@softberries
softberries / README.md
Created October 23, 2017 12:15
run the job locally ver2

hadoop jar target/scala-2.11/emr-scalding-tutorial-assembly-0.1.jar com.softwaremill.AgeCounterJob — local — input “data/hello.txt” — output data-output.txt

@softberries
softberries / README.md
Created October 23, 2017 12:18
copy the jar file to s3 location

aws s3 cp target/scala-2.11/emr-scalding-tutorial-assembly-0.1.jar s3://grajo001log/emr-scalding-tutorial-assembly-0.1.jar