Skip to content

Instantly share code, notes, and snippets.

View newfront's full-sized avatar
:shipit:
Working on open-source and internal projects

Scott Haines newfront

:shipit:
Working on open-source and internal projects
View GitHub Profile
@newfront
newfront / simpleMemoryStreamExample.scala
Last active March 3, 2021 19:29
Simple Memory Stream
test("Should aggregate call events") {
implicit val sqlContext: SQLContext = spark.sqlContext
import spark.implicits._
val scenario = TestHelper.loadScenario[CallEvent](s"$pathToTestScenarios/pdd_events.json")
val scenarioIter = scenario.toIterator
scenario.nonEmpty shouldBe true
val kafkaData = MemoryStream[MockKafkaDataFrame]
val processingStream = EventAggregation(appConfig).process(kafkaData.toDF())(session)
@newfront
newfront / memoryStreamExample.scala
Created March 14, 2019 05:38
MemoryStream Example
class EventMemoryStreamSpec extends FunSuite with Matchers with SparkSqlTest {
val log: Logger = LoggerFactory.getLogger(classOf[EventAggregation])
private val pathToTestScenarios = "src/test/resources/scenarios"
lazy val session: SparkSession = sparkSql
override def conf: SparkConf = {
new SparkConf()
.setMaster("local[*]")
public class SecretSantaSelection {
private final long phoneNumber;
private final String message;
public SecretSantaSelection(final long phoneNumber, final String message) {
this.phoneNumber = phoneNumber;
this.message = message;
}
@newfront
newfront / Helper::Directory usage
Created February 17, 2012 04:46
Ruby Helper Module for Automatic creating directories recursively from a path
#!/usr/bin/env ruby
$: << File.join(File.dirname(__FILE__), "")
$: << $APP_ROOT = File.expand_path(File.dirname(__FILE__))
require $APP_ROOT+"/helper/directory"
@path = File.join($APP_ROOT, '/libs/modules/protected')
Helper::Directory.write(@path) do |result|
unless !result do
# be creative, now that you have written your directory, write some files to it
@newfront
newfront / reverse_dottify.rb
Created July 1, 2011 16:40
Reverse and Dottify a Number for Enum
#!/usr/bin/ruby
#5109926224
#4226299015
#4.2.2.6.2.9.9.0.1.5.1.e164.convorelay.com
#100 10 "u" "E2U+sip" "!^.*$!sip:[email protected]!".
raise "You need to add a 10D number to reverse and dottify as ARGV[0]" unless !ARGV.empty?
$num = ARGV[0]
def enum_reverse_dottify(num)
#Ruby Meetup Group
Ezra
Limitations of SQL
(horizontal databases) - not covered in mysql
- Limitations
- don't scale past a single master
#lesssql
@newfront
newfront / repo initialization info
Created March 30, 2011 19:19
github's directions for initializing a new Git repository
Global setup:
Download and install Git
git config --global user.name "<name>"
git config --global user.email <email>
Next steps:
mkdir dailyhacking
cd dailyhacking
git init
touch README
Frame-type options:
• --keyint <integer> (x264)
• -g <integer> (FFmpeg)
• Keyframe interval, also known as GOP length. This determines the maximum distance between I-frames. Very high GOP lengths will result in slightly more efficient compression, but will make seeking in the video somewhat more difficult. Recommended default: 250
• --min-keyint <integer> (x264)
• -keyint_min <integer> (FFmpeg)
• Minimum GOP length, the minimum distance between I-frames. Recommended default: 25
• --scenecut <integer> (x264)
@newfront
newfront / x264 encoding options
Created March 17, 2011 21:01
x264 --fullhelp
Syntax: x264 [options] -o outfile infile [widthxheight]
Infile can be raw YUV 4:2:0 (in which case resolution is required),
or YUV4MPEG 4:2:0 (*.y4m),
or Avisynth if compiled with support (no).
or libav* formats if compiled with lavf support (no) or ffms support (no).
Outfile type is selected by filename:
.264 -> Raw bytestream
.mkv -> Matroska
.flv -> Flash Video
@newfront
newfront / Fixing GStreamer gst-plugins-ugly Install
Created March 16, 2011 23:39
I had an issue when install gst-plugins-ugly with gstx264enc
Symptoms
--------
Undefined symbols for architecture x86_64:
"_x264_encoder_open_114", referenced from:
_gst_x264_enc_sink_set_caps in libgstx264_la-gstx264enc.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
Fix
------