samklr

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.avro.is.great</groupId>
  <artifactId>protobuff-avro-demo</artifactId>
  <packaging>jar</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>Demo of protobuff integration with Avro</name>
  <build>
    <plugins>

samklr

class HyperLogLogStoreUDAF extends UserDefinedAggregateFunction {

  override def inputSchema = new StructType()
    .add("stringInput", BinaryType)

  override def update(buffer: MutableAggregationBuffer, input: Row) = {
    // This input Row only has a single column storing the input value in String (or other Binary data).
    // We only update the buffer when the input value is not null.
    if (!input.isNullAt(0)) {
      if (buffer.isNullAt(0)) {

samklr

private static String WRITE_STATEMENT = "INSERT INTO avatars (id, image_type, avatar) VALUES (?,?,?);";

private final BoundStatement writeStatement=writeStatement = session.prepare(WRITE_STATEMENT)
				.setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
try {
  		BoundStatement stmt = driver.getWriteStatement();
			stmt.enableTracing();
			stmt.setLong("id", accountId);
			stmt.setString("image_type", image.getType());
			stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));

samklr

Step by step guide for multi node Confluent Kafka Platform and Cassandra cluster;

It is a multi node deployment of https://github.com/ferhtaydn/sack

Assume that, we have five Ubuntu 14.04 nodes. Their IPs are as follows;

• 12.0.5.4
• 12.0.5.5
• 12.0.5.6
• 12.0.1.170

samklr

//==================================================================
// SPARK INSTRUMENTATION
//==================================================================

import com.codahale.metrics.{MetricRegistry, Meter, Gauge}
import org.apache.spark.{SparkEnv, Accumulator}
import org.apache.spark.metrics.source.Source
import org.joda.time.DateTime
import scala.collection.mutable

samklr

Introduction

This document describes sample process of implementing part of existing Dim_Instance ETL.

I took only Clound Block Storage source to simplify and speedup the process. I also ignnored creation of extended tables (specific for this particular ETL process). Below are code and final thoughts about possible Spark usage as primary ETL tool.

TL;DR

Implementation

Basic ETL implementation is really straightforward. The only real problem (I mean, really problem) is to find correct and comprehensive Mapping document (description what source fields go where).

samklr

Advanced Functional Programming with Scala - Notes

Copyright © 2017 Fantasyland Institute of Learning. All rights reserved.

1. Mastering Functions

A function is a mapping from one set, called a domain, to another set, called the codomain. A function associates every element in the domain with exactly one element in the codomain. In Scala, both domain and codomain are types.

val square : Int => Int = x => x * x

samklr

<configuration>
 <property>
  <name>fs.s3a.access.key</name>
  <description>AWS access key ID. Omit for Role-based authentication.</description>
  <value>YOUR_ACCESS_KEY</value>
 </property>

 <property>
  <name>fs.s3a.secret.key</name>
  <description>AWS secret key. Omit for Role-based authentication.</description>

samklr

How many times shouldn't it happen...

-- https://news.ycombinator.com/item?id=11396045

SELECT count(*)
FROM (SELECT id, repo_name, path
        FROM [bigquery-public-data:github_repos.sample_files]
 ) AS F

samklr

package test

import scala.reflect.runtime.universe._

object ReflectionHelpers extends ReflectionHelpers

trait ReflectionHelpers {

  protected val classLoaderMirror = runtimeMirror(getClass.getClassLoader)