jaceklaskowski

DC/OS

1. The latest documentation at https://dcos.io/docs/latest.

Mesosphere DC/OS 1.9

1. Announcing Mesosphere DC/OS 1.9
2. DC/OS 1.9.0 Release Candidate 2
3. Bringing Production-Proven Data Services to DC/OS 1.9 with our Partners

• DC/OS provides one-click installation of data services such as databases, message queues, and analytics engines, on-par with cloud providers such as Amazon Web Services.

jaceklaskowski

Spark-a-thon — Development Activities

Spark Structured Streaming

1. Developing custom Source to handle formats like XML
   • Publish the streaming source / format to https://spark-packages.org/
   • As a follow-up answer How to read streaming data in XML format from Kafka?
2. Explore what's stored in checkpointLocation
3. How is streaming groupBy different with Append and Complete output modes?
4. Answering the question on StackOverflow — How to count items per time window?

jaceklaskowski

Notes

• Combinators == building blocks
• functions and higher order functions
• composition
• immutable == less moving parts to worry about
• a routine job == a boilerplate == a boring stuff
• a context (so the job varies)
• abstracting away == happening behind the scenes == cutting down repetitive code == eliminating boilerplate
• "The code becomes small, succinct, and more readable"

jaceklaskowski

Parquet

Introduction

• Stores schema information along with the data
• Columnar storage/file format
  • "reference file format on Hadoop HDFS"
  • "read-optimized view of data"
• excellent for local file storage on HDFS (instead of external databases).
• writing very large datasets to disk

jaceklaskowski

Moved the notes to https://github.com/jaceklaskowski/blockchain-notes repository. See you there.

I'm going to remove the gist after Oct, 7th.

jaceklaskowski

Exercise 1

Union only those rows (from large table) with keys in left small table, i.e. union two dataframes together but only those with the key in my small table.

Exercise 2

Aggregation on an array of nested json = How to sum the quantities across all lines for a given order (which would give 1 + 3 = 4 for the below sample dataset):

{

jaceklaskowski

IDEA:

• breaks on demand given the number of exercises
• break man who says we should have one

val wholeJsonRDD = sc.wholeTextFiles("input.json").map(_._2)
val mySchema = new StructType().add($"n".int)
wholeJsonRDD.toDF.withColumn("json", from_json($"value", mySchema)).show(truncate = false)

val jsonDF=spark.read.json("output.json")

jaceklaskowski

Exercise

Develop a Spark standalone application (using IntelliJ IDEA) with Spark MLlib and LogisticRegression to classify emails.

Think about command line and what parameters you'd like to accept for various use cases.

TIP Use scopt

1. libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.1.1"

jaceklaskowski

// Let's create a sample dataset with just a single line, i.e. facebook profile
val facebookProfile = "ActivitiesDescription:703 likes, 0 talking about this, 4 were here; Category:; Email:joe@pvhvac.com; Hours:Mon-Fri: 8:00 am - 5:00 pm; Likes:703; Link:https://www.facebook.com/pvhvac; Location:165 W Wieuca Rd NE, Ste 310, Atlanta, Georgia; Name:PV Heating & Air; NumberOfPictures:0; NumberOfReviews:26; Phone:(404) 798-9672; ShortDescription:We specialize in residential a/c, heating, indoor air quality & home performance.; Url:http://www.pvhvac.com; Visitors:4"
val fbs = Seq(facebookProfile).toDF("profile")

scala> fbs.show(truncate = false)
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

jaceklaskowski

  case class Person(id: Long, name: String)

  class PersonSerializer extends Serializer[Person] {
    override def configure(configs: util.Map[String, _], isKey: Boolean): Unit = {}

    override def serialize(topic: String, data: Person): Array[Byte] = {
      println(s">>> serialize($topic, $data)")
      s"${data.id},${data.name}".map(_.toByte).toArray
    }