ludflu

1. How to log into slaves

---

One can log into the slave nodes using Agent Forwarding by SSH. Suppose that I find in the YARN log

java.nio.channels.CancelledKeyException
14/09/25 23:12:58 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(ip-172-31-xx-xx.ec2.internal,49972)
java.io.IOException: Connection reset by peer
...

I would like to SSH to the slave ip-172-31-xx-xx.ec2.internal to check the logs. To do so, firstly we have to add EMR identity to the agent

ludflu

We have a large set of medical claims, each of which is assigned an integer that uniquely identifies the claim as 
belonging to some otherwise anonymous person. (If multiple claims exist for the same person, they will have the same identifier) 
Each claim includes a diagnosis code that indicates what malady they suffer from. Suppose that we pivot these claims to produce a list of files, each named for the diagnosis, which would
contain a list of the integer patient identifiers who's medical claims indicated that they were diagnosed with that
condition.

Now, given that list of files, we want to find the number of people who suffer from BOTH migraines (G43) and gastric ulcers (K25):

G43.txt:
3453234

ludflu

Using Scala implicit conversions to extend existing types and simplify code flow

When working with various frameworks, you'll find yourself dealing with particular datatypes over and over. In Apache Spark, one of those that you're frequently encounter is the Dataframe. Dataframes hold tabular data and allow you to perform parallelized queries and aggregations over practically limitless amounts of data. They're everywhere, and they're super useful, so you'll probably have lots of them. If you seek to avoid repetition in your code, you'll probably define a bunch of functions that take Dataframes as arguments and return Dataframes as well:

  def transformSomehow(df : DataFrame) : DataFrame = {
    //magic in here
  }

ludflu

In a classic hadoop job, you've got mappers and reducers. The "thing" being mapped and reduced are key-value pairs for some arbitrary pair of types. Most of your parallelism comes from the mappers, since they can (ideally) split the data and transform it without any coordination with other processes.

By contrast, the amount of parallelism in the reduction phase has an important limitation: although you may have many reducers, any given reducer is guaranteed to receive all the values for some particular key. So if there are a HUGE number of values for some particular key, you're going to have a bottleneck because they're all going to be processed by a single reducer.

However, there is another way! Certain types of data fit into a pattern:

• they can be combined with other values of the same type to form new values.
• the combining operation is associative. For example, integer addition: ((1 + 2) + 3) == (1 + (2 + 3))
• they have an identity value. (for

ludflu

#!/usr/bin/env stack
-- stack --install-ghc runghc
import qualified Data.Map as M

type Distribution = M.Map String Float

-- letter frequencies from https://en.wikipedia.org/wiki/Letter_frequency
dist :: Distribution
dist = M.fromList [("a", 0.08167), ("b", 0.01492), ("c", 0.02782), ("d", 0.04253), ("e", 0.12702), ("f",0.02228),
                   ("g", 0.02015), ("h",0.06094), ("i",0.06966), ("j",0.00153), ("k",0.00772), ("l",0.04025),

ludflu

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future

/**
  * Created by jsnavely on 1/30/17.
  */

object DefaultedErrorHandlerExample {

ludflu

ludflu

ludflu

The following is a (probably incomplete) list of breaking changes to the API:

- The collections endpoint has been removed. Use the Packages endpoint instead
- creating a new package now requires a dataset id
- for ETL superuser requests, we now require that the client sets and X-ORGANIZATION-ID header. (this is because data catalog items are stored in a different schema for each org)
- the "ancestors" fields no longer include the datasetId, since datasets are no longer packages
- when creating a package, the "parent" field is now optional - since it might be nested directly under a dataset

ludflu

import Euterpea

import System.Random
import Data.Map hiding (map,foldr,foldl)
 
fisherYatesStep :: RandomGen g => (Map Int a, g) -> (Int, a) -> (Map Int a, g)
fisherYatesStep (m, gen) (i, x) = ((insert j x . insert i (m ! j)) m, gen')
  where
    (j, gen') = randomR (0, i) gen