ASvyatkovskiy

Example of stat analysis with Spark and Histogrammar

Join the cosine and jaccard output files on the key-key pair, and convert it to dataframe:

val data = cosineRDD.join(jaccardRDD).toDF("cosine","jaccard")
data.write.parquet("/user/alexeys/correlations_3state")

Launch spark-shell session with histogrammar pre-loaded:

ASvyatkovskiy

//Dark Matter with Spark
import org.dianahep.histogrammar._
import org.dianahep.histogrammar.sparksql._
import org.dianahep.histogrammar.bokeh._
case class InfoVars(var runNum: Long, var lumiSec: Long, var evtNum: Long, var metfilter: Long, var scale1fb: Double, var evtWeight: Double, var pfmet: Double, var pfmetphi: Double, var puppet: Double, var puppetphi: Double, var fakepfmet: Double, var fakepfmetphi: Double, var fakepuppet: Double, var fakepuppetphi: Double)
case class GenEvtInfoVars(var genVPt: Double, var genVPhi: Double)
case class MuonVars(var pt: Double, var eta: Double, var phi: Double, var m: Double)
case class ElectronVars(var pt: Double, var eta: Double, var phi: Double, var m: Double)
case class TauVars(var pt: Double, var eta: Double, var phi: Double)
case class PhotonVars(var NLoose: Int, var NMedium: Int, var pt: Double, var eta: Double, var phi: Double)

ASvyatkovskiy

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math
import sys
import tempfile
import time
import os

ASvyatkovskiy

Running notebook in a Docker container inside a VM via ssh tunnel

Inside VM, determine the IP address of the docker container:

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' container_name_or_id

Container ID would be a hash string like f9bb667e8d59.

Next, shell into a running Docker container on your VM:

ASvyatkovskiy

#!/usr/bin/env python
from mpi4py import MPI
import numpy as np
import time

comm = MPI.COMM_WORLD
comm_rank = comm.Get_rank()
comm_size = comm.Get_size()
sizes = 16000

ASvyatkovskiy

#!/usr/bin/env python
from mpi4py import MPI
import numpy as np
import time

import pycuda.autoinit
from pycuda import gpuarray

comm = MPI.COMM_WORLD
comm_rank = comm.Get_rank()

ASvyatkovskiy

from __future__ import print_function

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding
from keras.layers import LSTM
from keras.datasets import imdb
from keras.callbacks import TensorBoard, Callback
import keras.backend as K
import tensorflow as tf

ASvyatkovskiy

import tensorflow as tf
import numpy as np
import math
import os


def float32_variable_storage_getter(getter, name, shape=None, dtype=None,
                                    initializer=None, regularizer=None,
                                    trainable=True,
                                    *args, **kwargs):