Animesh Trivedi animeshtrivedi

Systems researcher

animeshtrivedi / SparkCrailFileWriteBroadcast.scala

Last active May 18, 2017 11:16

This file has two functions to write raw 128 byte[] arrays vs. serialized array.

	import java.nio.ByteBuffer
	import java.util.concurrent.atomic.{AtomicBoolean, AtomicInteger, AtomicLong}
	import java.util.concurrent.{ConcurrentHashMap, Future, LinkedBlockingQueue, TimeUnit}

	import com.ibm.crail._
	import com.ibm.crail.conf.CrailConfiguration
	import com.ibm.crail.utils.CrailImmediateOperation
	import org.apache.spark._
	import org.apache.spark.common._
	import org.apache.spark.executor.ShuffleWriteMetrics

animeshtrivedi / spark-default

Last active August 11, 2023 17:02

A spark configuration for some performance knobs

	# RDD settings
	spark.rdd.compress false

	# Shuffle settings
	spark.shuffle.manager sort
	spark.shuffle.compress false
	spark.shuffle.spill false
	spark.shuffle.spill.compress false
	spark.shuffle.sort.initialBufferSize 4194304
	spark.shuffle.sort.bypassMergeThreshold 200

animeshtrivedi / UnsafeCopy.java

Created May 10, 2017 07:38

a sample code to unsafe copy from byte[] to double[] (and vice-versa)

	public static void copyTest(String[] args) {
	Unsafe us = null;
	try {
	Field f = Unsafe.class.getDeclaredField("theUnsafe");

	f.setAccessible(true);
	us = (Unsafe) f.get(null);
	System.err.println(" oh my, I have it : " + us);
	} catch (Exception e) {
	e.printStackTrace();

animeshtrivedi / GetUnsafeClass.scala

Created May 9, 2017 16:33 — forked from ramn/GetUnsafeClass.scala

sun.misc.Unsafe use in Scala

	def getUnsafeInstance: sun.misc.Unsafe = {
	val f = classOf[sun.misc.Unsafe].getDeclaredField("theUnsafe")
	f.setAccessible(true)
	val unsafe = f.get(null).asInstanceOf[sun.misc.Unsafe]
	unsafe
	}

animeshtrivedi / CrailTest.scala

Created May 5, 2017 14:58

Crail benchmark tests as Spark program

	import java.nio.ByteBuffer
	import java.util.Random
	import java.util.concurrent.Future

	import com.ibm.crail._
	import com.ibm.crail.conf.CrailConfiguration
	import com.ibm.crail.memory.OffHeapBuffer


	object CrailTest extends Serializable {

animeshtrivedi / GroupByTest.scala

Created May 4, 2017 15:26

Spark shuffle benchmark with variations of the groupBy test

	import java.util.Random

	import org.apache.spark.SparkContext

	object GroupByTest {
	def test(numMappers:Int = 10, numKVPairs:Int = 100, valSize:Int = 1024, numReducers:Int = 10) {
	val pairs1 = sc.parallelize(0 until numMappers, numMappers).flatMap { p =>
	val ranGen = new Random
	val arr1 = new Array[(Int, Array[Byte])](numKVPairs)
	for (i <- 0 until numKVPairs) {

animeshtrivedi / BroadcastTest.scala

Created May 4, 2017 11:10

A broadcast stress test for Spark.

	/*
	* Broadcast stress test for Spark
	*
	* Author: Animesh Trivedi <[email protected]>
	*
	* Copyright (C) 2017, IBM Corporation
	*
	* Licensed under the Apache License, Version 2.0 (the "License");
	* you may not use this file except in compliance with the License.
	* You may obtain a copy of the License at

animeshtrivedi / AtrOutputWriter.scala

Last active January 11, 2017 13:24

Null writer class, that discards all Row/InternalRow passed to it. The class, however, does some accounting.

	package org.apache.spark.sql.execution.datasources.atr

	import org.apache.hadoop.mapreduce.TaskAttemptContext
	import org.apache.spark.sql.Row
	import org.apache.spark.sql.catalyst.InternalRow
	import org.apache.spark.sql.execution.datasources.OutputWriter

	import scala.tools.jline_embedded.internal.Log

	/**

animeshtrivedi / AtrFileFormat.scala

Last active January 11, 2017 12:59

AtrFileFormat which creates a null sink writer (AtrOutputWriter)

	package org.apache.spark.sql.execution.datasources.atr

	import org.apache.hadoop.fs.FileStatus
	import org.apache.hadoop.mapreduce.{TaskAttemptContext, Job}
	import org.apache.spark.sql.SparkSession
	import org.apache.spark.sql.execution.datasources.{OutputWriter, OutputWriterFactory, FileFormat}
	import org.apache.spark.sql.sources.DataSourceRegister
	import org.apache.spark.sql.types.StructType

	import scala.tools.jline_embedded.internal.Log

animeshtrivedi / ScalaInterpreterExample.java

Created October 7, 2016 14:46

	package com.ibm.crail.spark.tools;

	/**
	* Created by atr on 07.10.16.
	*/
	import scala.collection.Iterator;
	import scala.collection.JavaConversions;
	import scala.collection.Seq;
	import scala.reflect.internal.util.AbstractFileClassLoader;
	import scala.reflect.internal.util.BatchSourceFile;