Skip to content

Instantly share code, notes, and snippets.

View dragos's full-sized avatar

Iulian Dragos dragos

View GitHub Profile
@dragos
dragos / scala-env.sh
Last active April 3, 2022 20:01
Simple bash functions to manage different Scala versions on the command line
#!/bin/bash -e
# where Scala tarballs are extracted
SCALA_BASE=$HOME/scala
# Usage:
#
# Install:
# . /path/to/scala-env.sh in your bash profile
#
@dragos
dragos / q.scala
Created October 18, 2018 08:18
Match exhaustiveness
sealed abstract class Tree
abstract class Expression extends Tree
case class Apply() extends Expression
class Foo(t: Tree) {
// is this match exhaustive?
t match {
case Apply() => ???
}
}
object Main {
private var things: Array[Object] = null
def allocate(size: Int): Unit = {
things = new Array[Object](size)
Array.fill(size)(new Object)
}
def main(args: Array[String]): Unit = {
1,2d0
< #!/bin/bash
<
33c31
< -e cpu/umask=0x01,event=0xC3,cmask=1,any=0,inv=0,edge,name=MACHINE_CLEARS.COUNT/ \
---
> -e cpu/umask=0x01,event=0xC3,cmask=1,any=0,inv=0,edge=1,name=MACHINE_CLEARS.COUNT/ \
148c146
< L2_BOUND=$(compute "($LOAD_L2_HIT / ($LOAD_L2_HIT + $L1D_PEND_MISS_FB_FULL))")
---
IPC: 1.08
Frontend Bound: 21.58474134716533659800
Backend Bound: 41.05539791232042026700
Memory Bound (local): 65.85114476875492319500
L1 Bound: 24.79441407483482076800
L2 Bound: 325.63202789719244855000
L3 Bound: 19.84881997132453514100
Main Memory Bound: -272.27838043263752965500
Stores Bound: 2.00311848928572519500
Core Bound (local): 34.14885523124507680500
class EnsimeActor(langServer: EnsimeLanguageServer, config: EnsimeConfig) extends Actor with LazyLogging {
private var project: ActorRef = _
override val supervisorStrategy = OneForOneStrategy(5, 1 minute) {
case e @ ActorInitializationException(actor, message, cause) =>
logger.error(s"Actor failed to initialize", e)
langServer.connection.logMessage(MessageType.Error, s"Error starting ensime: $message")
SupervisorStrategy.Restart
case e =>
$ scala
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_25).
Type in expressions to have them evaluated.
Type :help for more information.
scala> println("hello") // show
object $read extends scala.AnyRef {
def <init>() = {
super.<init>;
()

Investigating memory leaks in the Spark Shell

Summary

Re-assigning a variable in the REPL leads to memory leaks.

Reproducible test-case

The original case had a few unnecessary features, but it boils down to re-assigning a fresh large array (100MB) to the same variable several times.

Keybase proof

I hereby claim:

  • I am dragos on github.
  • I am dragos (https://keybase.io/dragos) on keybase.
  • I have a public key whose fingerprint is 98FA BD33 37FB 1EA8 A8CD F67D DC08 1D4C 730E C746

To claim this, I am signing this object:

// fails with:
// $ /opt/scala-2.10.4/bin/scalac -d /tmp src/main/scala/infer.scala -cp ../spark/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.2.0.jar
// src/main/scala/infer.scala:27: error: missing parameter type for expanded function ((x$2) => x$2._2)
// ds.groupBy(_._1).agg(sum(_._2), sum(_._3)).collect()
// ^
// one error found
//
import org.apache.spark.sql.functions._