Skip to content

Instantly share code, notes, and snippets.

@peteroid

peteroid/Spark-random-nums.scala

Created July 2, 2016 03:51
Show Gist options
  • Save peteroid/b3335eb0bf1ed317611c655e7b1ba42f to your computer and use it in GitHub Desktop.
Save peteroid/b3335eb0bf1ed317611c655e7b1ba42f to your computer and use it in GitHub Desktop.
Download ZIP
Generate random numbers and save to hdfs
Raw
Spark-random-nums.scala
import scala.util.Random;
var NUM_SAMPLES = 2000000000;
val count = sc.parallelize(1 to NUM_SAMPLES).map{ _ => Random.nextLong };
var sum = count;for (_ <- 1 to 20){sum = sum ++ count};
sum.saveAsTextFile("hdfs://10.0.72.14:54310/data/big_num_700g.txt");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment