Skip to content

Instantly share code, notes, and snippets.

View ScrapCodes's full-sized avatar

Prashant Sharma ScrapCodes

View GitHub Profile
#!/bin/bash
# bash generate random alphanumeric string
#
# bash generate random 32 character alphanumeric string (upper and lowercase) and
NEW_UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
# bash generate random 32 character alphanumeric string (lowercase only)
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1
@ScrapCodes
ScrapCodes / error.log
Created August 28, 2018 07:52
Error while running Spark-r on K8s
18/08/28 07:01:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Loading required package: methods
Attaching package: ‘SparkR’
The following objects are masked from ‘package:stats’:
cov, filter, lag, na.omit, predict, sd, var, window
The following objects are masked from ‘package:base’:
as.data.frame, colnames, colnames<-, drop, endsWith, intersect,
rank, rbind, sample, startsWith, subset, summary, transform, union
Warning message:
package ‘SparkR’ was built under R version 3.4.4
@ScrapCodes
ScrapCodes / gist:232ea0d36af0bd6017d6
Created May 25, 2015 06:24
transcript of errors.
15/05/25 11:52:22 INFO org.apache.spark.SecurityManager: Changing view acls to: prashant
15/05/25 11:52:22 INFO org.apache.spark.SecurityManager: Changing modify acls to: prashant
15/05/25 11:52:22 INFO org.apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(prashant); users with modify permissions: Set(prashant)
15/05/25 11:52:22 ERROR org.apache.spark.executor.Executor: Exception in task 3.0 in stage 6.0 (TID 27)
java.lang.ExceptionInInitializerError
at $line31.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:20)
at $line31.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:25)
at $line31.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:27)
at $line31.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:29)
at $line31.$read$$iw$$iw$$iw$$iw.<init>(<console>:31)
@ScrapCodes
ScrapCodes / generated_dep_tree.log
Last active August 29, 2015 14:07
A project to verify spark artifacts. It depends on all artifacts of spark.
prashant@sc:~/work/spark-examples$ mvn dependency:tree
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Examples 0.1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ spark-examples_2.10 ---
[INFO] org.apache.spark:spark-examples_2.10:jar:0.1.0-SNAPSHOT
@ScrapCodes
ScrapCodes / A.scala
Last active August 9, 2016 09:48
Understanding type inference. Why do I have to specify the types explicitly ?
scala> import scala.reflect._
import scala.reflect._
scala> def f[K: ClassTag, V: ClassTag, T<:Product2[K, V]](names: T, values: T) = {
val x = values;
println(classTag[K]);
names.productIterator.toList.zip(x.productIterator.map(_.toString).toSeq).map(kv => kv._1+"="+kv._2)
}
| | | | f: [K, V, T <: Product2[K,V]](names: T, values: T)(implicit evidence$1: scala.reflect.ClassTag[K], implicit evidence$2: scala.reflect.ClassTag[V])List[String]
@ScrapCodes
ScrapCodes / Akka-2.1.4.scala
Last active December 27, 2015 23:49
Akka 2.2.3 and 2.1.x remoting difference.
import akka.actor.{Props, ActorRef, Actor, ActorSystem}
import com.typesafe.config.ConfigFactory
import scala.util.Random
import scala.collection.mutable.LinkedList
import scala.reflect.ClassTag
import scala.collection.mutable
import akka.remote.RemoteActorRefProvider
case class SubscribeReceiver(receiverActor: ActorRef)
find scala/ akka/ spark/ -name '*scala' -type f | xargs -n 1 wc -l >> lines_scala
cat lines_scala | awk '{ SUM += $1} END { print SUM }'
587927
find hadoop/ -name '*java' -type f | xargs -n 1 wc -l >> lines_java
cat lines_java | awk '{ SUM += $1} END { print SUM }'
881574
@ScrapCodes
ScrapCodes / temp.scala
Created August 8, 2013 05:51
temp.scala
sealed trait PacketType { def name: String; def id: Int }
case object disconnect extends PacketType { val name = "disconnect"; val id = 0 }
case object connect extends PacketType { val name = "connect"; val id = 1 }
case object heartbeat extends PacketType { val name = "heartbeat"; val id = 2 }
case object message extends PacketType { val name = "message"; val id = 3 }
case object Json extends PacketType { val name = "json"; val id = 4 }
case object event extends PacketType { val name = "event"; val id = 5 }
case object ack extends PacketType { val name = "ack"; val id = 6 }
case object error extends PacketType { val name = "error"; val id = 7 }
case object noop extends PacketType { val name = "noop"; val id = 8 }
@ScrapCodes
ScrapCodes / etags.sh
Last active November 10, 2016 16:20
file for etags --regex@${file} to generate scala tags.
find -name '*scala' -type f -exec etags [email protected] {} +
@ScrapCodes
ScrapCodes / erlMessage.erl
Created November 1, 2011 12:13
p2p erlang messanger
-module(erlMessage).
-export([start/1,sendMessage/2,msgReceive/0]).
% msgReceive blocks on the message queue and waits for a message!
% and prints the message as soon as it arrives and again tail recurses!
msgReceive() ->
receive
{Message,Message_from} -> io:format("~p said:~p~n",[Message_from,Message]);
Message ->io:format("~p ~n",[Message])