eavidan’s gists

eavidan / subprocess_example.py

Created July 31, 2017 10:23

Easy mining

	import subprocess

	# simply running a command and getting the output
	cmd = ['ps', '-ax']
	s = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
	(stdoutdata, stderrdata) = s.communicate()
	#print(stdoutdata)

	# you can chain commands
	# the following runs 'ls /etc \| grep ntp' by sending the 'ls etc' output to the input of the 'grep ntp'

eavidan / spark_df_rows_into_cols

Created October 26, 2017 11:03

working on spark DF partitions. the following creates a Map of columns (key: column name, value: list of values) from the Rows supplied by the DF in each partition

	val names = df.columns.toList
	println(names)
	df.foreachPartition(rows => {
	var cols = scala.collection.mutable.Map[String, List[Any]]()
	names.foreach(col => cols(col) = List())
	rows.foreach(row => names.zip(row.toSeq).map(x => {
	cols(x._1) = cols(x._1) :+ x._2
	}))
	println(cols)
	})

eavidan / predictResponse_into_nparray.py

Created February 11, 2018 13:42

converting Tensorflow serving PredicResponse into np array

	def predictResponse_into_nparray(response, output_tensor_name):
	dims = response.outputs[output_tensor_name].tensor_shape.dim
	shape = tuple(d.size for d in dims)
	print(shape)
	return np.reshape(response.outputs[output_tensor_name].float_val, shape)

eavidan / BigPanda.scala

Last active March 5, 2018 10:16

	import org.apache.spark.ml.Pipeline
	import org.apache.spark.ml.classification.RandomForestClassifier
	import org.apache.spark.ml.evaluation.{BinaryClassificationEvaluator, MulticlassClassificationEvaluator}
	import org.apache.spark.ml.feature._
	import org.apache.spark.sql.{Column, DataFrame, Row, SparkSession}
	import org.apache.spark.sql.functions._
	import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder}


	/**

eavidan / obfuscated_dror.py

Created March 6, 2018 15:18

	def a_friendly_professor():

	return "".join(
	[chr (int (x*i+i0.01 ))for i, x in enumerate(
	[0, 68, 10+ 0.677 ,4.8058 ,3.2+
	0.06+ 0.0075 ,2, 2.0348 ,1.7279 ,1.54
	,1.6032 ,1.5864 ,1.527 ,1.486 ,1.4261 ,1.3
	+0.0971,1.372, 1.3422 ,1.318505429 ])])

eavidan / tensorflow_mnist_grpc_vs_rest_inference_rate.py

Last active October 22, 2020 15:57

comparing tensorflow mnist inference rate using gRPC vs REST

	import pickle
	import numpy as np
	import time
	import requests
	import subprocess
	import re
	from grpc.beta import implementations
	import tensorflow as tf
	from tensorflow_serving.apis import predict_pb2
	from tensorflow_serving.apis import prediction_service_pb2

eavidan / tensorflow_serving_rest_client.py

Last active August 5, 2018 09:17

	import pickle
	import numpy as np
	import requests

	host = 'localhost'
	port = '8501'
	batch_size = 1
	image_path = "./mnist_image.pkl"
	model_name = 'mnist'
	signature_name = 'predict_images'

eavidan / Dockerfile

Last active August 5, 2018 09:24

prebuilt tensorflow serving dockerfile

	FROM ubuntu:16.04

	MAINTAINER Eran Avidan

	# Install general packages
	RUN apt-get update && apt-get install -y \
	curl \
	libcurl3-dev \
	unzip \
	wget \

eavidan / tensorflow_grpc_vs_rest_requests_prep.py

Created August 5, 2018 11:41

benchmark for the preparation rate of MNIST requests for tensorflow serving REST vs gRPC

	import tensorflow as tf
	from tensorflow_serving.apis import predict_pb2
	import time
	import base64
	import requests
	import pickle
	import numpy as np
	import random

eavidan / ray_distributed_sort.py

Last active December 17, 2018 13:29

an implementation of distributed bucket sort for pandas using Ray

	import ray
	import pandas as pd
	import numpy as np

	ray.init(local_mode=True)


	def ge(a, b, by):
	gt = a > b
	eq = a == b

Eran Avidan eavidan