yashk’s gists

yashk / gist:53a257b595faf86ef3d41162a91b5b9f

Last active May 21, 2017 14:44

nycdata

https://archive.org/download/nycTaxiTripData2013

yashk / flock

Created May 18, 2017 19:21

http://www.kfirlavi.com/blog/2012/11/06/elegant-locking-of-bash-program/

yashk / nc

Created May 3, 2017 04:21

nc

	while true
	while> do
	while> nc -l -p 9999 localhost
	while> done


	cat test.yml\| nc -c localhost 9999

yashk / tibco_jobs.png

Last active April 26, 2017 03:26

openings at tibco

test

yashk / gist:40b4093c55fccf247fbb83a1090b9010

Created April 5, 2017 06:38

Tibco Data Engineer JD

	Role Overview:
	We have an immediate need for a senior-level Big data Engineer with core Java skills to join our TIBCO Mashery team. You'll have an opportunity to work on the Mashery Datamart that leverages Big data and Cloud technologies such as Kafka, Spark, Hadoop, AWS, etc. Mashery Datamart is a critical component of the Mashery API Management product that provides data to customers that can be used for performance and usage measurements and for billing functions.
	Responsibilities:
	Creating complex data processing pipelines, using apache spark/hadoop.
	Maintain and expand batch aggregations/reports to existing data mart processing using Hive/Oozie.
	Maintain and expand real-time real-time aggregations using Spark streaming,Cassandra,Graphite, Elasticsearch.
	Writing Unit tests, integration tests and help in defining and executing performance Tests for Datamart components.
	Deploying data pipelines in production based on Continuous Delivery practices.
	Create Dashboards/Alerts exhibiting critical metrics and

yashk / jmap.sh

Created January 31, 2017 13:05

jmap -F -dump:format=b,file=<pid>_hd.hprof <pid>

yashk / hiveanalysis.md

Last active January 20, 2017 14:18

Which table HiveMetaStoreClient.getPartition queries and does that table have index on partitions

This [code][1] is exectuted to get parition list

private MPartition getMPartition(String dbName, String tableName, List<String> part_vals)
     .........
     ........
     query =
          pm.newQuery(MPartition.class,
              "table.tableName == t1 && table.database.name == t2 && partitionName == t3");

yashk / idea as diff-merge tool

Created December 12, 2016 06:54

	[merge]
	tool = intellij
	[mergetool "intellij"]
	cmd = idea merge $(cd $(dirname "$LOCAL") && pwd)/$(basename "$LOCAL") $(cd $(dirname "$REMOTE") && pwd)/$(basename "$REMOTE") $(cd $(dirname "$BASE") && pwd)/$(basename "$BASE") $(cd $(dirname "$MERGED") && pwd)/$(basename "$MERGED")
	trustExitCode = true
	[diff]
	tool = intellij
	[difftool "intellij"]
	cmd = idea diff $(cd $(dirname "$LOCAL") && pwd)/$(basename "$LOCAL") $(cd $(dirname "$REMOTE") && pwd)/$(basename "$REMOTE")

yashk / Problems.md

Last active October 12, 2016 02:36

Interview problems i have solved

yashk / fsync logs

Created September 21, 2016 14:24

    cat ~/Downloads/fsync.log | jq  -r '.results[] | [.time, .host , (.message | split("ms")[0] | split(" ")[8])] | @csv'

yash yashk