airawat’s gists

airawat / 00-LogParser-PythonMR-UsingRegex

Last active December 19, 2015 06:59

Mapper and Reducer in python for log parsing using python regex

	This gist includes a mapper and reducer in python that can parse log files using
	regex; Usecase: Count the number of occurances of processes that got logged by month.

	Includes:
	---------
	Sample data
	Review of log data structure
	Sample data and scripts for download
	Mapper
	Reducer

airawat / 00-LogParser-JavaMapReduce-Regex

Last active September 18, 2016 09:36

00-JavaMapperReducerUsingRegex

	This gist includes a mapper, reducer and driver in java that can parse log files using
	regex; The code for combiner is the same as reducer;
	Usecase: Count the number of occurances of processes that got logged, inception to date.

	Includes:
	---------
	Sample data and scripts for download:01-ScriptAndDataDownload
	Sample data and structure: 02-SampleDataAndStructure
	Mapper: 03-LogEventCountMapper.java
	Reducer: 04-LogEventCountReducer.java

airawat / 00-LogParser-Hive-Regex

Last active October 4, 2020 01:56

Log parser in Hive using regex serde

	This gist includes hive ql scripts to create an external partitioned table for Syslog
	generated log files using regex serde;
	Usecase: Count the number of occurances of processes that got logged, by year, month,
	day and process.

	Includes:
	---------
	Sample data and structure: 01-SampleDataAndStructure
	Data download: 02-DataDownload
	Data load commands: 03-DataLoadCommands

airawat / 00-LogParser-PigLatin-UsingRegex

Last active December 19, 2015 06:59

	This gist includes a pig latin script to parse Syslog generated log files using regex;
	Usecase: Count the number of occurances of processes that got logged, by month,
	day and process.

	Includes:
	---------
	Sample data and structure: 01-SampleDataAndStructure
	Data and script download: 02-DataAndScriptDownload
	Data load commands: 03-HdfsLoadCommands
	Pig script: 04-PigLatinScript

airawat / 00-LogParserPigLatinNativeMapReduce

Last active December 19, 2015 07:49

There might be situations were you may have to reuse java map reduce programs within a pig program. This blog includes a sample pig script, with associated jars and sample data. The input is Syslog generated log files, and the output is a count of occurrences of processes logged inception to date.

	This gist includes a pig latin script to parse Syslog generated log files through a
	java mapreduce program that uses regex;

	Usecase: Count the number of occurances of processes that got logged, by month,
	day and process.

	Related gist that covers the java code - https://gist.github.com/airawat/5915374

	Pig version: version 0.10.0

airawat / 00-OozieWorkflowWithPigAction

Last active January 25, 2022 21:41

Sample of an Oozie workflow with pig action - parses Syslog generated log files using regex.

	This gist includes oozie workflow components to run a pig latin script to parse
	(Syslog generated) log files using regex;
	Usecase: Count the number of occurances of processes that got logged, by month,
	day and process.

	Pictorial overview of workflow:
	-------------------------------
	http://hadooped.blogspot.com/2013/07/apache-oozie-part-7-oozie-workflow-with_3.html

	Includes:

airawat / 00-OozieWorkflowStreamingMRAction-Python

Last active November 21, 2018 06:24

Sample of an Oozie workflow with streaming action - parses Syslog generated log files using python -regex

	This gist includes oozie workflow components (streaming map reduce action) to execute
	python mapper and reducer scripts to parse Syslog generated log files using regex;
	Usecase: Count the number of occurances of processes that got logged, by month, and process.

	Pictorial overview of workflow:
	--------------------------------
	http://hadooped.blogspot.com/2013/07/apache-oozie-part-5-oozie-workflow-with.html

	Includes:
	---------

airawat / 00-OozieCoordinatorJobWithTimeAsTrigger

Last active October 21, 2017 15:40

Oozie coordinator job example with time as trigger

	This gist includes components of a oozie (time initiated) coordinator application - scripts/code, sample data
	and commands; Oozie actions covered: hdfs action, email action, java main action,
	hive action; Oozie controls covered: decision, fork-join; The workflow includes a
	sub-workflow that runs two hive actions concurrently. The hive table is partitioned;
	Parsing uses hive-regex serde, and Java-regex. Also, the java mapper, gets the input
	directory path and includes part of it in the key.

	Usecase: Parse Syslog generated log files to generate reports;

	Pictorial overview of job:

airawat / 00-OozieCoordinatorJobWithFileAsTrigger

Last active February 12, 2018 10:10

Oozie coordinator job example with trigger file as trigger

	This gist includes components of a oozie (trigger file initiated) coordinator job -
	scripts/code, sample data and commands; Oozie actions covered: hdfs action, email action,
	java main action, hive action; Oozie controls covered: decision, fork-join; The workflow
	includes a sub-workflow that runs two hive actions concurrently. The hive table is
	partitioned; Parsing uses hive-regex serde, and Java-regex. Also, the java mapper, gets
	the input directory path and includes part of it in the key.

	Usecase
	-------
	Parse Syslog generated log files to generate reports;

airawat / 00-OozieCoordinatorJobWithDatasetCreationAsTrigger

Last active May 21, 2021 16:09

Sample Oozie coordinator job that executes upon availability of a specified dataset. Includes scripts/code, sample data, commands.

	This gist includes components of a oozie, dataset availability initiated, coordinator job -
	scripts/code, sample data and commands; Oozie actions covered: hdfs action, email action,
	sqoop action (mysql database); Oozie controls covered: decision;

	Usecase
	-------
	Pipe report data available in HDFS, to mysql database;

	Pictorial overview of job:
	--------------------------

Airawat airawat