airawat

This gist covers a simple Pig eval UDF in Java, that mimics NVL2 functionality in Oracle.

Included:
1.  Input data
2.  UDF code in java
3.  Pig script to demo the UDF
4.  Expected result
5.  Command to execute script
6.  Output

airawat

This gist covers a simple Hive eval UDF in Java, that mimics NVL2 functionality in Oracle.
NVL2 is used to handle nulls and conditionally substitute values.
 
Included:
1.  Input data
2.  Expected results
3.  UDF code in java
4.  Hive query to demo the UDF
5.  Output

airawat

This gist covers a simple Hive genericUDF in Java, that mimics NVL2 functionality in Oracle.
NVL2 is used to handle nulls and conditionally substitute values.
 
Included:
1.  Input data
2.  Expected results
3.  UDF code in java
4.  Hive query to demo the UDF
5.  Output
 

airawat

......

List<String> artifactList = new List<String> ();
var scanOpts = new ScanOptions();

String rowRegex = rowID + ".*";
IteratorSetting iterSttng = new IteratorSetting();
iterSttng.Priority = 15;
iterSttng.Name = "rowIDRegexFilter";
iterSttng.IteratorClass="org.apache.accumulo.core.iterators.user.RegExFilter";

airawat

About this gist:
================
This gist is a part of a series of log parsers in Java Mapreduce, Pig, Hive, Python...
This one covers a log parser in Cascading.
It reads syslogs in HDFS -
a) Parses them based on a regex pattern & writes parsed files to HDFS
b) Writes records that dont match pattern to HDFS
c) Writes a report to HDFS that contains the count of distinct processes logged.

Other gists/blogs:

airawat

The sample programs, for Cascading(2.5.1) for Accumulo(1.5.0) are in github -
https://github.com/airawat/cascading.accumulo.examples

The source code for the extensions are at-
https://github.com/airawat/cascading.accumulo

airawat

# The following documentation details configuring an application ID to execute a SSH action

# In the illustration-
# edge node=cdh-sn03
# oozie server=cdh-mn01
# applicaiton ID=akhanolk


# ==========================================
# 1.  On edge node, as application ID

airawat

Kerberos
Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography

Kerberos Principals
A user in Kerberos is called a principal, which is made up of three distinct components: the primary, instance, and realm. 
A Kerberos principal is used in a Kerberos-secured system to represent a unique identity. 
The first component of the principal is called the primary, or sometimes the user component. 
The primary component is an arbitrary string and may be the operating system username of the user or the name of a service. 
The primary component is followed by an optional section called the instance, which is used to create principals that are used by users in special roles or to define the host on which a service runs, for example. 
An instance, if it exists, is separated from the primary by a slash and then the content is used to disambiguate multiple principals for a single user or service. 

airawat

using System;
using System.Text;
using Microsoft.ServiceBus.Messaging;
using System.Net;
using System.IO;

namespace StreamingAnalyticsEventPublisher
{
    class MeetupRSVPEventSender
    {

airawat

spark-submit --class com.khanolkar.bda.util.CompactRawLogs \
............
MyJar-1.0.jar \
"/user/akhanolk/data/raw/streaming/to-be-compacted/" \
"/user/akhanolk/data/raw/compacted/" \
"2" "128" "oozie-124"