Skip to content

Instantly share code, notes, and snippets.

View jvwing's full-sized avatar

James Wing jvwing

7 followers · 0 following
View GitHub Profile
@jvwing
jvwing / DynamoEnumConverter
Created May 30, 2014 19:46
For use with Amazon DynamoDB DataModel API for .NET. This class can be used to automagically serialize enum properties to DynamoDB, by decorating the property with the attribute [DynamoDBProperty(typeof(DynamoEnumConverter<AccountStatus>))], where "AccountStatus" is the name of the enumerated type..
public class DynamoEnumConverter<TEnum> : IPropertyConverter
{
public object FromEntry(DynamoDBEntry entry)
{
string valueAsString = entry.AsString();
TEnum valueAsEnum = (TEnum)Enum.Parse(typeof(TEnum), valueAsString);
return valueAsEnum;
}
public DynamoDBEntry ToEntry(object value)
@jvwing
jvwing / TestEvaluateXPath.xml
Last active August 11, 2019 17:04
NiFi Template for Testing EvaluateXPath in NiFi 0.6.0
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Test template for validation issues with EvaluateXPath in NiFi 0.6.0 (April 2016).</description><name>TestEvaluateXPath</name><snippet><processGroups><id>099c3c68-1720-4ddf-9173-25352325feb3</id><parentGroupId>6952f8cb-8b86-4446-a6ee-a7a98d7779dd</parentGroupId><position><x>780.0</x><y>72.0</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>9818db0d-fa04-4201-9597-9d1cee51e0e0</id><parentGroupId>099c3c68-1720-4ddf-9173-25352325feb3</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><bends><x>503.0</x><y>224.0</y></bends><bends><x>503.0</x><y>274.0</y></bends><destination><groupId>099c3c68-1720-4ddf-9173-25352325feb3</groupId><id>1105d248-9554-4f0c-960b-807dccff774b</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></na
@jvwing
jvwing / Database_Insert.xml
Last active August 11, 2019 17:04
Apache NiFi flow template for inserting data into relational databases. See https://www.batchiq.com/database-ingest-with-nifi.html for details.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Demonstrates a simple pattern for inserting FlowFiles into a relational database using the PutSQL processor.</description><name>Database Insert</name><snippet><processGroups><id>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</id><parentGroupId>23b56e53-a310-48e7-8a15-99d81109bdd9</parentGroupId><position><x>387.0</x><y>24.0</y></position><activeRemotePortCount>0</activeRemotePortCount><comments></comments><contents><connections><id>5e7de26e-13db-4b1b-8010-8f19f6611d49</id><parentGroupId>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><bends><x>1030.955965197994</x><y>203.67437167941813</y></bends><bends><x>1067.0</x><y>253.0</y></bends><destination><groupId>f478ace0-9295-4ee9-9a25-96e3b1c3e94f</groupId><id>a0b3418c-096e-4138-9400-043dcc7da5ba</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowF
@jvwing
jvwing / streamsetsdc.yaml
Created April 28, 2016 21:49
Cloud-Init Template for running StreamSets DC on EC2 (or other service)
#cloud-config
runcmd:
- echo "$(date) - Starting StreamSets Data Collector Install"
- yum install -y https://archives.streamsets.com/datacollector/1.3.0.1/rpm/streamsets-datacollector-1.3.0.1-1.noarch.rpm
- service sdc start
- chkconfig --add sdc
- echo "$(date) - Finished StreamSets Data Collector Install"
# StreamSets UI will be at http://<public dns>:18630/, u:admin, p:admin
# Be sure to open port 18630 on your security group, or use SSH port forwarding
# See https://streamsets.com/opensource/
@jvwing
jvwing / DatabaseExtract-Incremental.xml
Last active August 11, 2019 17:03
Apache NiFi flow template for extracting changed records out of a relational databases and into your NiFi flow. See https://www.batchiq.com/database-extract-with-nifi.html for details.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Simple flow for extracting records from a relational database and into JSON text files.</description><name>DatabaseExtract-Incremental</name><snippet><connections><id>5f3f82ab-fc02-4322-938c-ad2fb628f094</id><parentGroupId>0ef63979-973e-43fc-82b2-a5cd52db0858</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>bb5fadae-29d0-4983-b76d-4185a5003685</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>merged</selectedRelationships><source><groupId>0ef63979-973e-43fc-82b2-a5cd52db0858</groupId><id>08cf9faa-f1cd-4e94-b3bd-ff6366973fbb</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>40e2bddf-ac38-4261-b64c-07571f77b8fd</id><parentGroupId>0ef63979-973e
@jvwing
jvwing / FlowFileFieldsTest.xml
Created September 19, 2016 21:53
NiFi template demonstrating use of built-in fields in Expression Language.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>Tests various built-in FlowFile fields using Expression Language.</description><name>FlowFileFieldsTest</name><snippet><connections><id>ffa84c5c-a236-45f6-a139-51d826e73d3c</id><parentGroupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</groupId><id>6f7d661c-e476-48b7-a6fe-e06ea4dc4fd3</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</groupId><id>53074840-f32c-459f-8739-7ed866bf7c3f</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>2c2150f3-b67f-4bbe-ad88-97e4adfc7dd0</id><parentGroupId>8524b3ae-950f-49c5-a936-44dd56b5a4d1</paren
@jvwing
jvwing / tweetsummary.avsc
Created October 31, 2016 22:05
Avro schema for a lightweight summary of a Tweet
{
"namespace": "com.batchiq.samples.tweet",
"type": "record",
"name": "tweetsummary",
"fields": [
{"name": "id", "type": "long"},
{"name": "id_str", "type": "string"},
{"name": "created_at", "type": "string"},
{"name": "timestamp_ms", "type": "string"},
{"name": "lang", "type": "string"},
@jvwing
jvwing / twitter_to_s3_sample_v1.xml
Last active August 11, 2019 17:02
Apache NiFi flow template for ingesting data to Amazon S3. See https://www.batchiq.com/s3-ingest-with-nifi.html for details.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description></description><name>twitter_to_s3_sample_v1</name><snippet><connections><id>91f2ec79-3351-44eb-aa8a-b39623e71259</id><parentGroupId>8f183367-57f8-4340-8482-d246e37bb904</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>98a2973a-191b-4a6e-90f1-1b7ad12efe7e</groupId><id>dbd8bd4b-733f-43b3-b67b-bd2212598b10</id><type>INPUT_PORT</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><source><groupId>9c9d76ac-0ae2-4dee-92a3-a78ad2613342</groupId><id>cd683e79-2b95-4437-b7c8-4fce4ca8beeb</id><type>OUTPUT_PORT</type></source><zIndex>0</zIndex></connections><connections><id>21cc3ca7-8a0a-4a38-b113-0bdc9e0abe87</id><parentGroupId>8f183367-57f8-4340-8482-d246e37bb904</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</bac
@jvwing
jvwing / CloudTrail_Event_Processing_Starter_Kit.xml
Created January 17, 2017 21:22
Apache NiFi flow template for processing events from Amazon CloudTrail. See https://www.batchiq.com/process-cloudtrail-events-with-nifi.html for details.
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Template for processing events from Amazon CloudTrail.
See https://www.batchiq.com/process-cloudtrail-events-with-nifi.html</description>
<groupId>f46d6392-0158-1000-73ba-16e80594f898</groupId>
<name>CloudTrail_Event_Processing_Starter_Kit</name>
<snippet>
<processGroups>
<id>363e44cf-ed77-17f2-0000-000000000000</id>
@jvwing
jvwing / S3_Event_Notifications_Template.xml
Created February 15, 2017 03:36
Apache NiFi flow template for processing Amazon S3 event notifications. See https://www.batchiq.com/nifi-process-s3-event-notifications.html for details.
<?xml version="1.0" ?>
<template encoding-version="1.0">
<description>Apache NiFi flow template for processing Amazon S3 event notifications.
See https://www.batchiq.com/nifi-process-s3-event-notifications.html for details. </description>
<groupId>caf31a0c-be81-46a6-1672-8cfe77634cbe</groupId>
<name>S3_Event_Notifications_Template</name>
<snippet>
<processGroups>
<id>f8805001-d079-1b7a-0000-000000000000</id>
<parentGroupId>caf31a0c-be81-46a6-0000-000000000000</parentGroupId>