mattyb149’s gists

mattyb149 / ExtractRecordFields.json

Created September 7, 2023 19:28

NiFi 1.23+ Flow Definition to extract field names from schema

mattyb149 / SequenceLookupExample.xml

Created August 22, 2018 15:14

A NiFi template with a ScriptedLookupService that retrieves a sequence number from an external database

	<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
	<template encoding-version="1.2">
	<description>This template gives an example of using a ScriptedLookupService to provide sequence numbers from a database.</description>
	<groupId>5d7e55f8-0165-1000-1170-d42741815ddb</groupId>
	<name>SequenceLookupExample</name>
	<snippet>
	<connections>
	<id>0cf6aae2-8e5a-361f-0000-000000000000</id>
	<parentGroupId>cedd92a0-903e-3f68-0000-000000000000</parentGroupId>
	<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

mattyb149 / SplitRecord_w_Conversion.xml

Created July 11, 2018 16:23

NiFi SplitRecord example that converts CSV to Avro while splitting files

	<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
	<template encoding-version="1.2">
	<description></description>
	<groupId>89f8c6b8-0164-1000-d937-82c55b6fdffd</groupId>
	<name>SplitRecord_w_Conversion</name>
	<snippet>
	<controllerServices>
	<id>350009ab-3f44-32bf-0000-000000000000</id>
	<parentGroupId>fa0118f1-ff03-3030-0000-000000000000</parentGroupId>
	<bundle>

mattyb149 / ISP_template.py

Created November 7, 2017 17:50

An InvokeScriptedProcessor template (in Jython) for running ExecuteScript Jython scripts faster

	#////////////////////////////////////////////////////////////
	#// imports go here
	#////////////////////////////////////////////////////////////
	from org.apache.nifi.processor import Processor,Relationship
	from java.lang import Throwable

	class E():
	def __init__(self):
	pass
	def executeScript(self,session, context, log, REL_SUCCESS, REL_FAILURE):

mattyb149 / testEL.groovy

Created October 25, 2017 21:27

Groovy script for testing NiFi Expression Language expressions

	@Grab(group='org.apache.nifi', module='nifi-expression-language', version='1.4.0')
	import org.apache.nifi.attribute.expression.language.*

	def cli = new CliBuilder(usage:'groovy testEL.groovy [options] [expressions]',
	header:'Options:')
	cli.help('print this message')
	cli.D(args:2, valueSeparator:'=', argName:'attribute=value',
	'set value for given attribute')
	def options = cli.parse(args)
	if(!options.arguments()) {

mattyb149 / csv_to_avroschema.groovy

Created May 19, 2017 19:04

A Groovy script for NiFi ExecuteScript to extract the schema from the header line of a CSV file

	import groovy.json.*

	def flowFile = session.get()
	if(!flowFile) return

	def delim = ','
	try {
	delim = delimiter?.value ?: ','
	} catch (MissingPropertyException mpe) { }

mattyb149 / ScriptedReaderTest.xml

Created April 25, 2017 15:57

A NiFi 1.2.0 template with sample data and script to illustrate the use of ScriptedReader

	<?xml version="1.0" ?>
	<template encoding-version="1.1">
	<description></description>
	<groupId>a5a9dae4-015b-1000-7a1c-eaf2db85b188</groupId>
	<name>ScriptedReaderTest</name>
	<snippet>
	<controllerServices>
	<id>bb8e2ece-1712-3f89-0000-000000000000</id>
	<parentGroupId>f3e4227c-c306-3c57-0000-000000000000</parentGroupId>
	<bundle>

mattyb149 / ConvertRecord_examples.xml

Created April 18, 2017 14:43

NiFi template with various tests / exercises of the ConvertRecord processor

	<?xml version="1.0" ?>
	<template encoding-version="1.1">
	<description></description>
	<groupId>7ebaf566-015b-1000-b416-922a420ea741</groupId>
	<name>ConvertRecord_examples</name>
	<snippet>
	<connections>
	<id>81dd9a51-07c4-39f0-0000-000000000000</id>
	<parentGroupId>868f7eec-2b0e-352b-0000-000000000000</parentGroupId>
	<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

mattyb149 / CDC_MySQL_Replication.xml

Created April 17, 2017 20:33

NiFi template with end-to-end table replication from a source MySQL database to a target MySQL database

	<?xml version="1.0" ?>
	<template encoding-version="1.1">
	<description>This template shows an end-to-end example of how to replicate table(s)
	from a source MySQL database to a target MySQL database</description>
	<groupId>68961054-015b-1000-2c21-01b42e84dbe0</groupId>
	<name>CDC_MySQL_Replication</name>
	<snippet>
	<connections>
	<id>9a0f0bca-5c0c-3f1e-0000-000000000000</id>
	<parentGroupId>23d80cd0-6e6e-31c9-0000-000000000000</parentGroupId>

mattyb149 / EnforceOrder_Examples.xml

Created April 7, 2017 13:24

A NiFi Template with 2 example use cases for the EnforceOrder processor

	<?xml version="1.0" ?>
	<template encoding-version="1.1">
	<description>This template contains two use cases for the EnforceOrder processor:

	1) Random Delay: FlowFiles are routed to two separate paths and delayed by a random amount of time (0-100 millis)
	2) MultipleSplits: FlowFiles are split twice with SplitText, the attributes must be updated to keep the original order
	</description>
	<groupId>48313cf9-015b-1000-cf31-bc9709ad2638</groupId>
	<name>EnforceOrder_Examples</name>
	<snippet>

Matt Burgess mattyb149