Skip to content

Instantly share code, notes, and snippets.

@mattyb149
mattyb149 / GenerateFlowFileWithContent.xml
Created February 25, 2016 02:28
A NiFi template using InvokeScriptedProcessor with Groovy to offer a GenerateFlowFile alternative that allows for setting the content and filename of a flow file
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template can be used like a GenerateFlowFile processor, where the content and filename can be specified. It uses InvokeScriptedProcessor and Groovy to achieve this.</description><name>GenerateFlowFileWithContent</name><snippet><processors><id>8e949039-0779-458d-b825-32b09ef80b38</id><parentGroupId>d3d19b0c-5876-41a2-8b09-e5ed911dfa67</parentGroupId><position><x>-172.94315107850957</x><y>-64.62506581228018</y></position><config><bulletinLevel>WARN</bulletinLevel><comments></comments><concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount><defaultConcurrentTasks><entry><key>TIMER_DRIVEN</key><value>1</value></entry><entry><key>EVENT_DRIVEN</key><value>0</value></entry><entry><key>CRON_DRIVEN</key><value>1</value></entry></defaultConcurrentTasks><defaultSchedulingPeriod><entry><key>TIMER_DRIVEN</key><value>0 sec</value></entry><entry><key>CRON_DRIVEN</key><value>* * * * * ?</value></entry></defaultSchedulingPeriod><d
@mattyb149
mattyb149 / SplitTextOnDelimiter.xml
Created February 23, 2016 20:35
A NiFi template using ExecuteScript with Groovy to split flow file lines of delimited text, outputting the middle two columns
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template uses ExecuteScript with Groovy to split lines of flowfile content on a delimiter, and outputs the middle two columns</description><name>SplitTextOnDelimiter</name><snippet><connections><id>c052f889-3ebd-4e44-acc0-8ce20fdf6d26</id><parentGroupId>d3d19b0c-5876-41a2-8b09-e5ed911dfa67</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>d3d19b0c-5876-41a2-8b09-e5ed911dfa67</groupId><id>4ef63acb-9201-408c-a274-b8103071b900</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>d3d19b0c-5876-41a2-8b09-e5ed911dfa67</groupId><id>4f7a6f81-c12c-4474-8fb9-bd3fe14d9c38</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>739611e7-d80f-4593-89d3-17793cfa
@mattyb149
mattyb149 / ExtractTextFromPDFWithScript.xml
Created February 20, 2016 01:29
Extract Text and Metadata from PDFs with NiFi's ExecuteScript processor (and Groovy)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template extracts text and metadata from PDF files using a Groovy script</description><name>ExtractTextFromPDFWithScript</name><snippet><connections><id>e749c17c-57ae-4f90-b741-37b9ce2fef70</id><parentGroupId>f2cfcff1-1730-4250-9535-f84a5af7d2a4</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>f2cfcff1-1730-4250-9535-f84a5af7d2a4</groupId><id>96e61f17-5b0c-4746-a221-daf286bc367f</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>f2cfcff1-1730-4250-9535-f84a5af7d2a4</groupId><id>9d2d13b7-0589-4b1d-8ca5-60f590ad8958</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>93f9f218-4ec6-4aba-90f5-f5bca4ff9d4c</id><parentGroupId>f2cfcff1-1730-4250-95
@mattyb149
mattyb149 / JeffAustralia.xml
Created February 17, 2016 03:19
An Apache NiFi template to create arbitrary URLs based on incoming file names
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>A sample template to route on filename to build a URL</description><name>JeffAustralia</name><snippet><connections><id>dec6cc8b-14ca-43f2-95ba-17463d587eb2</id><parentGroupId>f813e419-f52e-40c2-8061-79ae1d7d880f</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>f813e419-f52e-40c2-8061-79ae1d7d880f</groupId><id>a6d90e5e-dfe6-4c0a-a558-4283c3bd95f0</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>fileA</selectedRelationships><source><groupId>f813e419-f52e-40c2-8061-79ae1d7d880f</groupId><id>43c814e3-848e-4a04-979e-5f37d3bd5f4b</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>1bd7760e-424d-4c93-84da-e8e0ffb8facc</id><parentGroupId>f813e419-f52e-40c2-8061-79ae1d7d880f</parentGroupId><backPress
@mattyb149
mattyb149 / GroovyJsonToJsonExample.xml
Last active January 21, 2019 23:23
An Apache NiFi template to transform a JSON document from one schema to another
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template illustrates how to use the ExecuteScript processor with Groovy to perform JSON-to-JSON transformations</description><name>GroovyJsonToJsonExample</name><snippet><connections><id>09e8aa98-c9b8-43f0-afbd-513f1d417061</id><parentGroupId>67fb1165-c100-47df-bde3-aa0a4f57af67</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>67fb1165-c100-47df-bde3-aa0a4f57af67</groupId><id>8e8aa799-d1ab-4f97-ab35-b06a5a104ae2</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>67fb1165-c100-47df-bde3-aa0a4f57af67</groupId><id>b52a6649-9570-4db5-8541-d5c0302007c0</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>b032a9c3-c1d7-4e62-9646-25fe680a0957</id><p
@mattyb149
mattyb149 / InvokeScriptedProcessor_Groovy_PrintFullName.xml
Created February 11, 2016 02:11
NiFi template to get CSV data from randomuser.me & capitalize first/last name then output full name
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template takes CSV data from randomuser.me and capitalizes the first name and last name from the input and outputs the full name</description><name>InvokeScriptedProcessor_Groovy_PrintFullName</name><snippet><connections><id>3287d7e9-d003-4b4e-8a84-83cba7972923</id><parentGroupId>af7523d7-add4-43b6-ba5d-efe10bfd727f</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>af7523d7-add4-43b6-ba5d-efe10bfd727f</groupId><id>a2fc65ea-dea4-4e31-aaae-02adf8cdbce0</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>af7523d7-add4-43b6-ba5d-efe10bfd727f</groupId><id>76d2915c-5ae5-484a-baaf-7640b36ba9e5</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>a7c6a
@mattyb149
mattyb149 / ScriptedHazelcastClient.xml
Created February 11, 2016 00:31
ExecuteScript example for Hazelcast client in JS and Groovy
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template uses Hazelcast JARs in ExecuteScript processors (both in Groovy and Javascript) to generate a flow file from a Hazelcast map.</description><name>ScriptedHazelcastClient</name><snippet><connections><id>840610c3-ce4a-413d-8f54-40ea0a1f1f90</id><parentGroupId>d657baec-19f5-4689-ab5f-f72604055136</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>d657baec-19f5-4689-ab5f-f72604055136</groupId><id>91c52bb1-ba4d-4bce-9d41-e49dd4f03851</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>d657baec-19f5-4689-ab5f-f72604055136</groupId><id>87256841-565d-4559-bb6b-72fd9d6d014b</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>f6c2e6aa-f4f7-4a80-9
@mattyb149
mattyb149 / seed_hazelcast.groovy
Created February 11, 2016 00:10
Groovy script to seed Hazelcast instance for ExecuteScript example
@Grab(group='com.hazelcast', module='hazelcast', version='3.6')
@Grab(group='com.hazelcast', module='hazelcast-client', version='3.6')
import com.hazelcast.client.*
import com.hazelcast.client.config.*
import com.hazelcast.core.*
ClientConfig clientConfig = new ClientConfig();
clientConfig.getGroupConfig().setName("dev").setPassword("dev-pass");
clientConfig.getNetworkConfig().addAddress("192.168.99.100", "192.168.99.100:32780");
@mattyb149
mattyb149 / TellTaleHeartGroovy.xml
Created February 10, 2016 20:29
Apache NiFi template - ExecuteScript (Groovy) Word Count
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This template is a self-contained example showing how to use ExecuteScript (with Groovy) to perform a classic &quot;Word Count&quot; of an incoming flow file (namely The Tell-Tale Heart by Edgar Allan Poe). The output file consists of lines, each containing a word in the corpus and its frequency of occurrence.</description><name>TellTaleHeartGroovy</name><snippet><connections><id>de8c38dd-af7b-4c23-8480-4b83a11635c7</id><parentGroupId>faeccde3-60c2-47af-86c7-3492795ef534</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>faeccde3-60c2-47af-86c7-3492795ef534</groupId><id>388f347c-7e84-44ef-aaa3-ef42565a7cc6</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>faeccde3-60c2-47af-86c7
@mattyb149
mattyb149 / ExecuteScriptHelloWorldGroovy.xml
Created February 10, 2016 18:43
Hello World example for NiFi ExecuteScript processor (with Groovy)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template><description>This is a self-contained example of using a Groovy script in an ExecuteScript processor to update the filename attribute of an incoming flowfile. A dummy JSON file is created (which is overkill for this instance, but is included to encourage playing around with the processor)</description><name>ExecuteScriptHelloWorldGroovy</name><snippet><connections><id>09e8aa98-c9b8-43f0-afbd-513f1d417061</id><parentGroupId>67fb1165-c100-47df-bde3-aa0a4f57af67</parentGroupId><backPressureDataSizeThreshold>0 MB</backPressureDataSizeThreshold><backPressureObjectThreshold>0</backPressureObjectThreshold><destination><groupId>67fb1165-c100-47df-bde3-aa0a4f57af67</groupId><id>8e8aa799-d1ab-4f97-ab35-b06a5a104ae2</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>67fb1165-c100-47df-bde3-aa0a4f57af67</groupId><i