alopresto / extract_text_regex.xml

Created October 10, 2016 19:21

Apache NiFi template which generates an empty flowfile, populates the contents with an example log HTTP request line, and extracts the HTTP response code.

	<?xml version="1.0" ?>
	<template encoding-version="1.0">
	<description>Generates an empty flowfile, populates the contents with an example log HTTP request line, and extracts the HTTP response code. </description>
	<groupId>b0049ce3-0157-1000-671f-21b7e01c3bcf</groupId>
	<name>ExtractText Regex</name>
	<snippet>
	<connections>
	<id>b0057ecb-0157-1000-0000-000000000000</id>
	<parentGroupId>b0049ce3-0157-1000-0000-000000000000</parentGroupId>
	<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

alopresto / header_and_footer.xml

Last active October 25, 2016 17:51

Apache NiFi template to demonstrate `RouteOnContent` capabilities.

	<?xml version="1.0" ?>
	<template encoding-version="1.0">
	<description>Generates flow files every 3 seconds, creates four copies:

	* Header and footer flag present
	* Header only
	* Footer only
	* Neither

	then uses RouteOnContent with a regex to route those with both header and footer to one LogAttribute, and all others to a different instance. </description>

alopresto / csr-to-keystore.txt

Created October 27, 2016 00:20

Steps and output for generating a public/private keypair, a CSR to get it signed by an external CA, and then importing the signed certificate and certificate chain into a JKS keystore for use in Apache NiFi.

	hw12203:/Users/alopresto/Workspace/scratch/csr-demo (master) alopresto
	🔓 3s @ 16:33:45 $ keytool -genkey -alias nifi -keyalg RSA -keysize 2048 -keystore keystore.jks
	Enter keystore password:
	Re-enter new password:
	What is your first and last name?
	[Unknown]: nifi.apache.org
	What is the name of your organizational unit?
	[Unknown]: NiFi
	What is the name of your organization?
	[Unknown]: Apache

alopresto / Encrypt-attribute-via-execute-script.xml

Last active October 27, 2016 23:37

Apache NiFi template that generates an empty flowfile, populates the content with plaintext, adds two attributes, uses an ExecuteScript processor to perform AES/GCM encryption with a default key, and updates and adds attributes with the cipher text results, then logs the attributes and content of the flowfile.

	<?xml version="1.0" ?>
	<template encoding-version="1.0">
	<description>This template generates an empty flowfile, populates the content with plaintext, adds two attributes, uses an ExecuteScript processor to perform AES/GCM encryption with a default key, and updates and adds attributes with the cipher text results, then logs the attributes and content of the flowfile. </description>
	<groupId>08499ab1-0158-1000-eb96-b892bf593753</groupId>
	<name>EncryptAttribute (via ExecuteScript)</name>
	<snippet>
	<connections>
	<id>084be0c5-0158-1000-0000-000000000000</id>
	<parentGroupId>08499ab1-0158-1000-0000-000000000000</parentGroupId>
	<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

alopresto / cert_commands.md

Created November 23, 2016 03:23

Commonly used certificate and keystore commands for verifying Apache NiFi API/UI TLS MA connections.

Certificate and key commands

1. Verify connection:

$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>

2. Export client cert from PKCS12 keystore to PEM:

alopresto / Encrypt-and-decrypt-attribute-processor-test-template.xml

Created December 6, 2016 04:39

Apache NiFi template to test the `EncryptAttributes` processor submitted in [PR 1294](https://github.com/apache/nifi/pull/1294).

	<?xml version="1.0" ?>
	<template encoding-version="1.0">
	<description>Must set the password and key for various encrypt/decrypt processors. </description>
	<groupId>d07d0681-0158-1000-1581-15fb569a9418</groupId>
	<name>Encrypt and Decrypt Attributes</name>
	<snippet>
	<connections>
	<id>d0813dba-0158-1000-0000-000000000000</id>
	<parentGroupId>d07d0681-0158-1000-0000-000000000000</parentGroupId>
	<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

alopresto / nifi_startup_times.txt

Created January 10, 2017 19:25

Grep for NiFi startup times across current PR and release verification logs.

	hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
	🔓 84s @ 11:21:49 $ find . -name "nifi-app.log" \| xargs grep -E 'Controller initialization took'
	find: ./pr1147/unreadable: Permission denied
	./migration_tmp/ncm/logs/nifi-app.log:2016-07-05 17:03:35,962 INFO [main] org.apache.nifi.NiFi Controller initialization took 10483456126 nanoseconds.
	./migration_tmp/ncm/logs/nifi-app.log:2016-07-05 17:37:28,920 INFO [main] org.apache.nifi.NiFi Controller initialization took 7087194577 nanoseconds.
	./migration_tmp/ncm/logs/nifi-app.log:2016-07-05 17:40:05,406 INFO [main] org.apache.nifi.NiFi Controller initialization took 6943513487 nanoseconds.
	./migration_tmp/ncm/logs/nifi-app.log:2016-07-05 17:41:48,861 INFO [main] org.apache.nifi.NiFi Controller initialization took 6842047525 nanoseconds.
	./migration_tmp/ncm/logs/nifi-app.log:2016-07-05 17:47:09,994 INFO [main] org.apache.nifi.NiFi Controller initialization took 17132013017 nanoseconds.
	./nifi-0.6.1/logs/nifi-app.log:2016-05-31 09:23:49,269 INFO [main] or

alopresto / pycrypto_nifi.xml

Created February 1, 2017 00:45

Generates a key file and an encrypted data file (both placeholders) and puts them in a directory, then watches the directory for those files and crafts a terminal command to execute a Python script with those arguments (currently mocked by 'echo').

	<?xml version="1.0" ?>
	<template encoding-version="1.0">
	<description>Generates a key file and an encrypted data file (both placeholders) and puts them in a directory, then watches the directory for those files and crafts a terminal command to execute a Python script with those arguments (currently mocked by 'echo'). </description>
	<groupId>f6f232be-0159-1000-9698-ab4d847269d1</groupId>
	<name>Pycrypto Demo</name>
	<snippet>
	<connections>
	<id>f6f62cff-0159-1000-0000-000000000000</id>
	<parentGroupId>f6f232be-0159-1000-0000-000000000000</parentGroupId>
	<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>

alopresto / Pycrypto results

Created February 17, 2017 05:27

Results of running a toy Python crypto example.

	hw12203:/Users/alopresto/Workspace/lightweight_crypto (master) alopresto
	🔓 6s @ 21:25:02 $ echo "This is a plaintext message." > plain.txt
	hw12203:/Users/alopresto/Workspace/lightweight_crypto (master) alopresto
	🔓 4s @ 21:25:07 $ ./lightweight_crypto.py plain.txt output.enc
	(E)ncrypt or (D)ecrypt?
	E
	done
	hw12203:/Users/alopresto/Workspace/lightweight_crypto (master) alopresto
	🔓 12s @ 21:25:20 $ more output.enc
	"output.enc" may be a binary file. See it anyway?

alopresto / Log output.txt

Created February 17, 2017 06:13

Example log output after newline replace and multiline split.

	2017-02-16 22:12:09,569 INFO [Timer-Driven Process Thread-4] o.a.n.processors.standard.LogAttribute LogAttribute[id=015a1000-919e-1aa8-b90e-7f11869b6d91] logging for flow file StandardFlowFileRecord[uuid=5cf9b054-a760-46dd-a118-d0bea7dc3bb9,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1487311925514-1, container=default, section=1], offset=520, length=124],offset=0,name=92509598133110,size=124]
	--------------------------------------------------
	Standard FlowFile Attributes
	Key: 'entryDate'
	Value: 'Thu Feb 16 22:12:09 PST 2017'
	Key: 'lineageStartDate'
	Value: 'Thu Feb 16 22:12:09 PST 2017'
	Key: 'fileSize'
	Value: '124'
	FlowFile Attribute Map Content

Andy LoPresto alopresto

Certificate and key commands

1. Verify connection:

2. Export client cert from PKCS12 keystore to PEM: