ijokarumawak

Sample NiFi Kafka data flow to verify Producer/Consumer flow file counts

1. ProduceKafka -> topic A
2. topic A -> ConsumeKafka -> ProduceKafka -> topic B
3. topic B -> ConsumeKafka

The GenerateFlowFile processor is configured to generate one Flow File every second. Started Consumers before starting GenerateFlowFile.

racheliurui

Nifi Kafka Cluster Test Cases

Topology

Due to environment limitation, the test environment are two Linux servers creating an Ambari cluster with,

• A zookeeper cluster shared by nifi and Kafka cluster,
• A nifi cluster with two nodes,
• A Kafka cluster

Message Sequencing Tests

ijokarumawak

NiFi example template, using Wait and Notify with different counter names

Configurations to note:

• Notify
  • Release Signal Identifier: ${batch.id}
  • Signal Counter Name: ${filetype}
• Wait
  • Release Signal Identifier: ${batch.id}
• Signal Counter Name: blank (to count total)

mattyb149

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.2">
    <description>This template gives an example of using a ScriptedLookupService to provide sequence numbers from a database.</description>
    <groupId>5d7e55f8-0165-1000-1170-d42741815ddb</groupId>
    <name>SequenceLookupExample</name>
    <snippet>
        <connections>
            <id>0cf6aae2-8e5a-361f-0000-000000000000</id>
            <parentGroupId>cedd92a0-903e-3f68-0000-000000000000</parentGroupId>
            <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>