ApacheNifi

ApachiNifi

Terminology
Write-Ahead Logging
https://www.xuchao.org/docs/postgresql/wal.html

Archi View & Use cases
https://www.slideshare.net/hortonworks/hortonworks-data-in-motion-series-part-4

http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy

Performance Tuning
https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html

====steps for a 2 nifi instances on same server==============
https://pierrevillard.com/2016/08/13/apache-nifi-1-0-0-cluster-setup/

The CN is the Common Name for the certificate. The CN is another name for the site's domain name. OU stands for Organizational Unit and, the most important entry, the O field, which stands for Organization.
1)
   ./bin/tls-toolkit.sh standalone -n 'localhost(2)' -C 'CN=ComName, OU=DepartmentName' -o './target'
   ./bin/tls-toolkit.bat standalone -n 'localhost(2)' -C 'CN=ComName, OU=DepartmentName' -o './target'
2) 
   cp localhost/* nifi-1/conf/
   cp localhost1/* nifi-1/conf/
   after copy, the generated client public key is configured on nifi
3) in nifi.properties
     change nifi config to start as cluster
     enable zookeeper on nifi 1
     enable security
   in zookeeper.properties
    configure zookeeper host and port
   in security related xml file
     configure created user to have access
4) 
   start nifi cluster
5) import client secret key into browser
   visit nifi and configure the security
   
   
   
   
 Ambari install (Hortonworsk)  
 https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.1.2/bk_dataflow-ambari-installation/content/download_the_ambari_repo.html
 
 The servers can ping each other by hostname
 ----all the server must be using same version of Linux
 
 step 1, follow the software requirement to install all the pre-requirements (yum install)
    do for both of the machine
 step 2, Set Up Password-less SSH
  then ,do below to both of the servers
  Set Up Service User Accounts
Enable NTP on the Cluster and on the Browser Host
Check DNS and NSCD
Configuring iptables
Disable SELinux and PackageKit and check the umask Value
 
   
   
 

**in dev env
[NiFi logging handler] org.apache.nifi.StdErr Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "NiFi Web
when too much flow file not being processed accordingly
clean up the flowfile_repository

Nifi Processor Annotation
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#processor-behavior-annotations
EventDriven --- can be scheduled
SideEffectFree -- can be retried
SupportsBatching -- batch processing, multi processing into one transaction (data persistence not guaranteed)
TriggerSerially -- single threading, no concurrent processing, sequence guarenteed
TriggerWhenAnyDestinationAvailable --
TriggerWhenEmpty --- trigger the processor regardless of the inbound data queue status
InputRequirement --- declare the whether the processor need inbound queue in canvas
@tags({"example", "documentation", "developer guide", "processor", "tags"}) , used to filter the processor