Demo Time

000_overview.md

┌──Payments───────┐                     _._
│ ┌─────────────┐ │                _.-``__ ''-._
│ │  Terminal   │ │           _.-``    `.  `_.  ''-._
│ └─────────────┘ │       .-`` .-```.  ```\/    _.,_ ''-._
└────────┬────────┘      (    '      ,       .-`  | `,    )
         │               |`-._`-...-` __...-.``-._|'` _.-'|
         │               |    `-._   `._    /     _.-'    |
         ▼                `-._    `-._  `-./  _.-'    _.-'
┌─────────────────┐      |`-._`-._    `-.__.-'    _.-'_.-'|
│  Apache Kafka   │◀─────|    `-._`-._        _.-'_.-'    |
└─────────────────┘       `-._    `-._`-.__.-'_.-'    _.-'
         ▲               |`-._`-._    `-.__.-'    _.-'_.-'|
         │               |    `-._`-._        _.-'_.-'    |
         │                `-._    `-._`-.__.-'_.-'    _.-'
         │                    `-._    `-.__.-'    _.-'
         │                        `-._        _.-'
         │                            `-.__.-'
         │
  ┌─────────────┐      ┌───────────────┐    ┌─────────────┐
  │  Logstash   │─────▶│ Elasticsearch │◀───│   Kibana    │
  └─────────────┘      └───────────────┘    └─────────────┘

001_data-stream.md

Apache Kafka

Confluent Open Source Platform

cd demos/confluent-5.1.0
bin/confluent start

Dockerized OSS Kafka cluster

cd development/projects/kafka-docker-compose/standalone
docker-compose up -d

Producer

Start Producer

source activate rasp_pi
python auto_producer.py

Open Control Center, check topic auto_trnx

002_flask-output.md

Redis

redis-server /usr/local/etc/redis.conf

Consumer

cd ~/lucky_lunch/streaming-demo
source activate rasp_pi
python auto_consumer.py

Flask

Open flask

cd ~/lucky_lunch/streaming-demo
source activate rasp_pi && cd flask
python run.py

004_es-demo.md

Logstash Stream

Endpoint Elastic Cloud

# Start Logstash demo
cd demos/logstash-6.5.4
bin/logstash -f ../logstash-config/finance/streaming-payments.conf

Output

Open Kibana

300_fraud-detection-overview.md

                                               ┌────────────────┐
┌───────────┐                            ┌────▶│     Issuer     │
│           │                            │     └────────────────┘
│ Merchant  │   ┌───────────────────┐    │
│ Terminal  │──▶│ Payment Processor │────┤
│           │   └───────────────────┘██  │
└───────────┘     ████████│████████████  │     ┌────────────────┐
                  ████████│████████████  └────▶│Fraud Detection │
                          │                    └────────────────┘
                          │                             │
                          ▼                             │
                ┌───────────────────┐                   │
                │   Elasticsearch   │◀──────────────────┘
                └───────────────────┘

301_demo.md

Shipper

Start filebeat for ingesting fraud transactions

cd lucky_lunch/filebeat-6.4.3-darwin-x86_64
./filebeat

Data

Open project test-data-generator Run FraudStatisticClient

Open Fraud Monitor

template_payments.txt

PUT _template/payments
{
  "index_patterns": [
    "payments-*"
  ],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "doc": {
      "dynamic": "strict",
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "keyword"
        },
        "name": {
          "type": "text",
          "analyzer": "english",
          "fields": {
            "keyword": {
              "type": "keyword"
            }
          }
        },
        "geocode": {
          "type": "geo_point"
        },
        "address": {
          "properties": {
            "city": {
              "type": "keyword"
            },
            "state": {
              "type": "keyword"
            },
            "street_number": {
              "type": "keyword"
            },
            "zip": {
              "type": "keyword"
            },
            "street_name": {
              "type": "text",
              "analyzer": "english"
            }
          }
        },
        "amount": {
          "type": "float"
        }
      }
    }
  }
}