Event Processing

EventProcessing.md

Event Sourcing

• It's root in Domain Driven Design community
  • Vaughn Vernon: Implementing Domain-Driven Design
• It isn't a good way to design databases. Instead, we should indivisualy record every change that happen to database.
• Events are immutable
• We don't loose historic information
• storing state:
  • Store raw events (Event Stores)
    • Used by analyst
    • Offline processing
      • Machine Learning
      • Recommender system
    • best for DB Writes (speed) (write optimized)
    • source of truth
    • immutable facts
    • append-only streams of immutable events
  • Aggregated Summaries; Store aggregated data (SQL databases, OLAP cubes, ...)
    • Scanning raw data too slow
    • react in real-time manner
    • Monitoring, alerting, trending
    • redis, memcached (atomic increment operation)
    • best for DB Reads (speed) (read optimized)
    • derived
    • denormilised
    • cached view of the event log
    • normolization make write fast and simple, but it means you must do more work (joins) at read time.
    • duplicate information in various places so tha it can be read faster
  • benefits:
    • Read and Write performance
    • scalability
    • Flexibility and Agility
    • Human Fault tollerance

Event

• timestamp
• action
• properties

Event Store

OlAP Cubes

• Multidimensional cube

Event Stream

• message queue
• event log
• multiple consumer
  • archive row events
  • aggregation
  • monitoring
  • event detection

CEP (Complex Event Processing)

Companies

• LinkedIn

Fan-out

DWH (Data Warehouse)

recovering from failure

Kafka

• kafka client library: kafka streams

Stream Processing Tools

• kafka streams
• Spark Streaming
• Storm
• Samza
• Flink

Stream query engines

• Stream query engines

Stream processing engines

• storm
• s4

CEP (complex event processing)

• SQL like query language
• fraud detection
• monitoring business processes
• full-text search

Actor Frameworks

• akka
• Erlang OTP
• Orleans

Reactive FRP/Dataflow

• bringing event streams to the user interface
• dataflow languages
• ReactiveX
• functional reactive programming (FRP)
• RxJava
• Elm
• Meteor
• React

Database Change Capture

• Golden Gate
• Databus

change data capture (CDC)

SEDA Architecture

Event Store

• append-only
• ordering -> kafka (AMQP or JMS does not support fixed ordering)

data integration (keeping data systems synchronized)

Dual Writes

• less work at first
• race condition problem
• inconsistency

Consistency

• Eventual
• Perpetual

Two Phase commit

Write-ahead log (WAL)