AWS Aurora to Maxwell Kinesis Producer

aws-aurora-to-maxwell-kinesis-producer.md

Prerequisites

1. AWS Aurora to Maxwell STDOUT Producer
2. Run Kafka Container
3. Create an AWS Kinesis Stream
4. Configuring the AWS Command Line Interface

After running through the prerequisites, you will have:

• a Maxwell image named osheroff/maxwell
• an AWS Aurora instance
• a Kafka service named kafka, listening on kafka:9092
• an AWS Kinesis Stream
• AWS Credentials stored in ~/.aws/credentials

Start Maxwell with Kinesis Producer

docker run -it --rm --name maxwell -v `cd && pwd`/.aws:/root/.aws osheroff/maxwell sh -c "\
    cp /app/kinesis-producer-library.properties.example /app/kinesis-producer-library.properties && \
    echo "Region=YOUR_REGION" >> /app/kinesis-producer-library.properties && \
    bin/maxwell --user=AURORA_USER --password=AURORA_PASSWORD --host=AURORA_HOST --producer=kinesis --kinesis_stream=KINESIS_STREAM "

This command will:

• share your host AWS credentials with the Maxwell container
• copy the Kinesis properties file into place
• append YOUR_REGION to the Kinesis properties file
• connect Maxwell to your Aurora instance
• connect Maxwell to your Kinesis Stream

output:

18:39:18,394 INFO  KinesisProducerConfiguration - Attempting to load config from file kinesis-producer-library.properties
18:39:18,425 WARN  KinesisProducerConfiguration - Property Port ignored as there is no corresponding set method in KinesisProducerConfiguration
18:39:18,436 INFO  KinesisProducer - Extracting binaries to /tmp/amazon-kinesis-producer-native-binaries
18:39:19,163 INFO  Daemon - Asking for trace
18:39:19,196 INFO  Maxwell - Maxwell v1.7.0 is booting (MaxwellKinesisProducer), starting at BinlogPosition[mysql-bin-changelog.000002:91000]
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.259719] [0x00000024][0x00007f2cdc4c0780] [info] [logging.cc:83] Set boost log level to info
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.259809] [0x00000024][0x00007f2cdc4c0780] [info] [logging.cc:170] Set AWS Log Level to WARN
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262697] [0x00000024][0x00007f2cdc4c0780] [info] [main.cc:346] Setting CA path to /tmp/amazon-kinesis-producer-native-binaries
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262716] [0x00000024][0x00007f2cdc4c0780] [info] [main.cc:382] Starting up main producer
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262736] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:87] Using Region: us-west-2
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262747] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:48] Using default Kinesis endpoint
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262775] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:87] Using Region: us-west-2
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262782] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:48] Using default CloudWatch endpoint
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262849] [0x00000024][0x00007f2cdc4c0780] [info] [main.cc:393] Entering join
18:39:19,903 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at BinlogPosition[mysql-bin-changelog.000002:3521])
18:39:23,243 INFO  OpenReplicator - starting replication at mysql-bin-changelog.000002:91000

The process is now waiting for new data events.

Create a consumer

...