Skip to content

Instantly share code, notes, and snippets.

@mannharleen
Created September 20, 2017 03:04
Show Gist options
  • Select an option

  • Save mannharleen/bf2b943f9e090e7e53a40d3ba62f78c1 to your computer and use it in GitHub Desktop.

Select an option

Save mannharleen/bf2b943f9e090e7e53a40d3ba62f78c1 to your computer and use it in GitHub Desktop.
flume: avro source ->jdbc channel->logger sink
agent3.sources = source1
agent3.channels = channel1
agent3.sinks = sink1
agent3.sources.source1.type = avro
agent3.sources.source1.port = 11112
agent3.sources.source1.bind = localhost
agent3.sinks.sink1.type = logger
agent3.channels.channel1.type = jdbc
agent3.sources.source1.channels = channel1
agent3.sinks.sink1.channel = channel1
#Usage:
# flume-ng agent --name agent3 -c . -f agent3.conf
# flume-ng avro-client -H localhost -p 11112 -F file1
# Input (file1): $ cat file1
1 First line
2 Second line
3 Third line
# Output:
17/09/19 19:54:42 INFO ipc.NettyServer: [id: 0x92ef3118, /127.0.0.1:46792 => /127.0.0.1:11112] OPEN
17/09/19 19:54:42 INFO ipc.NettyServer: [id: 0x92ef3118, /127.0.0.1:46792 => /127.0.0.1:11112] BOUND: /127.0.0.1:11112
17/09/19 19:54:42 INFO ipc.NettyServer: [id: 0x92ef3118, /127.0.0.1:46792 => /127.0.0.1:11112] CONNECTED: /127.0.0.1:46792
17/09/19 19:54:43 INFO ipc.NettyServer: [id: 0x92ef3118, /127.0.0.1:46792 :> /127.0.0.1:11112] DISCONNECTED
17/09/19 19:54:43 INFO ipc.NettyServer: [id: 0x92ef3118, /127.0.0.1:46792 :> /127.0.0.1:11112] UNBOUND
17/09/19 19:54:43 INFO ipc.NettyServer: [id: 0x92ef3118, /127.0.0.1:46792 :> /127.0.0.1:11112] CLOSED
17/09/19 19:54:43 INFO ipc.NettyServer: Connection to /127.0.0.1:46792 disconnected.
17/09/19 19:54:46 INFO sink.LoggerSink: Event: { headers:{} body: 31 20 46 69 72 73 74 20 6C 69 6E 65 1 First line }
17/09/19 19:54:46 INFO sink.LoggerSink: Event: { headers:{} body: 32 20 53 65 63 6F 6E 64 20 6C 69 6E 65 2 Second line }
17/09/19 19:54:46 INFO sink.LoggerSink: Event: { headers:{} body: 33 20 54 68 69 72 64 20 6C 69 6E 65 3 Third line }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment