The fine folks at Datamountaineer have developed the stream-reactor making it easier to integrate data piplines with Kafka.
As a demo I'm streaming Twitter data to Kafka, and then from Kafka to RethinkDB.
Configs using Landoop's fast-data-dev environment:
Source: Twitter
name=TwitterSourceConnector
connector.class=com.eneco.trading.kafka.connect.twitter.TwitterSourceConnector
| # Start a search against multiple sites from the command line | |
| # Usage: python duckshift.py <search terms> | |
| import sys | |
| import subprocess | |
| import webbrowser | |
| try: | |
| from urllib.parse import quote | |
| except ImportError: | |
| from urllib import quote # python 2 |
| # Diff pip freeze after pip install or uninstall | |
| # Example usage: pif install foo | |
| pif() { | |
| if [[ $@ == *'install'* ]]; then | |
| r1=$(mktemp) && pip freeze > r1 | |
| pip "$@" | |
| r2=$(mktemp) && pip freeze > r2 | |
| echo ' Pip freeze diff:' | |
| diff r1 r2 | |
| rm -f r1 r2 |