CLONE REPO

git clone [email protected]:basho-labs/marqs-ingest.git

SET ENV VARS

Use python script to query Marathon and set env vars (or maybe do it instead in each step so they're not dependent on this step?)
- KAFKA_BROKERS
- KAFKA_TOPIC
- MESOS_MASTER
- RIAK_DIRECTOR_HOST
- RIAK_HTTP_PORT
- RIAK_PB_PORT
- RIAK_TABLE
- ZOOKEEPER

CREATE TOPIC

docker run -it -rm basho/dcos

dcos kafka topic add...

WRITE MESSAGES TO TOPIC

docker run *some-dockerhub-image*/kafka (or maybe we make our own kafka docker image?

command line producer write contents of data-file-from-repo: /opt/kafka_2.10-0.8.2.2/bin/kafka-console-producer.sh --broker-list $KAFKA_BROKERS --topic $KAFKA_TOPIC < /data-file-from-repo.json

CREATE TABLE

docker run -it basho/riak-python

import riak

marathonAPI = 'http://marathon.mesos:8080'
url = '%s/v2/apps/riak-director/tasks' % marathonAPI

j = json.load(urllib2.urlopen(url))

host = j['tasks'][0]['host']
# the protobuf port is the second item in the list of ports
pb_port =  j['tasks'][0]['ports'][1]

hostAndPort = ':'.join([host, str(pb_port)])

client = riak.RiakClient(host=host, pb_port=pb_port)
tableName = 'ingest9999'
table = client.table(tableName)

create_sql = '''CREATE TABLE %(tableName)s (
site varchar not null,
species varchar not null,
measurementDate timestamp not null,
value double,
PRIMARY KEY ((site, species, quantum(measurementDate, 24, h)),
    site, species, measurementDate))
''' % ({'tableName': tableName})

result = table.query(create_sql)

SUBMIT INGEST TASK TO MARATHON

Use https://github.com/basho-labs/marqs-ingest/blob/master/src/test/resources/marathon.json

docker run spark/dcos-spark

dcos marathon app add marathon.json

WAIT FOR INGEST TO RUN

sleep 180

MAKE SURE ALL DATA HAS MADE THE TRIP

check all test data has been written to riak
compare to data file from github