Joel Jacobson joeljacobson

#Use Case

A bank wants to help locate and tag all their expenses/transactions in their bank account to help them categorise their spending. The users will be able to tag any expense/transaction to allow for efficient retrieval and reporting. There will be 10 millions customers with on average 500 transactions a year. Some business customers may have up to 10,000 transactions a year. The client wants the tagged items to show up in searches in less than a second to give users a seamless experience between devices.

The client would like

to understand how this can be done with DSE
some latency examples of how they can search all of a users data
the user to be able to filter the queries using time as well as tags e.g. show me all shopping vs luxury in the last 3 months.
some assurances that peak traffic of over 10,000 writes, 4,000 reads and 1000 searches per second can be accommodated by the solution using DSE.

Setup

These instructions are for DC/OS 1.11.0 and Kubernetes package 1.0.0-1.9.3.

Service Accounts

Install the DC/OS Enterprise CLI, then create a keypair and a service account for Kubernetes. Securely store the private key in the DC/OS secrets store.

Data and Innovation Summit hands-on Workshop

Lab 1

CREATE SCHEMA demo_datalake_catalog.transactions;

Lab 2

CREATE TABLE demo_datalake_catalog.transactions."orders" (custkey ,orderkey , totalprice , orderdate ,orderstatus, orderyear , ordermonth) WITH (partitioned_by = ARRAY[ 'orderyear', 'ordermonth' ] , format= 'ORC') AS SELECT custkey, orderkey, totalprice,orderdate,orderstatus , extract( year from orderdate) , extract (month from orderdate) FROM tpch.tiny.orders;

Self-Service Demo

Demonstrate Conduktor Self-Service capabilities using three applications:

Order app: owning all resources starting with order-. BI app: own all resources starting with bi, also show-cases that we can own more than one topic-prefix, which is useful when there is no strict topic naming standard on the clusters. (JDBC) sink-connector app: this application is just a consumer and does not own any resources. It's owners can still request access to topics being owned by other applications.

Run the following:

Self Service Demo

This demo illustrates how a central platform team can provide govenance and control within an organisation enabling application teams to deploy, manage and share Kafka resources within their environments, empowering them to act autonomously.

Accompanying deck - https://docs.google.com/presentation/d/1qi4ufN0T3anZO8MS-BAG5qwWSmmdMperguAsSypFU78/edit#slide=id.g3275e09e81c_0_113

For this demo we will have 3 terminal windows open. The first one is for the central platform team to configure predefined rules for the application teams. The second terminal represents an application team called Orders, and the last represents a Business Intelligence application team.

First, let's start Conduktor Scale, add a sink connector, and add the Self Service demo.
./start.sh\

	### DML ###

	# Keyspace Name
	keyspace: stresscql

	# The CQL for creating a keyspace (optional if it already exists)
	keyspace_definition: \|
	CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};

	# Table name

	#!/bin/bash
	# Maintainer Joel Jacobson

	VERSION="5.0.0"

	# to start the services you must be root
	if [ "$EUID" -ne 0 ]; then
	echo "Please run as root"
	exit 1
	fi

	Latency Comparison Numbers
	--------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD