joeljacobson

Self Service Demo

This demo illustrates how a central platform team can provide govenance and control within an organisation enabling application teams to deploy, manage and share Kafka resources within their environments, empowering them to act autonomously.

Accompanying deck - https://docs.google.com/presentation/d/1qi4ufN0T3anZO8MS-BAG5qwWSmmdMperguAsSypFU78/edit#slide=id.g3275e09e81c_0_113

For this demo we will have 3 terminal windows open. The first one is for the central platform team to configure predefined rules for the application teams. The second terminal represents an application team called Orders, and the last represents a Business Intelligence application team.

First, let's start Conduktor Scale, add a sink connector, and add the Self Service demo.
./start.sh\

joeljacobson

Self-Service Demo

Demonstrate Conduktor Self-Service capabilities using three applications:

Order app: owning all resources starting with order-. BI app: own all resources starting with bi, also show-cases that we can own more than one topic-prefix, which is useful when there is no strict topic naming standard on the clusters. (JDBC) sink-connector app: this application is just a consumer and does not own any resources. It's owners can still request access to topics being owned by other applications.

Run the following:

joeljacobson

Data and Innovation Summit hands-on Workshop

Lab 1

CREATE SCHEMA demo_datalake_catalog.transactions;

Lab 2

CREATE TABLE demo_datalake_catalog.transactions."orders" (custkey ,orderkey , totalprice , orderdate ,orderstatus, orderyear , ordermonth) WITH (partitioned_by = ARRAY[ 'orderyear', 'ordermonth' ] , format= 'ORC') AS SELECT custkey, orderkey, totalprice,orderdate,orderstatus , extract( year from orderdate) , extract (month from orderdate) FROM tpch.tiny.orders;

joeljacobson

Setup

These instructions are for DC/OS 1.11.0 and Kubernetes package 1.0.0-1.9.3.

Service Accounts

Install the DC/OS Enterprise CLI, then create a keypair and a service account for Kubernetes. Securely store the private key in the DC/OS secrets store.

joeljacobson

Latency Comparison Numbers
--------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

joeljacobson

#Use Case

A bank wants to help locate and tag all their expenses/transactions in their bank account to help them categorise their spending. The users will be able to tag any expense/transaction to allow for efficient retrieval and reporting. There will be 10 millions customers with on average 500 transactions a year. Some business customers may have up to 10,000 transactions a year. The client wants the tagged items to show up in searches in less than a second to give users a seamless experience between devices.

The client would like

1. to understand how this can be done with DSE
2. some latency examples of how they can search all of a users data
3. the user to be able to filter the queries using time as well as tags e.g. show me all shopping vs luxury in the last 3 months.
4. some assurances that peak traffic of over 10,000 writes, 4,000 reads and 1000 searches per second can be accommodated by the solution using DSE.

joeljacobson

#!/bin/bash
# Maintainer Joel Jacobson

VERSION="5.0.0"

       # to start the services you must be root
       if [ "$EUID" -ne 0 ]; then
       echo "Please run as root"
       exit 1
       fi

joeljacobson

docker rmi -f $(docker images -f "dangling=true" -q)

joeljacobson

### DML ###

# Keyspace Name
keyspace: stresscql

# The CQL for creating a keyspace (optional if it already exists)
keyspace_definition: |
  CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};

# Table name

joeljacobson

1  cd Downloads/
    2  chmod +x NVIDIA-Linux-x86_64-352.30.run 
    3  sudo ./NVIDIA-Linux-x86_64-352.30.run 
    4  cd
    5  cat /var/log/nvidia-installer.log 
    6  sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
    7  sudo apt-get update
    8  sudo apt-get install nvidia-current
    9  cd Downloads/
   10  sudo ./NVIDIA-Linux-x86_64-352.30.run --no-x-check