demo2.adoc

Cassandra and Kafka workshop demo instructions

We recommend doing all the hands-on work including registering for Instaclustr account from Ubuntu VM image that was provided to you.

If you still do not have this image, then ask and we will provide the OVA image file to you.

Opening Instaclustr console website in VM makes it easy to copy connection credentials when connecting to Cassandra or Kafka clusters.

Signup for Instaclustr account

• Go to https://instaclustr.com

• Sign up for a new account by clicking on Login / Signup

• Choose Sign up

• Fill the form. Please note:

  • E-mail Address: Use your company email address. Instaclustr automatically enables free trials for company email domains; this is not the case Gmail, Hotmail, etc …​ if you do not have a company email address, please sign-up with a personal account and also notify your instructor.

  • How Did You Find Us: Please pick "Other" and type "Free your city workshop".

How to create your Instaclustr free trial cluster

• Sign in to your Instaclustr account

• Click Create Cluster, Unless instructed otherwise, leave all options as default.

• Give your cluster a name

• Choose your Base Application according to the workshop: Cassandra or Kafka and ensure the latest version is selected.

• Add-ons: do not select any

• Select AWS as the Infrastructure Provider

• Ask your instructor for the Region to use

• Pick "Starter" t2.small for the Node Size. This is the only one that offers free trial.

• Leave RF to 3 and chose 3 nodes (already predefined).

• Security: Make sure to check Add <current IP address> to cluster firewall allowed addresses under Settings, at worse you can add 0.0.0.0/0 to enable all IP addresses to connect from later under Settings after cluster is provisioned. For knowing what your IP address is, just type "what is my ip" in any browser.

• Accept the Terms and Conditions, and click on Create Cluster.

Note	Wait a few minutes for the cluster to provision. If there are any issues, let your instructor know.

Note

Ask your instructor for help if needed. The most likely issues for clusters not provisioning are: You signed up with a non-company email address, and free trial is not enabled, ask the instructor for help. Instaclustr has reached capacity in a given AWS region. Ask the instructor if another region should be used.

Using Instaclustr Managed C* cluster

Note	At this moment, you should have both Cassandra and Kafka clusters provisioned and they should not be deferred state but in running.

• Connect to cluster using already downloaded cqlsh tool using commands below.

Note	Password for Cassandra is in Connection info tab after cluster is provisioned. As host, choose the public address of any node.

Open Terminal and enter:

$ cqlsh -u iccassandra -p [password] [host]

For example, when fully specified, it would look like:

$ cqlsh -u iccassandra -p mysupersecretpassword 50.20.30.40

After successful connection, you will see CQL shell:

Connected to mycluster at 52.42.55.85:9042.
[cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
iccassandra@cqlsh>

Once connected and in CQL shell, let’s create our keyspace.

Warning

You need to know what data center your cluster is in, you get this information from Connection Info from Name. E.g. "AWS_VPC_US_WEST_2". This data center will be likely different on the workshop itself, it depends on where that workshop is run.

After you know what DC you have your cluster in, copy this snippet into CQLSH.

CREATE KEYSPACE training_ks
WITH replication = {'class': 'NetworkTopologyStrategy', 'NAME OF YOUR DC': 3};

Next, create table (copy this snippet and paste it into CQLSH):

CREATE TABLE training_ks.sensor_events (
   sensor_id text,
   event_ts timestamp,
   reading double,
   PRIMARY KEY (sensor_id, event_ts));

Let’s insert some data into your newly created table:

INSERT INTO training_ks.sensor_events (sensor_id,event_ts,reading)
VALUES ('S123456','2019-04-25T10:02:03.123', 5.67);

INSERT INTO training_ks.sensor_events (sensor_id,event_ts,reading)
VALUES ('S123456','2019-04-25T10:02:03.124', 2.35);

INSERT INTO training_ks.sensor_events (sensor_id,event_ts,reading)
VALUES ('S123457','2019-04-25T10:03:03.124', 4.56);

You see what you inserted by:

iccassandra@cqlsh:training_ks> SELECT * FROM training_ks.sensor_events ;

 sensor_id | event_ts                        | reading
-----------+---------------------------------+---------
   S123456 | 2019-04-25 10:02:03.123000+0000 |    5.67
   S123456 | 2019-04-25 10:02:03.124000+0000 |    2.35
   S123457 | 2019-04-25 10:03:03.124000+0000 |    4.56

You can check your created schema by invoking:

DESCRIBE training_ks;

Using Instaclustr Managed Kafka cluster

Create Kafka cluster similarly as you created Cassandra one and edit file kafka.properties in home directory

instac@instac-ubuntu:~$ vim kafka.properties

The content will be there already, just update password:

security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="ickafka" \
    password="password";

Verify that you can connect to Kafka cluster by creating a topic for your application.

Note	Supply node IP from console, take public ip, any node is just fine.

$ ic-kafka-topics.sh --bootstrap-server <your_node_IP>:9092 --properties-file kafka.properties --create --topic events --replication-factor 3 --partitions 3

You can check that your topic was created by

$ ic-kafka-topics.sh --bootstrap-server <your_node_IP>:9092 --properties-file kafka.properties --list

Project walk-through - eventsproducer

The objective of this application is to produce a message which will be sent to Kafka topic we just created above.

This will be achieved by sending a message to REST endpoint of a web application based on Spring Boot framework.

Start IntelliJ IDEA from application panel, there are two projects already. Open eventsproducer one.

Note

You have to modify the properties of the application in order to successfully connect to Kafka cluster. There is file under src/main/resources/application.yaml. Follow the instructions here and run that application by either via Application class or by run configuration which is already pre-configured.

Sending of an event to Kafka topic from Postman

In order to see incoming messages into events topic, start console consumer first in shell:

$ kafka-console-consumer.sh --bootstrap-server YOUR.IP.FROM.CONSOLE:9092 --consumer.config kafka.properties --topic events

We will use Postman to reach application’s /events endpoint.

• Create new POST request by duplicating existing POST Ingest events request by right click on the request tab.

• Specify Body as shown below.

• Make sure Headers has one entry with key set to 'Content-Type' and value is set to application/json.

• Make sure timestamp is in correct format, e.g. dd-MM-yyyy HH:mm:ss.SSS

• Once ready, send it via Send button.

If all went fine, status should be `201 - Created `.

Check that your event is present in Kafka’s topic, you should see something like:

{"id": "S1234", "timestamp": "10-10-2019 10:04:07.000", "reading": 0.26 }

Project walk-through - eventsconsumer

The objective of this application is to persist an event into Cassandra table.

This will be achieved by consuming a message from Kafka topic and saving it into Cassandra table.

Open eventsconsumer application by going to File → Open and navigate to /home/instac/dev/training/eventsconsumer.

Note

You have to modify the properties of the application in order to successfully connect to Cassandra and Kafka cluster. There is file under src/main/resources/application.yaml. Follow the instructions here and run that application by either via Application class or by run configuration which is already pre-configured.

Receiving of an event from Kafka topic and

Now if you send an event to Postman, it should go to Kafka topic and then it should be read by consumer application and it will appear in training_ks.sensor_events table.