Davide Icardi davideicardi

Alpakka HBase connector

In this example we create an Akka Sink that write message to HBase.

Add the following dependencies:

Simple solution to use Alpakka Kafka connector to produce and consume kafka messages.

I assume that you have 2 scala apps, a producer and a consumer.

Add the following dependencies:

Assume that you have the following directory structure:

This installation of Cloudera is for development only, do not use it in production.

Steps to convert .csproj to new VS 2017 format (.NET 4.5 to .NET Core)

mongod --config d:\data\mongodb\mongod-cluster1.cfg

	import akka._
	import akka.stream._
	import akka.stream.scaladsl._
	import akka.actor._

	// WARNING: Ordering is not guaranteed.
	// The problem with this Graph is that when splitting the flow
	// we obtain 2 concurrent flows.
	// The result of conditionalFlow can returns after another messages is processed in the brodcast.
	// ie. If you have 10 messages from 0 to 9. Let's say that we run a conditional flow only for element 0.

	/*
	https://scalafiddle.io/sf/sniohcZ/3
	(old version: https://scalafiddle.io/sf/sniohcZ/1)

	Use PassThroughFlow when you have a message that should be used in a
	flow that trasform it but you want to maintain the original message for
	another following flow.
	For example if you consume messages from Kafka (CommittableMessage).
	You process the message (transform, save it inside a database, ...) and then you need again the original message
	to commit the offset.

	<?xml version="1.0" encoding="UTF-8" ?>
	<!--
	Licensed to the Apache Software Foundation (ASF) under one or more
	contributor license agreements. See the NOTICE file distributed with
	this work for additional information regarding copyright ownership.
	The ASF licenses this file to You under the Apache License, Version 2.0
	(the "License"); you may not use this file except in compliance with
	the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	// This method convert a json to a valid string json
	// It should handle all mongodb data types correctly (Date, ObjectId, ...) thanks to the
	// Newtonsoft.Json.Bson.BsonReader serializer

	public string ToJson(BsonDocument bson)
	{
	using (var stream = new MemoryStream())
	{
	using (var writer = new BsonBinaryWriter(stream))
	{