Gabor Somogyi gaborgsomogyi

    @tailrec
    def logExceptionChain(e: Throwable): Unit = {
      logError("Exception in chain: %s(%s)".format(e.getClass.getName, e.getMessage))
      if (e.getCause != null) {
        logExceptionChain(e.getCause)
      }
    }
    logExceptionChain(e)

AdminClient

As a general rule, the AdminClient protocol is backward compatible (protocol negotiation happens in the background). The listed APIs used by Spark are working with metadata API and the latest AdminClient is compatible with it (please see the bullet points for details).

listTopics:
describeTopics:
These APIs are working with metadata API, which is an integral part of Kafka almost from the beginning. The functionality has been added in Kafka 0.8.1 in KAFKA-1227, 17263. Since it exists there were no breaking changes introduced.
listOffsets:
This API is working with metadata API, which is an integral part of Kafka almost from the beginning. The functionality has been added in Kafka 0.10.1 in KAFKA-4148, 1852 and [1897](https://github.com/apache/kaf

cd "c:\Program Files\Oracle\VirtualBox"
VBoxManage internalcommands createrawvmdk -filename "%USERPROFILE%"\.VirtualBox\usb1.vmdk -rawdisk \\.\PhysicalDrive1

CDP Private Cloud Base 7.1.5
Cloudera Runtime Update 7.2.3.0

KafkaConsumer

assign, subscribe, subscribePattern:
- On topic resourse describe operation
seek, seekToBeginning, seekToEnd
- On topic resourse describe operation
poll
- On group resourse read operation
- On topic resourse read operation
assignment: None
pause: None

Commands

Spark console

git co https://github.com/gaborgsomogyi/spark.git
cd spark
sbuild -d
sbin/start-master.sh
sbin/start-worker.sh spark://gsomogyi-MBP16:7077

Check that http://localhost:8080/ is open and the worker is available.

Kafka auto topic creation is an interesting and somewhat really dangerous feature. I would like to write down my findings related this area.

Feature description

Broker and Consumer side KIP
Producer side KIP
AdminClient side: Not supported in 2.6

Behavior and danger

The feature is simple when a Producer or Consumer has an assignment with a broker and detects that a topic is deleted

Set environment variables

USERPROFILE=/database/config/db2inst1/sqllib/userprofile
echo "export DB2_KRB5_PRINCIPAL=db2/[email protected]" >> $USERPROFILE
echo "export KRB5_KTNAME=/var/custom/db2.keytab" >> $USERPROFILE

Pass evironment variables (variables starting with DB2 passed automatically)

su - db2inst1 -c "db2set DB2ENVLIST=KRB5_KTNAME"

Download .rpm package here
- curl -O https://download.oracle.com/otn_software/linux/instantclient/193000/oracle-instantclient19.3-basic-19.3.0.0.0-1.x86_64.rpm
- curl -O https://download.oracle.com/otn_software/linux/instantclient/193000/oracle-instantclient19.3-devel-19.3.0.0.0-1.x86_64.rpm
- curl -O https://download.oracle.com/otn_software/linux/instantclient/193000/oracle-instantclient19.3-sqlplus-19.3.0.0.0-1.x86_64.rpm
apt-get -y install alien libaio1
Convert the rpm files and install them
- alien -i oracle-instantclient*-basic-*.rpm
- alien -i oracle-instantclient*-devel-*.rpm
- alien -i oracle-instantclient*-sqlplus-*.rpm
ldconfig

	Format:
	appattempt_{clusterTimestamp}_{appId}_{attemptId}
	e.g.: appattempt_1410901177871_0001_01.

	attemptId starts from 1.