Skip to content

Instantly share code, notes, and snippets.

View JensRantil's full-sized avatar

Jens Rantil JensRantil

231 followers · 99 following
View GitHub Profile
@JensRantil
JensRantil / channel_demo.go
Last active August 18, 2019 01:44
Simplification of sample code at https://blog.gopheracademy.com/advent-2015/automi-stream-processing-over-go-channels/ Best practise is 1) to inject channels and 2) avoid concurrency in APIs.
func ingest(out <-chan []string) {
out <- []string{"aaaa", "bbb"}
out <- []string{"cccccc", "dddddd"}
out <- []string{"e", "fffff", "g"}
close(out)
}
func process(in <-chan []string, out <-chan int) {
for data := range in {
for _, word := range data {
@JensRantil
JensRantil / kafka-broker.log
Created December 11, 2015 16:43
kafka-broker.log
[2015-12-11 13:52:12,502] INFO [GroupCoordinator 0]: Preparing to restabilize group default with old generation 0 (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:52:12,503] INFO [GroupCoordinator 0]: Stabilized group default generation 1 (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:52:12,505] INFO [GroupCoordinator 0]: Assignment received from leader for group default for generation 1 (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:53:12,507] INFO [GroupCoordinator 0]: Preparing to restabilize group default with old generation 1 (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:53:12,508] INFO [GroupCoordinator 0]: Group default generation 1 is dead and removed (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:53:12,687] INFO [GroupCoordinator 0]: Preparing to restabilize group default with old generation 0 (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:53:12,687] INFO [GroupCoordinator 0]: Stabilized group default generation 1 (kafka.coordinator.GroupCoordinator)
[2015-12-11 13:53:12,
@JensRantil
JensRantil / kafka-remarks.md
Created December 3, 2015 12:19
Kafka conclusions regarding at-most-once delivery.

Asked a short question on Facebook for Kafka expireriences. Got this response:

Jag skulle nog använda en kö som jobbkö istället. Det är som vanligt lite dåliga defaults tex. Man måste tänka på om man behöver bry sig om ordning på medelanden, hur man partitionerar etc. Sen även om ni uppdaterar offsetten innan processning kan jag tänka mig att vid en rebalance tex så kan ni lyckas få samma medelande mer än en gång. Jag skulle nog tänka två gånger innan jag använde något som är gjort för at least once när jag behöver at most once. Det går säker, men känns inte optimalt.

It triggered me to ask this on #apache-kafka on IRC:

Hi. We are looking into using Kafka (partially) as a task queue. Crucial here is that we want at-most once delivery. Our plan is to simply poll a couple of tasks, manually updating the offset and then handling the tasks. Are there any pitfa

@JensRantil
JensRantil / example.java
Last active November 22, 2015 22:02
Spark example
import org.apache.bcel.util.ByteSequence;
minRdd.map(new Function<String, ByteSequence> {
public ByteSequence call(String str) {
return new ByteSequence(str.toBytes(Encodings.UTF-8));
}
});
@JensRantil
JensRantil / gist:6dff9729079d04165e6f
Last active November 19, 2015 22:13
Why https://github.com/rssvihla/spark_commons/blob/master/examples/maven_example/pom.xml doesn't work with DSE 4.7. This is an answer to http://stackoverflow.com/questions/33782857/can-i-use-regular-spark-core-application-library-to-talk-to-dse-spark?noredirect=1#comment55337542_33782857.

The serialization version that https://github.com/rssvihla/spark_commons/blob/master/examples/maven_example/pom.xml is using:

$ serialver -classpath ./spark-core_2.10-1.2.1.jar:./scala-library-2.10.5.jar org.apache.spark.deploy.ApplicationDescription
org.apache.spark.deploy.ApplicationDescription:    static final long serialVersionUID = 7674242335164700840L;

The serialization version that spark-core_2.10.1.2.2 is using (the latest published 1.2 Spark Core artifact):

$ serialver -classpath ./spark-core_2.10-1.2.2.jar:./scala-library-2.10.5.jar org.apache.spark.deploy.ApplicationDescription
org.apache.spark.deploy.ApplicationDescription:    static final long serialVersionUID = 7674242335164700840L;
@JensRantil
JensRantil / master.log
Created November 18, 2015 14:38
Spark master exception.
ERROR [sparkMaster-akka.actor.default-dispatcher-2] 2015-11-17 17:50:42,442 Slf4jLogger.scala:66 - org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 7674242335164700840, local class serialVersionUID = 2812534333379744973
java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 7674242335164700840, local class serialVersionUID = 2812534333379744973
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617) ~[na:1.7.0_65]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) ~[na:1.7.0_65]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) ~[na:1.7.0_65]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) ~[na:1.7.0_65]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ~[na:1.7.0_65]
at java.io
@JensRantil
JensRantil / askpasswd.sh
Created September 22, 2015 12:55
Script to generate salted SHA512 /etc/shadow password on/for Ubuntu 12.04 LTS
#!/bin/bash
read -p "Enter username: " username
read -s -p "Enter password: " mypassword
echo
echo -n $username:$mypassword | chpasswd -S -c SHA512
@JensRantil
JensRantil / curator_exception.txt
Created September 13, 2015 16:03
Curator exception trickling through to our application
Sep 10 10:02:18 10.11.1.42 myproc[17660]: [my-thread-8] org.apache.curator.ConnectionState Connection timed out for connection string (X.X.X.2:2181,X.X.X.3:2181,X.X.X.4:2181) and timeout (15000) / elapsed (15521)
Sep 10 10:02:18 10.11.1.42 org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
Sep 10 10:02:18 10.11.1.42 at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197)
Sep 10 10:02:18 10.11.1.42 at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87)
Sep 10 10:02:18 10.11.1.42 at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
Sep 10 10:02:18 10.11.1.42 at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492)
Sep 10 10:02:18 10.11.1.42 at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:238)
Sep 10 10:02:18 10.11.1.42 at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(Dele
@JensRantil
JensRantil / retryer.sh
Created September 3, 2015 13:37
A wrapping retrying script.
local runscript=1
while [ $runscript -eq 1 ]; do
./your-script
local exitcode=$?
if [ $status -eq 0 ]; then
runscript=1
else
runscript=0
fi
done
@JensRantil
JensRantil / error_example.txt
Created August 3, 2015 16:45
WARN 14:55:36,093 At level 1, SSTableReader(path='/var/lib/cassandra/data/XXX/YYY/XXX-YYY-jb-13742-Data.db') [DecoratedKey(-9222888731763965546, 71855c01897f456bb39064fbb422b2a5), DecoratedKey(-6917488716996414524, 311af6e84b9b4245bb8f30cbe0426711)] overlaps SSTableReader(path='/var/lib/cassandra/data/XXX/YYY/XXX-YYY-jb-13436-Data.db') [DecoratedKey(-9222888731763965546, 71855c01897f456bb39064fbb422b2a5), DecoratedKey(-6880442365347943082, 9eab2913983541fb93f6ab8bb88938af)]. This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0. If you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable