holgerbrandl’s gists

holgerbrandl / Univocity.kt

Created September 5, 2018 20:05

univocity as file reader backend for krangl

	package krangl.experimental

	import com.univocity.parsers.common.record.Record
	import com.univocity.parsers.tsv.TsvParser
	import com.univocity.parsers.tsv.TsvParserSettings
	import krangl.*
	import java.io.Reader

	/**
	* @author Holger Brandl

holgerbrandl / mpg.txt

Created August 24, 2018 20:11

Fuel economy data from 1999 and 2008 for 38 popular models of car This dataset contains a subset of the fuel economy data that the EPA makes available on http://fueleconomy.gov. It contains only models which had a new release every year between 1999 and 2008 - this was used as a proxy for the popularity of the car. Exported from R with: readr::w…

	manufacturer model displ year cyl trans drv cty hwy fl class
	audi a4 1.8 1999 4 auto(l5) f 18 29 p compact
	audi a4 1.8 1999 4 manual(m5) f 21 29 p compact
	audi a4 2 2008 4 manual(m6) f 20 31 p compact
	audi a4 2 2008 4 auto(av) f 21 30 p compact
	audi a4 2.8 1999 6 auto(l5) f 16 26 p compact
	audi a4 2.8 1999 6 manual(m5) f 18 26 p compact
	audi a4 3.1 2008 6 auto(av) f 18 27 p compact
	audi a4 quattro 1.8 1999 4 manual(m5) 4 18 26 p compact
	audi a4 quattro 1.8 1999 4 auto(l5) 4 16 25 p compact

holgerbrandl / kshell_0.2_kscript_header.kts

Created August 21, 2018 08:51

kshell-0.2-Snaphot dependency heaer for kscript

	@file:DependsOn("org.apache.hadoop:hadoop-common:2.7.0")
	@file:DependsOn("com.github.khud:kshell-repl-api:0.2.2-1.2.41")
	@file:DependsOn("sparklin:jline3-shaded:0.2-SNAPSHOT")
	@file:DependsOn("sparklin:kshell:0.2-SNAPSHOT")

holgerbrandl / kafka_string_serializer_not_found.kts

Created June 18, 2018 08:07

Example to illustrate issue when running kafka code with kotlin

	#!/usr/bin/env kscript

	@file:DependsOn("org.apache.kafka:kafka-clients:1.1.0")

	import java.util.Properties
	import org.apache.kafka.clients.producer.KafkaProducer
	import org.apache.kafka.clients.producer.ProducerRecord
	import org.apache.kafka.common.serialization.StringSerializer

	val properties = Properties()

holgerbrandl / kshell_blastn.kts

Created June 14, 2018 10:56

Custom Kshell-Launcher Example

	// repl class-path
	@file:DependsOn("org.apache.hadoop:hadoop-common:2.7.0")
	@file:DependsOn("com.github.khud:kshell-repl-api:0.2.2-1.2.41")
	@file:DependsOn("sparklin:jline3-shaded:0.2-SNAPSHOT")
	@file:DependsOn("sparklin:kshell:0.2-SNAPSHOT")


	@file:DependsOn("de.mpicbg.scicomp:kutils:0.10")
	@file:DependsOn("com.github.holgerbrandl:kscript-support:1.2.4")
	@file:DependsOn("de.mpicbg.scicomp.joblist:joblist-kotlin:1.2")

holgerbrandl / kshell_launcher.kts

Created June 6, 2018 08:09

Launch kshell with a maven resolved classpath

	#!/usr/bin/env kscript

	// repl class-path
	@file:DependsOn("org.apache.hadoop:hadoop-common:2.7.0")
	@file:DependsOn("com.github.khud:kshell-repl-api:0.2.2-1.2.41")
	@file:DependsOn("sparklin:jline3-shaded:0.2-SNAPSHOT")
	@file:DependsOn("sparklin:kshell:0.2-SNAPSHOT")

	// user classpath
	@file:DependsOn("com.offbytwo:docopt:0.6.0.20150202","log4j:log4j:1.2.14")

holgerbrandl / IrisShape.kt

Last active May 15, 2018 16:44

example to illustrate issue with RecordReaderDataSetIterator

	package org.deeplearning4j.datasets.datavec

	import org.datavec.api.records.reader.impl.csv.CSVRecordReader
	import org.datavec.api.split.FileSplit
	import org.nd4j.linalg.io.ClassPathResource

	fun main(args: Array<String>) {

	val recordReader = CSVRecordReader(0, ',').apply{
	// initialize(FileSplit(File("/Users/brandl/projects/deep_learning/dl4j/dl4j-test-resources/target/classes/iris.txt")))

holgerbrandl / RecorderReaderExample.kt

Last active May 9, 2018 08:53

	package playground

	import DATA_ROOT
	import org.datavec.api.io.labels.PathLabelGenerator
	import org.datavec.api.split.CollectionInputSplit
	import org.datavec.api.writable.IntWritable
	import org.datavec.api.writable.Writable
	import org.datavec.image.recordreader.ImageRecordReader
	import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator
	import java.io.File

holgerbrandl / gist:095c8eb7a486626676882c94269bc741

Created April 25, 2018 04:56

Thread dump of deadlock in org.deeplearning4j.streaming.kafka.NDArrayPublisherTests#testPublish

	Full thread dump

	"main@1" prio=5 tid=0x1 nid=NA waiting
	java.lang.Thread.State: WAITING
	at sun.misc.Unsafe.park(Unsafe.java:-1)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.camel.component.seda.SedaPollingConsumer.receive(SedaPollingConsumer.java:41)
	at org.apache.camel.impl.ConsumerCache.receive(ConsumerCache.java:202)

holgerbrandl / gist:821313db8ecaf17134b2b3ce089eb111

Created April 24, 2018 16:05

thread dump of deadlock in org.deeplearning4j.bagofwords.vectorizer.TfidfVectorizerTest#testTfIdfVectorizer

	"main@1" prio=5 tid=0x1 nid=NA waiting
	java.lang.Thread.State: WAITING
	at java.lang.Object.wait(Object.java:-1)
	at java.lang.Object.wait(Object.java:502)
	at org.deeplearning4j.models.word2vec.wordstore.VocabConstructor$VocabRunnable.awaitDone(VocabConstructor.java:536)
	at org.deeplearning4j.models.word2vec.wordstore.VocabConstructor.buildJointVocabulary(VocabConstructor.java:221)
	at org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer.buildVocab(BaseTextVectorizer.java:53)
	at org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer.fit(BaseTextVectorizer.java:58)
	at org.deeplearning4j.bagofwords.vectorizer.TfidfVectorizerTest.testTfIdfVectorizer(TfidfVectorizerTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)

Holger Brandl holgerbrandl