Holger Brandl holgerbrandl
holgerbrandl
/ Univocity.kt
Created
September 5, 2018 20:05
univocity as file reader backend for krangl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package krangl.experimental | |
| import com.univocity.parsers.common.record.Record | |
| import com.univocity.parsers.tsv.TsvParser | |
| import com.univocity.parsers.tsv.TsvParserSettings | |
| import krangl.* | |
| import java.io.Reader | |
| /** | |
| * @author Holger Brandl |
holgerbrandl
/ mpg.txt
Created
August 24, 2018 20:11
Fuel economy data from 1999 and 2008 for 38 popular models of car This dataset contains a subset of the fuel economy data that the EPA makes available on http://fueleconomy.gov. It contains only models which had a new release every year between 1999 and 2008 - this was used as a proxy for the popularity of the car. Exported from R with: readr::w…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| manufacturer model displ year cyl trans drv cty hwy fl class | |
| audi a4 1.8 1999 4 auto(l5) f 18 29 p compact | |
| audi a4 1.8 1999 4 manual(m5) f 21 29 p compact | |
| audi a4 2 2008 4 manual(m6) f 20 31 p compact | |
| audi a4 2 2008 4 auto(av) f 21 30 p compact | |
| audi a4 2.8 1999 6 auto(l5) f 16 26 p compact | |
| audi a4 2.8 1999 6 manual(m5) f 18 26 p compact | |
| audi a4 3.1 2008 6 auto(av) f 18 27 p compact | |
| audi a4 quattro 1.8 1999 4 manual(m5) 4 18 26 p compact | |
| audi a4 quattro 1.8 1999 4 auto(l5) 4 16 25 p compact |
holgerbrandl
/ kshell_0.2_kscript_header.kts
Created
August 21, 2018 08:51
kshell-0.2-Snaphot dependency heaer for kscript
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @file:DependsOn("org.apache.hadoop:hadoop-common:2.7.0") | |
| @file:DependsOn("com.github.khud:kshell-repl-api:0.2.2-1.2.41") | |
| @file:DependsOn("sparklin:jline3-shaded:0.2-SNAPSHOT") | |
| @file:DependsOn("sparklin:kshell:0.2-SNAPSHOT") |
holgerbrandl
/ kafka_string_serializer_not_found.kts
Created
June 18, 2018 08:07
Example to illustrate issue when running kafka code with kotlin
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env kscript | |
| @file:DependsOn("org.apache.kafka:kafka-clients:1.1.0") | |
| import java.util.Properties | |
| import org.apache.kafka.clients.producer.KafkaProducer | |
| import org.apache.kafka.clients.producer.ProducerRecord | |
| import org.apache.kafka.common.serialization.StringSerializer | |
| val properties = Properties() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // repl class-path | |
| @file:DependsOn("org.apache.hadoop:hadoop-common:2.7.0") | |
| @file:DependsOn("com.github.khud:kshell-repl-api:0.2.2-1.2.41") | |
| @file:DependsOn("sparklin:jline3-shaded:0.2-SNAPSHOT") | |
| @file:DependsOn("sparklin:kshell:0.2-SNAPSHOT") | |
| @file:DependsOn("de.mpicbg.scicomp:kutils:0.10") | |
| @file:DependsOn("com.github.holgerbrandl:kscript-support:1.2.4") | |
| @file:DependsOn("de.mpicbg.scicomp.joblist:joblist-kotlin:1.2") |
holgerbrandl
/ kshell_launcher.kts
Created
June 6, 2018 08:09
Launch kshell with a maven resolved classpath
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env kscript | |
| // repl class-path | |
| @file:DependsOn("org.apache.hadoop:hadoop-common:2.7.0") | |
| @file:DependsOn("com.github.khud:kshell-repl-api:0.2.2-1.2.41") | |
| @file:DependsOn("sparklin:jline3-shaded:0.2-SNAPSHOT") | |
| @file:DependsOn("sparklin:kshell:0.2-SNAPSHOT") | |
| // user classpath | |
| @file:DependsOn("com.offbytwo:docopt:0.6.0.20150202","log4j:log4j:1.2.14") |
holgerbrandl
/ IrisShape.kt
Last active
May 15, 2018 16:44
example to illustrate issue with RecordReaderDataSetIterator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package org.deeplearning4j.datasets.datavec | |
| import org.datavec.api.records.reader.impl.csv.CSVRecordReader | |
| import org.datavec.api.split.FileSplit | |
| import org.nd4j.linalg.io.ClassPathResource | |
| fun main(args: Array<String>) { | |
| val recordReader = CSVRecordReader(0, ',').apply{ | |
| // initialize(FileSplit(File("/Users/brandl/projects/deep_learning/dl4j/dl4j-test-resources/target/classes/iris.txt"))) |
holgerbrandl
/ RecorderReaderExample.kt
Last active
May 9, 2018 08:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package playground | |
| import DATA_ROOT | |
| import org.datavec.api.io.labels.PathLabelGenerator | |
| import org.datavec.api.split.CollectionInputSplit | |
| import org.datavec.api.writable.IntWritable | |
| import org.datavec.api.writable.Writable | |
| import org.datavec.image.recordreader.ImageRecordReader | |
| import org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator | |
| import java.io.File |
holgerbrandl
/ gist:095c8eb7a486626676882c94269bc741
Created
April 25, 2018 04:56
Thread dump of deadlock in org.deeplearning4j.streaming.kafka.NDArrayPublisherTests#testPublish
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Full thread dump | |
| "main@1" prio=5 tid=0x1 nid=NA waiting | |
| java.lang.Thread.State: WAITING | |
| at sun.misc.Unsafe.park(Unsafe.java:-1) | |
| at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) | |
| at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) | |
| at org.apache.camel.component.seda.SedaPollingConsumer.receive(SedaPollingConsumer.java:41) | |
| at org.apache.camel.impl.ConsumerCache.receive(ConsumerCache.java:202) |
holgerbrandl
/ gist:821313db8ecaf17134b2b3ce089eb111
Created
April 24, 2018 16:05
thread dump of deadlock in org.deeplearning4j.bagofwords.vectorizer.TfidfVectorizerTest#testTfIdfVectorizer
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| "main@1" prio=5 tid=0x1 nid=NA waiting | |
| java.lang.Thread.State: WAITING | |
| at java.lang.Object.wait(Object.java:-1) | |
| at java.lang.Object.wait(Object.java:502) | |
| at org.deeplearning4j.models.word2vec.wordstore.VocabConstructor$VocabRunnable.awaitDone(VocabConstructor.java:536) | |
| at org.deeplearning4j.models.word2vec.wordstore.VocabConstructor.buildJointVocabulary(VocabConstructor.java:221) | |
| at org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer.buildVocab(BaseTextVectorizer.java:53) | |
| at org.deeplearning4j.bagofwords.vectorizer.BaseTextVectorizer.fit(BaseTextVectorizer.java:58) | |
| at org.deeplearning4j.bagofwords.vectorizer.TfidfVectorizerTest.testTfIdfVectorizer(TfidfVectorizerTest.java:71) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1) |