Skip to content

Instantly share code, notes, and snippets.

@robinkraft
Last active December 15, 2015 13:38
Show Gist options
  • Save robinkraft/5268273 to your computer and use it in GitHub Desktop.
Save robinkraft/5268273 to your computer and use it in GitHub Desktop.
Stacktrace for issue with Kryo/Cascading/Cascalog in forma-clj (as of March 28, 2013).

Discussion here. Fix seems to have been rolled out in Kryo 2.17, which is picked up as a dep in Cascalog 1.10.1. Possible fix in forma-clj project here

cascading.pipe.OperatorException: [1eceaf05-5138-48a1-af0...][cascalog.workflow$buffer$fn__4479.invoke(workflow.clj:249)] operator Every failed executing operation: ClojureBuffer[decl:'?pixel-idx', '?start', '?end', '?series']
  at cascading.flow.stream.BufferEveryWindow.receive(BufferEveryWindow.java:139)
	at cascading.flow.stream.BufferEveryWindow.receive(BufferEveryWindow.java:41)
	at cascading.flow.hadoop.stream.HadoopGroupGate.run(HadoopGroupGate.java:90)
	at cascading.flow.hadoop.FlowReducer.reduce(FlowReducer.java:129)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:527)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:428)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.esotericsoftware.kryo.util.IntArray.resize(IntArray.java:185)
	at com.esotericsoftware.kryo.util.IntArray.add(IntArray.java:60)
	at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:745)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:713)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:111)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:634)
	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:634)
	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:634)
	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:515)
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:612)
	at cascading.kryo.KryoDeserializer.deserialize(KryoDeserializer.java:42)
	at cascading.tuple.hadoop.TupleSerialization$SerializationElementReader.read(TupleSerialization.java:590)
	at cascading.tuple.hadoop.io.HadoopTupleInputStream.readType(HadoopTupleInputStream.java:105)
	at cascading.tuple.hadoop.io.HadoopTupleInputStream.getNextElement(HadoopTupleInputStream.java:52)
	at cascading.tuple.io.TupleInputStream.readTuple(TupleInputStream.java:78)
	at cascading.tuple.hadoop.io.TupleDeserializer.deserialize(TupleDeserializer.java:40)
	at cascading.tuple.hadoop.io.TupleDeserializer.deserialize(TupleDeserializer.java:28)
	at org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1271)
	at org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1211)
	at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:255)
	at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:251)
	at cascading.flow.hadoop.util.TimedIterator.next(TimedIterator.java:74)
	at cascading.flow.hadoop.HadoopGroupByClosure$1.next(HadoopGroupByClosure.java:79)
	at cascading.flow.hadoop.HadoopGroupByClosure$1.next(HadoopGroupByClosure.java:69)
	at cascading.pipe.joiner.InnerJoin$JoinIterator.next(InnerJoin.java:119)
	at cascading.pipe.joiner.InnerJoin$JoinIterator.next(InnerJoin.java:48)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment