Some of my thoughts on pickling.

pickling.md

Why Pickling does not solve most of my serialization problems

From what I can tell, scala pickling is two things:

1. a good type that composes well: Pickler
2. Macros for generating that type.

What are my pain points?

I work a lot on chill which is code for configuring Kryo in a composible, reusable (and serializable) way (see KryoInstantiator and KryoRegistrar). Chill also has some tricks to make Kryo work well with most scala objects and functions.

Why would I do such a horrible thing?

Cascading and Storm are distributed systems that Scalding and Summingbird sit on top of. Both require serialization of both logic and data. Users don't want to write serializers for every data object they want.

But wait, does't pickling meet your needs?

As far as I can tell, no. We already have three nice composable types: Bufferable[T], Injection[T, Array[Byte]], and kryo.Serializer[T]. I'd love macros to generate them (as it might help me as the library auther), but there is no short supply of composable types.

Next, users don't want to do anything about serialization. Now, with Kryo, in scalding it usually just works without any action on their part. If we used pickling, I can't yet see a way they don't have to declare all the serialized types. But which need to be serialized? Not many. Only those that cross map/reduce boundaries. Even knowing which those are is complex for users. So, to be safe, they would have to list every single type they use.

Closures

This is the big issue: closures capturing the world. Spores are meant to address this. At first I was disappointed. I hoped there could be a way that better static analysis could limit the scope of closures. I was hoping that if you are closing over x.y and y is a val, we could just automatically do: z = x.y and only keep z. I think this is a win for GC not just serialization. I'm not sure what rule of scala prevents this from being an always on optimization. But even if you limit the scope to what you need, what about those objects that are not java serializable? Does pickling support FunctionN? I didn't think so (that would be a cool macro, pickledFn(x => x*BigObj)).

So...

Maybe I'm just uninformed. My time is quite limited and since Twitter is not running 2.10 internally, this is all academic anyway. I have to find solutions that work on 2.9.2 and 2.9.3. I'm looking forward to more macros, but not convinced I wand to totally replumb chill, scalding, and summingibird with the new hotness.