I did not know that System.Drawing is a cheetah at gif, but let it be their problem! :) We need to find a better baseline then!
What was made me to do this benchmark was seeing the OctreeeQuantizer's code. It's is very allocation-heavy. I think we can make a great improvement simply by doing a DOD refactor on the Octree class. It could be a very useful OOP vs DOD case study!
However ...
The numbers make me think there might be also an algorithmical issue. Is the whole algorithm running in O(PixelCount)? Does the tree have a depth limit?
I can also pick this task if you want! :)
I think @dlemstra is concerned about VS2017. For me it's OK, we need to switch sooner or later anyway. But I'm also fine with a custom NuGet package.
I already managed to pack the latest System.Runtime.CompilerServices.Unsafe with different class name to avoid collision with the official class.