Many people naively assume an ordinary .NET HashSet preserves insertion order. Indeed HashSet accidentally preserves insertion order until you remove and re-add some elements. There is such a data structure in Java - LinkedHashSet which respects order and has O(1) RW times.
No, I did not find a (working) corresponding implementation in .NET. That's I wrote this one.
The implementation uses linked list in combination with dictionary to define the iteration, ordering and at the same time allow O(1) removal.
The order is not affected if an element is re-inserted into the set it retains it's old position.