Updating in a loop benchmarks

Program.cs

using System.Linq;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Diagnostics.Windows;
using BenchmarkDotNet.Running;

namespace ParallelForEachUpdateBenchmark
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var config = ManualConfig.Create(DefaultConfig.Instance);

            // uncomment code below to faster results, but at the cost of accuracy

            /*
            config.Add(Job.Default
                        .WithLaunchCount(1)     // benchmark process will be launched only once
                        .WithIterationTime(100) // 100ms per iteration
                        .WithWarmupCount(3)     // 3 warmup iteration
                        .WithTargetCount(3)     // 3 target iteration
                    );
            */

            config.Add(StatisticColumn.Max);

            // MemoryDiagnoser and the parallel library combined withe the default job results in the some
            // super long running benchmarks. Bewarned
            // config.Add(new MemoryDiagnoser());

            BenchmarkRunner.Run<UpdateBenchmark>(config);
        }
    }

    public class UpdateBenchmark
    {
        [Params(
            1000,
            10000,
            100000,
            1000000)]
        public int BatchSize { get; set; }
        private MyData[] _data;

        [Setup]
        public void SetupData()
        {
            _data = new MyData[BatchSize];
            for (var i = 0; i < BatchSize; i++)
            {
                _data[i] = new MyData();
            }
        }

        [Benchmark]
        public void ParallelUpdate()
        {
            Parallel.ForEach(_data, i => i.Success = true);
        }

        [Benchmark]
        public void Linq()
        {
            _data.ToList().ForEach(i => i.Success = true);
        }

        [Benchmark]
        public void ForEach()
        {
            foreach (var myData in _data)
            {
                myData.Success = true;
            }
        }

        [Benchmark(Baseline = true)]
        public void ForLoop()
        {
            for (var i = 0; i < _data.Length; i++)
            {
                _data[i].Success = true;
            }
        }
    }

    public class MyData
    {
        public string A { get; set; } = "Foo";
        public string B { get; set; } = "Bar"; 
        public bool Success { get; set; }
    }
}

Also the LINQ test is a bit unfair as it's allocating extra memory by creating a copy of the list first (call to ToList()), which adds overhead.

Although I'm not sure of a good way to avoid this, as it's what using LINQ forces you to do in this case.

Thanks for the feedback. So this is in release mode on my machine, but when I originally posted the screenshot I was, however, on a local Hyper-V instance. I've since learned my lesson and running it outside of the VM gives me this result which is more closely in line with your results.

BenchmarkDotNet=v0.9.6.0
OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz, ProcessorCount=4
Frequency=3417961 ticks, Resolution=292.5721 ns, Timer=TSC
HostCLR=MS.NET 4.0.30319.42000, Arch=32-bit RELEASE
JitModules=clrjit-v4.6.1055.0

Type=UpdateBenchmark  Mode=Throughput  

Method	BatchSize	Median	StdDev	Scaled	Max
ParallelUpdate	1000	16,319.4656 ns	360.6425 ns	52.82	16,742.6630 ns
Linq	1000	2,900.9831 ns	20.9845 ns	9.39	2,950.4241 ns
ForEach	1000	250.9420 ns	14.3245 ns	0.81	297.6356 ns
ForLoop	1000	308.9535 ns	13.1557 ns	1.00	355.8867 ns

I think this is kind of interesting, but I'm not 100% sure what to make of it. I suspect I'm getting faster ForEach / ForLoop perf on my box compared to you just due to raw CPU speed. But the numbers on the parallel work are pretty significantly slower for you compared to me which I think is kind of neat. I wonder if this is a situation where you having more cores than me causes the TPL to have to juggle more threads and overhead causing the slow down. I kind of expect that, and it's one of the reasons I wrote the "benchmark". One of my predecessors spread the TPL as almost "free performance". The fact that we are getting wildly different results here was kind of the outcome I was hoping for, but probably an abuse of the tool all things being equal. Too many variables and external things going on with that little one line of code for it to give good numbers consistently so I'd be hard pressed to call this a true benchmark of one algorithm vs another. But that's what I wanted my team to be thinking about - there is "stuff" the runtime needs to do behind the scenes that has costs, and in this case pretty significant costs compared to the cheapness of a good ol' for loop.

The "think about what happens" rational is the same reason I threw that ToList() example in there. Just a quick addition to get my team thinking that just because it is only 8 characters to type on the keyboard doesn't mean it's a cheap operation to perform for runtime. Not really fair to "benchmark" .ForEach vs for each and for because we can look up the underlying implementation pretty easily and see it's just a for loop under the covers. But there are many out there that are going to see that method and think "it is in the framework, so it must be there for a reason and that reason almost certainly is performance" so I wanted to nudge people to start thinking outside of that world view.

Too many variables and external things going on with that little one line of code for it to give good numbers consistently so I'd be hard pressed to call this a true benchmark of one algorithm vs another.

Agree, BenchmarkDotNet doesn't help as much for "large" benchmarks, with lots of variations run-to-run, it's sweet spot is micro-benchmarks.

But that's what I wanted my team to be thinking about - there is "stuff" the runtime needs to do behind the scenes that has costs, and in this case pretty significant costs compared to the cheapness of a good ol' for loop.

That's a good aim, I hope you managed to get that across!!

But there are many out there that are going to see that method and think "it is in the framework, so it must be there for a reason and that reason almost certainly is performance" so I wanted to nudge people to start thinking outside of that world view.

Likewise, that's what I'm trying to do with BenchmarkDotNet, i.e. give people tools to figure this out!