Created
December 8, 2017 13:19
-
-
Save smudge202/48e49f1354a59412941f796a5ffe4ad8 to your computer and use it in GitHub Desktop.
Test shenanigans
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <Project Sdk="Microsoft.NET.Sdk"> | |
| <PropertyGroup> | |
| <TargetFramework>netcoreapp2.0</TargetFramework> | |
| <RuntimeFrameworkVersion>2.0.3</RuntimeFrameworkVersion> | |
| <IsPackable>false</IsPackable> | |
| </PropertyGroup> | |
| <ItemGroup> | |
| <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.5.0" /> | |
| <PackageReference Include="xunit" Version="2.3.1" /> | |
| <PackageReference Include="xunit.runner.visualstudio" Version="2.3.1" /> | |
| <DotNetCliToolReference Include="dotnet-xunit" Version="2.3.1" /> | |
| </ItemGroup> | |
| </Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| using System; | |
| using System.Collections; | |
| using System.Collections.Generic; | |
| using System.Linq; | |
| using System.Net; | |
| using Xunit; | |
| namespace TestDiffRepro.Tests | |
| { | |
| public class UnitTest1 | |
| { | |
| [Theory, ClassData(typeof(TheoryRepro))] | |
| public void Test1(HttpStatusCode statusCode) | |
| { | |
| Assert.True(true); | |
| } | |
| } | |
| public class TheoryRepro : IEnumerable<object[]> | |
| { | |
| private static readonly IEnumerable<object[]> Data = Enum.GetValues(typeof(HttpStatusCode)) | |
| .Cast<HttpStatusCode>() | |
| .Select(statusCode => new object[] { statusCode }); | |
| public IEnumerator<object[]> GetEnumerator() => Data.GetEnumerator(); | |
| IEnumerator IEnumerable.GetEnumerator() => Data.GetEnumerator(); | |
| } | |
| } |
TestRunners will agree on count if Distinct applied to the data:
private static readonly IEnumerable<object[]> Data = Enum.GetValues(typeof(HttpStatusCode))
.Cast<HttpStatusCode>()
.Distinct()
.Select(statusCode => new object[] { statusCode });As Matt Ellis points out on twitter this is likely something to do with shared status codes (i.e. Ambiguous & MultipleChoices both have the value 300).
Resharper still fails one of the tests consistently with the message Aborted even after the Distinct change..
Logs from Resharper EAP 6:
2017.12.08 14:00:02.302 INFO DiscoveryStage completed
2017.12.08 14:00:02.303 INFO BuildPipelineStage started
2017.12.08 14:00:02.310 INFO BuildPipelineStage completed
2017.12.08 14:00:02.310 INFO ApplyRulesStage started
2017.12.08 14:00:02.317 INFO Got 44 elements (1 explicit) to run
2017.12.08 14:00:02.318 VERBOSE Marking 44 elements as Pending
2017.12.08 14:00:02.338 VERBOSE Marking elements as Pending completed
2017.12.08 14:00:02.341 VERBOSE Distributing 44 elements into runs
2017.12.08 14:00:02.389 VERBOSE Distributing elements completed
2017.12.08 14:00:02.389 INFO Created 1 unit-test runs
2017.12.08 14:00:02.390 INFO ApplyRulesStage completed
2017.12.08 14:00:02.390 INFO RunTestsStage started
2017.12.08 14:00:02.412 VERBOSE Running up to 1 unit-test runs in parallel
2017.12.08 14:00:02.417 INFO Run: df207619-a3f9-4b6b-948c-02c5eb7ec010 - Started
2017.12.08 14:00:02.418 VERBOSE Provider: xUnit.net
Project: TestDiffRepro.Tests
Target Framework: .NETCoreApp,Version=v2.0
Strategy: JetBrains.ReSharper.UnitTestProvider.Xunit.DotNetCore.XunitDotNetVsTestRunStrategy
Runtime Enviroment:
Platform Type: x64
Platform Version: v4_0
2017.12.08 14:00:02.433 TRACE Register handler for message type 'TestSession.Message', protocol version '1'
2017.12.08 14:00:02.433 TRACE Register handler for message type 'ProtocolVersion', protocol version '<unset>'
2017.12.08 14:00:02.433 TRACE Register handler for message type 'TestSession.Connected', protocol version '<unset>'
2017.12.08 14:00:02.433 TRACE Register handler for message type 'TestExecution.CustomTestHostLaunch', protocol version '1'
2017.12.08 14:00:02.434 TRACE Register handler for message type 'TestExecution.Completed', protocol version '1'
2017.12.08 14:00:02.434 TRACE Register handler for message type 'TestExecution.StatsChange', protocol version '1'
2017.12.08 14:00:02.436 INFO Starting process: dotnet vstest /port:52930
2017.12.08 14:00:02.438 INFO Started process with pid 8972
2017.12.08 14:00:02.687 TRACE Processing message 'TestSession.Connected'
2017.12.08 14:00:02.687 TRACE Processed message 'TestSession.Connected'
2017.12.08 14:00:02.694 TRACE Processing message 'ProtocolVersion'
2017.12.08 14:00:02.696 INFO Using test adapter: C:\Users\tommy.local\source\repos\TestDiffRepro\TestDiffRepro.Tests\bin\Debug\netcoreapp2.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll
2017.12.08 14:00:02.743 TRACE Processed message 'ProtocolVersion'
2017.12.08 14:00:02.824 TRACE Processing message 'TestExecution.CustomTestHostLaunch'
2017.12.08 14:00:02.826 TRACE Processed message 'TestExecution.CustomTestHostLaunch'
2017.12.08 14:00:02.830 INFO Starting process: C:\Program Files\dotnet\dotnet.exe exec --runtimeconfig "C:\Users\tommy.local\source\repos\TestDiffRepro\TestDiffRepro.Tests\bin\Debug\netcoreapp2.0\TestDiffRepro.Tests.runtimeconfig.json" --depsfile "C:\Users\tommy.local\source\repos\TestDiffRepro\TestDiffRepro.Tests\bin\Debug\netcoreapp2.0\TestDiffRepro.Tests.deps.json" "C:\Users\tommy.local\.nuget\packages\microsoft.testplatform.testhost/15.5.0\lib/netstandard1.5/testhost.dll" --port 52933 --parentprocessid 8588
2017.12.08 14:00:02.831 INFO Started process with pid 5620
2017.12.08 14:00:03.701 TRACE Processing message 'TestSession.Message'
2017.12.08 14:00:03.701 INFO [xUnit.net 00:00:00.4212567] Starting: TestDiffRepro.Tests
2017.12.08 14:00:03.701 TRACE Processed message 'TestSession.Message'
2017.12.08 14:00:03.922 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:00:03.950 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:00:03.952 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:00:03.954 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:00:03.962 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:00:03.964 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:00:03.975 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:00:03.978 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:00:03.980 TRACE Processing message 'TestSession.Message'
2017.12.08 14:00:03.980 INFO [xUnit.net 00:00:00.6489009] Finished: TestDiffRepro.Tests
2017.12.08 14:00:03.980 TRACE Processed message 'TestSession.Message'
2017.12.08 14:00:04.006 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:00:04.008 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:00:04.018 INFO Process C:\Program Files\dotnet\dotnet.exe:5620 has exited
2017.12.08 14:00:04.022 INFO Run: df207619-a3f9-4b6b-948c-02c5eb7ec010 - Completed
2017.12.08 14:00:04.022 INFO RunTestsStage completed
2017.12.08 14:00:04.038 INFO Finishing run
2017.12.08 14:00:04.041 WARN Element TestDiffRepro.Tests.UnitTest1 was left running after its run completion.
2017.12.08 14:00:04.041 WARN Element TestDiffRepro.Tests.UnitTest1.Test1 was left running after its run completion.
2017.12.08 14:00:04.042 INFO Run is finished
Logs from Resharper EAP 9:
2017.12.08 14:16:20.219 TRACE Processing message 'TestSession.Connected'
2017.12.08 14:16:20.220 TRACE Processed message 'TestSession.Connected'
2017.12.08 14:16:20.226 TRACE Processing message 'ProtocolVersion'
2017.12.08 14:16:20.228 INFO Using test adapter: C:\Users\tommy.local\source\repos\TestDiffRepro\TestDiffRepro.Tests\bin\Debug\netcoreapp2.0\xunit.runner.visualstudio.dotnetcore.testadapter.dll
2017.12.08 14:16:20.283 TRACE Processed message 'ProtocolVersion'
2017.12.08 14:16:20.378 TRACE Processing message 'TestExecution.CustomTestHostLaunch'
2017.12.08 14:16:20.380 TRACE Processed message 'TestExecution.CustomTestHostLaunch'
2017.12.08 14:16:20.384 INFO Starting process: C:\Program Files\dotnet\dotnet.exe exec --runtimeconfig "C:\Users\tommy.local\source\repos\TestDiffRepro\TestDiffRepro.Tests\bin\Debug\netcoreapp2.0\TestDiffRepro.Tests.runtimeconfig.json" --depsfile "C:\Users\tommy.local\source\repos\TestDiffRepro\TestDiffRepro.Tests\bin\Debug\netcoreapp2.0\TestDiffRepro.Tests.deps.json" "C:\Users\tommy.local\.nuget\packages\microsoft.testplatform.testhost/15.5.0\lib/netstandard1.5/testhost.dll" --port 53198 --parentprocessid 20380 --diag "C:\Users\tommy.local\AppData\Local\Temp\JetLogs\vstest.2017-12-08T14-16-19.host.17-12-08_14-16-20_33531_7.log"
2017.12.08 14:16:20.386 INFO Started process with pid 5284
2017.12.08 14:16:20.762 TRACE Processing message 'TestSession.Message'
2017.12.08 14:16:20.762 INFO Logging TestHost Diagnostics in file: C:\Users\tommy.local\AppData\Local\Temp\JetLogs\vstest.2017-12-08T14-16-19.host.17-12-08_14-16-20_33531_7.log
2017.12.08 14:16:20.762 TRACE Processed message 'TestSession.Message'
2017.12.08 14:16:21.330 TRACE Processing message 'TestSession.Message'
2017.12.08 14:16:21.330 INFO [xUnit.net 00:00:00.4359111] Starting: TestDiffRepro.Tests
2017.12.08 14:16:21.330 TRACE Processed message 'TestSession.Message'
2017.12.08 14:16:21.558 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:16:21.597 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:16:21.599 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:16:21.601 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:16:21.602 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:16:21.604 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:16:21.611 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:16:21.613 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:16:21.616 TRACE Processing message 'TestSession.Message'
2017.12.08 14:16:21.616 INFO [xUnit.net 00:00:00.6650847] Finished: TestDiffRepro.Tests
2017.12.08 14:16:21.616 TRACE Processed message 'TestSession.Message'
2017.12.08 14:16:21.642 TRACE Processing message 'TestExecution.StatsChange'
2017.12.08 14:16:21.643 TRACE Processed message 'TestExecution.StatsChange'
2017.12.08 14:16:21.666 INFO Process C:\Program Files\dotnet\dotnet.exe:5284 has exited
2017.12.08 14:16:21.669 INFO Run: 9044eed2-5bac-407e-b358-99cac0b26dac - Completed
2017.12.08 14:16:21.669 INFO RunTestsStage completed
2017.12.08 14:16:21.690 INFO Finishing run
2017.12.08 14:16:21.693 WARN Element TestDiffRepro.Tests.UnitTest1 was left running after its run completion.
2017.12.08 14:16:21.694 WARN Element TestDiffRepro.Tests.UnitTest1.Test1 was left running after its run completion.
2017.12.08 14:16:21.696 INFO Run is finished
Pastebin of full log file for the EAP 9 test run: https://pastebin.com/EynMDkwa
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 test runners reporting 3 different result sets. Why!?