Document with requirements for Xamarin.Android's APK tests to run on Test Cloud

XA-TestCloud.md

Xamarin.Android on Test Cloud

Leveraging Test Cloud seems like a good idea, this document aims at figuring out the requirements for our team.

Current APK Tests

Xamarin.Android has various APK-based tests that currently run on emulators/devices.

In general, they operate by running adb commands:

adb install sometestapp.apk
adb shell am instrument -w $(InstrumentationType)
adb pull /path/to/TestResults.xml

The tests APKs are using NUnitLite, its instrumentation runner drops the test results in a directory on-device. Jenkins ingests the NUnit reports after they are pulled from the device.

Requirements for Test Cloud

We need a way to:

• Run these NUnitLite on-device tests in Test Cloud
• Get the on-device test results locally
• Should work within a CI build from Jenkins or VSTS

There are a few problems here:

• Test Cloud operates by running an NUnit project on desktop
• How do the on-device tests get back to the desktop machine in Test Cloud?
• If running test-cloud.exe (CI machine, etc.), how do these test results get back to the calling machine?

Parallelization (Nice to Have)

• Is there a way Test Cloud can parallelize these tests? In particular, the BCL tests are slow due to the number of tests.
• Test Cloud parallelizes tests bases on test fixture or test methods. But since the tests are on-device, how will this work?

Exploration

I took a stab at running a set of APK tests on Test Cloud. Github repo here.

My approach was:

• Modify XA's NUnitLite (code here), that adds a "backdoor" to run the tests and return the NUnit xml results as a string
• Write a desktop NUnit project with a single test that calls the "backdoor" method
• Then what do we do with the XML? For now I printed them to the console.

See a successful test run here. View the Test Log to see the on-device test results.