CI Test Shield - Compliance Levels Proposal

CI Test Shiled Update.md

Problem

The current tests are rather rigid in what they consider passing. Partners are requesting more modular tests that allow for a wider range of hardware to pass.

Proposed Solution

To address this I am proposing a tiered test infrastructure. This would allow us to certify different levels of compliance while providing maximum flexibility. The idea is to have 4 levels.

0. Bare Implementation - First stage of development.

• Constructors / destructors
• Assumptions
• At least 1 of every peripheral possible implimented

1. Basic Compliance - things actually start working.

• Basic Read / Write operations for each API implimented
• Single instance of each peripheral works
• Timing of API measured to 10%

2. mbed Enabled Compliance - required to be mbed enabled, this level must pass for it to be marked public

• At least one of each peripheral is broken out to Arduino Hardware spec header
• Full API implimented
• Every pin available is tested with single instance
• Multiple instances of each peripheral can be implimented at same time
• peripherals work in interrupt context
• Timing of API measured to 5%

3. Corner Case Compliance - Goal level, not everyone will get here, sets goals for the best platforms to achieve

• Edge / Corner case tests
• Timing of API measured to 1%
• Check for corner cases (really beat on the API hard)
• Instantiate everything and test all at once

Outline of tests per level [UPDATED Feb/8/2017]

x	Bare (L0) All: Constructor, Destructor	Basic (L1) All:	Mbed Enabled (L2) All:	Corner Case (L3):
		- Single Instance, full API	- single instances on each possible pin,	- Instantiate and deconstruct multiple instances within same program
		- Run tests 1, 10 times each,	- Run tests 100 times, compliance within 1%	- Try to break things
		- compliance within 5%	- Multiple instances at same time	- Initialize in interrupt context
				- Initialize in callback
Digital IO	-	Check set timings between read / writes	Check random timings between read / writes	-
		- Define Time bounds
Analog In	-	- Read values that are between 0 and 1	Read values	x
		- Make sure values monotonically increase
		- Read value multiple times at each stage, make sure variance is within 5%,1% each time
Analog Out	-	- Increase value monotonically, validate with analog in (verify with return that it doesn’t blow up)	x	x
		- Verify it can be set to 0->1 in 0.1 incriments
I2C	-	- Read / Write 1, 10, 100 bytes using set data	- Read / write 1, 10, 100 bytes using random data to random locations	- Read / write random bytes of random size to random locations quite a few times
		- Write / Read diff data twice to the same location to verify not reading old data	- Write / Read diff data twice to the same location to verify not reading old data	- Write / Read diff data twice to the same location to verify not reading old data
SPI	-	- Read / Write 1 byte, 10 bytes, 100 bytes using set data	- Read / write 1, 10, 100 bytes using random data to random locations	- Read / write random bytes of random size to random locations quite a few times
		- Write / Read diff data twice to the same location to verify not reading old data	- Write / Read diff data twice to the same location to verify not reading old data	- Write / Read diff data twice to the same location to verify not reading old data
PWM	-	- Frequency at 10ms, 30ms, 100ms	- Frequency steps of 1ms from 10 to 200	Random duty cycles with random frequency's to within 1%
		- Duty Cycle at 10%,50%,90%	- Duty Cycle every 5% from 5 to 95
		- Increase monotonically	- Increase monotonically
Interrupt In	-	- Single object with set timings	- Test on all available pins individually, with random timings (one test for each pin, don’t have more than one object at a time)	- Trigger with PWM's
Bus IO	-	Create buses of 1 to 4 pins, read / write	Create busses of 8 - 20 pins, read / write	x
Arduino HW	- Validate pins are defined A0-A5, D0-D15	x	x	x
	- Validate that expected peripherals are on Arduino Pins per R3 spec (analog In on A0-5...etc)
	- If pin is NC then don’t use it in later tests (ie analog Out...etc) (not sure how to do this)
	- Test NC is handled correctly

Goal

Impliment this proposal and release along side the 5.4 mbed-os release in March / April 2017

TODO / To Figure out how to do

• Random Number Generation test ( rand() is not actually random between reboots)
  • Use python to seed random number to rand() ?
• Some way to detect peripherals and pins available.
  • Fully specified mbed_app.json? (too big / complicated?)
  • Generate mbed_app.json from mbed pindefs?

Feedback

This is just a proposal, I would appreciate your feedback, input, and suggestions for either improvements, more robust test requirements, or really anything else you can think of. Please submit feedback in the comments below.

Added initial testing matrix, suggestions appreciated!

We're using the Arduino R3 hardware spec, it is well documented. It is not required to be mbed enabled, but it is strongly suggested.

Again, per Arduino R3 hardware spec, that means I2C, SPI, PWM, DIO, AIO, UART. CAN/Ethernet/Wifi/ are not considered and are not standard peripherals.

I am confused , so hw spec is as I see only about some basic configuration (analog in, dio pins). I could not find A5 to be DAC capable (or PWM in some cases) in the quick search for it thus I asked. If you can pinpoint me to those spec , I'll have a look.

I want to be careful that we do not make the board too configurable. I want some reasonable default assumptions made to prevent this from turning into a nightmare to maintain.

We need to come up with something easy to maintain :-) reasonable default assumptions - sounds good but what these will be based on, those assumptions?
A failing test because a board does not have one PWM pin wired is not something I expect to see (if 2 pins are tested that means PWM peripheral is +/- tested, thus because MCU do not provide one more PWM or bad hw desing, should not result in failure (having this in nightly it would always be red and ignored after some time), we dont do full spec tests (test all pwm peripherals or entire pin configuration) with this testing anyway). I rather see it green with limitations (is testing just one PWM enough? is it two PWM pins?).

Addition: An idea:

Would it make sense to have one test for arduino compability (that one would fail in thecase as not all pins are PWM capable that should be) but PWM test would not fail as it would take input from config file for what pins are available for testing (=PWM is working as it was tested on some pins, but not full set in some cases). Then if we read the test report, it would show this platform does not conform to some assumptions about pinout, but the peripheral driver is green. This would not result in misleading results (red test just because one pin is not wired, etc).

per our conversation, I'm planning on collecting a list of capabilities that individual interfaces should have. Will provide in a table and share with the team for editing.

Let me know once done, I'm interested !

A failing test because a board does not have one PWM pin wired is not something I expect to see

Hence why we break them up into levels, level 1 only tests a single PWM, level 2 tests all the PWM's available per spec, individually, that way if the board doesnt have PWM3 then PWM1,2 pass and PWM3 fails, but only on the level 2 tests, all level 1 tests pass. Ultimately having some way of telling what PWM's the board has would be best, but Im not aware of any HW Descriptor file like that. Something to think about for the future?

Would it make sense to have one test for arduino compability

yes, that is already the intention

but PWM test would not fail as it would take input from config file for what pins are available for testing

this is not possible in the existing framework as far as I can figure. Do you know how to do something like this in greentea? That is have the outcome of one test suite affect the test parameters for another test suite?

Let me know once done, I'm interested !

See table above, went ahead and posted it into the GIST (scroll to the top)

Based on Feedback I have updated the Tests matrix as follows:

• Level 1 tests a single instance of the peripheral on a single pin. Tests both in global and interrupt / sub function context
• Level 2 tests all available instances on pins individually, then tests multiple instances at once. Tests in both global context and interrupt context.
• Level 3 tests multiple instances in multiple contexts

Ultimately I would like a method of dynamically figuring out what pins are available on a platform (ie knowing that there are 3 PWM's on a platform, two of which are broken out to the correct Arduino Headers and should be tested, the others should not be tested). However this is currently not possible. As a workaround untill a solution is found (maybe a hardware description file or something?... tbd, beyond the scope of this) having Level 1 test a single instance while level 2 tests all instances individually will provide a basic way of testing if the underlying software works.
Using this method the above platform would pass PWM 0 / 1 and Fail the other PWM tests because the peripherals would not be on the assumed headers. The Partner Manager would then make a note of this and approve it as an exception, or possibly turn the tests off on PWM 2-4 for that platform.

Does this sound like an acceptable workaround?

Framework now in place to dynamically find available pins of a specific type (i.e. AnalogOut, PWM, DigitalIn) and only use the pins from that list for testing that are connected to the CI test shield.
Latest commit on my repository: https://github.com/mray19027/ci-test-shield/tree/level-testing has examples for Analog, Digital, and PWM. Working on I2C, SPI, and BusInOut next