The current tests are rather rigid in what they consider passing. Partners are requesting more modular tests that allow for a wider range of hardware to pass.
To address this I am proposing a tiered test infrastructure. This would allow us to certify different levels of compliance while providing maximum flexibility. The idea is to have 4 levels.
- Bare Implementation - First stage of development.
- Constructors / destructors
- Assumptions
- At least 1 of every peripheral possible implimented
- Basic Compliance - things actually start working.
- Basic Read / Write operations for each API implimented
- Single instance of each peripheral works
- Timing of API measured to 10%
- mbed Enabled Compliance - required to be mbed enabled, this level must pass for it to be marked public
- At least one of each peripheral is broken out to Arduino Hardware spec header
- Full API implimented
- Every pin available is tested with single instance
- Multiple instances of each peripheral can be implimented at same time
- peripherals work in interrupt context
- Timing of API measured to 5%
- Corner Case Compliance - Goal level, not everyone will get here, sets goals for the best platforms to achieve
- Edge / Corner case tests
- Timing of API measured to 1%
- Check for corner cases (really beat on the API hard)
- Instantiate everything and test all at once
| x | Bare (L0) All: Constructor, Destructor | Basic (L1) All: | Mbed Enabled (L2) All: | Corner Case (L3): |
|---|---|---|---|---|
| - Single Instance, full API | - single instances on each possible pin, | - Instantiate and deconstruct multiple instances within same program | ||
| - Run tests 1, 10 times each, | - Run tests 100 times, compliance within 1% | - Try to break things | ||
| - compliance within 5% | - Multiple instances at same time | - Initialize in interrupt context | ||
| - Initialize in callback | ||||
| Digital IO | - | Check set timings between read / writes | Check random timings between read / writes | - |
| - Define Time bounds | ||||
| Analog In | - | - Read values that are between 0 and 1 | Read values | x |
| - Make sure values monotonically increase | ||||
| - Read value multiple times at each stage, make sure variance is within 5%,1% each time | ||||
| Analog Out | - | - Increase value monotonically, validate with analog in (verify with return that it doesn’t blow up) | x | x |
| - Verify it can be set to 0->1 in 0.1 incriments | ||||
| I2C | - | - Read / Write 1, 10, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| SPI | - | - Read / Write 1 byte, 10 bytes, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| PWM | - | - Frequency at 10ms, 30ms, 100ms | - Frequency steps of 1ms from 10 to 200 | Random duty cycles with random frequency's to within 1% |
| - Duty Cycle at 10%,50%,90% | - Duty Cycle every 5% from 5 to 95 | |||
| - Increase monotonically | - Increase monotonically | |||
| Interrupt In | - | - Single object with set timings | - Test on all available pins individually, with random timings (one test for each pin, don’t have more than one object at a time) | - Trigger with PWM's |
| Bus IO | - | Create buses of 1 to 4 pins, read / write | Create busses of 8 - 20 pins, read / write | x |
| Arduino HW | - Validate pins are defined A0-A5, D0-D15 | x | x | x |
| - Validate that expected peripherals are on Arduino Pins per R3 spec (analog In on A0-5...etc) | ||||
| - If pin is NC then don’t use it in later tests (ie analog Out...etc) (not sure how to do this) | ||||
| - Test NC is handled correctly |
Impliment this proposal and release along side the 5.4 mbed-os release in March / April 2017
- Random Number Generation test ( rand() is not actually random between reboots)
- Use python to seed random number to rand() ?
- Some way to detect peripherals and pins available.
- Fully specified mbed_app.json? (too big / complicated?)
- Generate mbed_app.json from mbed pindefs?
This is just a proposal, I would appreciate your feedback, input, and suggestions for either improvements, more robust test requirements, or really anything else you can think of. Please submit feedback in the comments below.
I am confused , so hw spec is as I see only about some basic configuration (analog in, dio pins). I could not find A5 to be DAC capable (or PWM in some cases) in the quick search for it thus I asked. If you can pinpoint me to those spec , I'll have a look.
We need to come up with something easy to maintain :-)
reasonable default assumptions- sounds good but what these will be based on, those assumptions?A failing test because a board does not have one PWM pin wired is not something I expect to see (if 2 pins are tested that means PWM peripheral is +/- tested, thus because MCU do not provide one more PWM or bad hw desing, should not result in failure (having this in nightly it would always be red and ignored after some time), we dont do full spec tests (test all pwm peripherals or entire pin configuration) with this testing anyway). I rather see it green with limitations (is testing just one PWM enough? is it two PWM pins?).
Addition: An idea:
Would it make sense to have one test for arduino compability (that one would fail in thecase as not all pins are PWM capable that should be) but PWM test would not fail as it would take input from config file for what pins are available for testing (=PWM is working as it was tested on some pins, but not full set in some cases). Then if we read the test report, it would show this platform does not conform to some assumptions about pinout, but the peripheral driver is green. This would not result in misleading results (red test just because one pin is not wired, etc).
Let me know once done, I'm interested !