The current tests are rather rigid in what they consider passing. Partners are requesting more modular tests that allow for a wider range of hardware to pass.
To address this I am proposing a tiered test infrastructure. This would allow us to certify different levels of compliance while providing maximum flexibility. The idea is to have 4 levels.
- Bare Implementation - First stage of development.
- Constructors / destructors
- Assumptions
- At least 1 of every peripheral possible implimented
- Basic Compliance - things actually start working.
- Basic Read / Write operations for each API implimented
- Single instance of each peripheral works
- Timing of API measured to 10%
- mbed Enabled Compliance - required to be mbed enabled, this level must pass for it to be marked public
- At least one of each peripheral is broken out to Arduino Hardware spec header
- Full API implimented
- Every pin available is tested with single instance
- Multiple instances of each peripheral can be implimented at same time
- peripherals work in interrupt context
- Timing of API measured to 5%
- Corner Case Compliance - Goal level, not everyone will get here, sets goals for the best platforms to achieve
- Edge / Corner case tests
- Timing of API measured to 1%
- Check for corner cases (really beat on the API hard)
- Instantiate everything and test all at once
| x | Bare (L0) All: Constructor, Destructor | Basic (L1) All: | Mbed Enabled (L2) All: | Corner Case (L3): |
|---|---|---|---|---|
| - Single Instance, full API | - single instances on each possible pin, | - Instantiate and deconstruct multiple instances within same program | ||
| - Run tests 1, 10 times each, | - Run tests 100 times, compliance within 1% | - Try to break things | ||
| - compliance within 5% | - Multiple instances at same time | - Initialize in interrupt context | ||
| - Initialize in callback | ||||
| Digital IO | - | Check set timings between read / writes | Check random timings between read / writes | - |
| - Define Time bounds | ||||
| Analog In | - | - Read values that are between 0 and 1 | Read values | x |
| - Make sure values monotonically increase | ||||
| - Read value multiple times at each stage, make sure variance is within 5%,1% each time | ||||
| Analog Out | - | - Increase value monotonically, validate with analog in (verify with return that it doesn’t blow up) | x | x |
| - Verify it can be set to 0->1 in 0.1 incriments | ||||
| I2C | - | - Read / Write 1, 10, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| SPI | - | - Read / Write 1 byte, 10 bytes, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| PWM | - | - Frequency at 10ms, 30ms, 100ms | - Frequency steps of 1ms from 10 to 200 | Random duty cycles with random frequency's to within 1% |
| - Duty Cycle at 10%,50%,90% | - Duty Cycle every 5% from 5 to 95 | |||
| - Increase monotonically | - Increase monotonically | |||
| Interrupt In | - | - Single object with set timings | - Test on all available pins individually, with random timings (one test for each pin, don’t have more than one object at a time) | - Trigger with PWM's |
| Bus IO | - | Create buses of 1 to 4 pins, read / write | Create busses of 8 - 20 pins, read / write | x |
| Arduino HW | - Validate pins are defined A0-A5, D0-D15 | x | x | x |
| - Validate that expected peripherals are on Arduino Pins per R3 spec (analog In on A0-5...etc) | ||||
| - If pin is NC then don’t use it in later tests (ie analog Out...etc) (not sure how to do this) | ||||
| - Test NC is handled correctly |
Impliment this proposal and release along side the 5.4 mbed-os release in March / April 2017
- Random Number Generation test ( rand() is not actually random between reboots)
- Use python to seed random number to rand() ?
- Some way to detect peripherals and pins available.
- Fully specified mbed_app.json? (too big / complicated?)
- Generate mbed_app.json from mbed pindefs?
This is just a proposal, I would appreciate your feedback, input, and suggestions for either improvements, more robust test requirements, or really anything else you can think of. Please submit feedback in the comments below.
Based on Feedback I have updated the Tests matrix as follows:
Ultimately I would like a method of dynamically figuring out what pins are available on a platform (ie knowing that there are 3 PWM's on a platform, two of which are broken out to the correct Arduino Headers and should be tested, the others should not be tested). However this is currently not possible. As a workaround untill a solution is found (maybe a hardware description file or something?... tbd, beyond the scope of this) having Level 1 test a single instance while level 2 tests all instances individually will provide a basic way of testing if the underlying software works.
Using this method the above platform would pass PWM 0 / 1 and Fail the other PWM tests because the peripherals would not be on the assumed headers. The Partner Manager would then make a note of this and approve it as an exception, or possibly turn the tests off on PWM 2-4 for that platform.
Does this sound like an acceptable workaround?