The current tests are rather rigid in what they consider passing. Partners are requesting more modular tests that allow for a wider range of hardware to pass.
To address this I am proposing a tiered test infrastructure. This would allow us to certify different levels of compliance while providing maximum flexibility. The idea is to have 4 levels.
- Bare Implementation - First stage of development.
- Constructors / destructors
- Assumptions
- At least 1 of every peripheral possible implimented
- Basic Compliance - things actually start working.
- Basic Read / Write operations for each API implimented
- Single instance of each peripheral works
- Timing of API measured to 10%
- mbed Enabled Compliance - required to be mbed enabled, this level must pass for it to be marked public
- At least one of each peripheral is broken out to Arduino Hardware spec header
- Full API implimented
- Every pin available is tested with single instance
- Multiple instances of each peripheral can be implimented at same time
- peripherals work in interrupt context
- Timing of API measured to 5%
- Corner Case Compliance - Goal level, not everyone will get here, sets goals for the best platforms to achieve
- Edge / Corner case tests
- Timing of API measured to 1%
- Check for corner cases (really beat on the API hard)
- Instantiate everything and test all at once
| x | Bare (L0) All: Constructor, Destructor | Basic (L1) All: | Mbed Enabled (L2) All: | Corner Case (L3): |
|---|---|---|---|---|
| - Single Instance, full API | - single instances on each possible pin, | - Instantiate and deconstruct multiple instances within same program | ||
| - Run tests 1, 10 times each, | - Run tests 100 times, compliance within 1% | - Try to break things | ||
| - compliance within 5% | - Multiple instances at same time | - Initialize in interrupt context | ||
| - Initialize in callback | ||||
| Digital IO | - | Check set timings between read / writes | Check random timings between read / writes | - |
| - Define Time bounds | ||||
| Analog In | - | - Read values that are between 0 and 1 | Read values | x |
| - Make sure values monotonically increase | ||||
| - Read value multiple times at each stage, make sure variance is within 5%,1% each time | ||||
| Analog Out | - | - Increase value monotonically, validate with analog in (verify with return that it doesn’t blow up) | x | x |
| - Verify it can be set to 0->1 in 0.1 incriments | ||||
| I2C | - | - Read / Write 1, 10, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| SPI | - | - Read / Write 1 byte, 10 bytes, 100 bytes using set data | - Read / write 1, 10, 100 bytes using random data to random locations | - Read / write random bytes of random size to random locations quite a few times |
| - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | - Write / Read diff data twice to the same location to verify not reading old data | ||
| PWM | - | - Frequency at 10ms, 30ms, 100ms | - Frequency steps of 1ms from 10 to 200 | Random duty cycles with random frequency's to within 1% |
| - Duty Cycle at 10%,50%,90% | - Duty Cycle every 5% from 5 to 95 | |||
| - Increase monotonically | - Increase monotonically | |||
| Interrupt In | - | - Single object with set timings | - Test on all available pins individually, with random timings (one test for each pin, don’t have more than one object at a time) | - Trigger with PWM's |
| Bus IO | - | Create buses of 1 to 4 pins, read / write | Create busses of 8 - 20 pins, read / write | x |
| Arduino HW | - Validate pins are defined A0-A5, D0-D15 | x | x | x |
| - Validate that expected peripherals are on Arduino Pins per R3 spec (analog In on A0-5...etc) | ||||
| - If pin is NC then don’t use it in later tests (ie analog Out...etc) (not sure how to do this) | ||||
| - Test NC is handled correctly |
Impliment this proposal and release along side the 5.4 mbed-os release in March / April 2017
- Random Number Generation test ( rand() is not actually random between reboots)
- Use python to seed random number to rand() ?
- Some way to detect peripherals and pins available.
- Fully specified mbed_app.json? (too big / complicated?)
- Generate mbed_app.json from mbed pindefs?
This is just a proposal, I would appreciate your feedback, input, and suggestions for either improvements, more robust test requirements, or really anything else you can think of. Please submit feedback in the comments below.
Hence why we break them up into levels, level 1 only tests a single PWM, level 2 tests all the PWM's available per spec, individually, that way if the board doesnt have PWM3 then PWM1,2 pass and PWM3 fails, but only on the level 2 tests, all level 1 tests pass. Ultimately having some way of telling what PWM's the board has would be best, but Im not aware of any HW Descriptor file like that. Something to think about for the future?
yes, that is already the intention
this is not possible in the existing framework as far as I can figure. Do you know how to do something like this in greentea? That is have the outcome of one test suite affect the test parameters for another test suite?
See table above, went ahead and posted it into the GIST (scroll to the top)