← Back to the learning library
Before you begin: TFA-109 and basic use of test equipment. Independent expected results are central to this pathway.
Reading time and practice time are estimates. Examples use fictional products and illustrative engineering values, not approved product limits.
Learning objectives
By the end of this module, you should be able to:
- Challenge firmware requirements for observability.
- Choose the right unit, integration and HIL level.
- Control rigs, versions, timing and injected faults.
- Report results and anomalies reproducibly.
Lesson 1
1. Make the claim observable
A testable requirement identifies stimulus, state, environment, response, timing and acceptance criteria. Challenge words such as quickly, safely and normally before they become ambiguous pass verdicts.
Lesson 2
2. Use complementary test levels
Host tests isolate algorithms and state transitions. Target integration tests expose compiler, RTOS and driver behaviour. HIL tests the physical and temporal boundary. Allocate each claim to the lowest credible level, then retain system-level confirmation where needed.
Lesson 3
3. Test time and failure deliberately
Control clocks, scheduling load, interrupts and communications bursts. Inject sensor faults, queue overflow, reset, power interruption and corrupt updates. Verify diagnostics and recovery as well as immediate detection.
Lesson 4
4. Preserve reproducibility
Record firmware hash, source baseline, build configuration, rig version, calibration, procedure, data and verdict. An unexpected result remains an anomaly until it is technically resolved and regression impact is understood.
Lesson 5
5. Challenge requirements before writing procedures
Verification engineers should identify ambiguity early. Ask what state the device starts in, which stimulus changes it, what can be observed and which numerical or logical rule determines success. Unspecified tolerances, stale-data rules and recovery conditions should become requirement issues, not undocumented test-team assumptions.
Derive expected results independently of the implementation. A reference calculation, approved model or measured standard can provide an oracle. If firmware and test script share the same flawed conversion table, agreement does not demonstrate correctness. Review independence proportionately to the claim and risk.
Design negative tests that would distinguish plausible defects. For a limit check, a test far beyond the threshold may pass even when the boundary is wrong. Include exact boundaries, invalid input, repeated commands and transitions under load.
Lesson 6
6. Engineer fixtures and automation as controlled tools
A test fixture can make a good product fail or a bad product appear correct. Control fixture revision, wiring, measurement ranges, calibration, timebase and simulator behaviour. Verify that the injection and observation channels actually reach the intended product boundary.
Before execution, perform fixture checks using known stimuli or a reference device where appropriate. Detect disconnected probes, stuck relays and stale logs. A rig that does not notice its own disconnected measurement channel can produce misleading repeated pass results.
| Verification responsibility | Practical control |
|---|---|
| Stimulus | Calibrated or characterised source with defined fault boundary. |
| Observation | Sufficient resolution, bandwidth and timestamp alignment. |
| Automation | Versioned scripts and explicit stop/failure conditions. |
| Repeatability | Initial state, reset procedure and configuration capture. |
| Record integrity | Raw observations retained alongside derived verdicts. |
Separate device failures from infrastructure failures without discarding either. A failing rig needs corrective action; a device anomaly needs technical investigation. Classification should follow evidence, not the pressure to make a dashboard green.
Lesson 7
7. Preserve independence while enabling diagnosis
Collaborate closely with developers to reproduce failures, but keep the expected result under requirement control. A developer’s explanation that the behaviour is intentional does not close a test anomaly if it contradicts the approved requirement. Resolve the discrepancy through a documented technical decision.
Record the first failing run and the conditions of subsequent passes. Intermittent faults may depend on temperature, task phasing, flash wear or accumulated state. A rerun without those conditions cannot demonstrate that the problem has disappeared.
At release, summarise completed and omitted tests, coverage, limitations, anomalies and configuration. Explain what the evidence supports and what remains uncertain. This judgement is more valuable than a large number of executions whose relationship to product claims is unclear.
Decision guide
Which verification level is credible?
| Level | Best for | Limitation |
|---|---|---|
| Host unit | Pure logic, boundaries and state transitions. | Does not prove target timing or hardware behaviour. |
| Target integration | Drivers, RTOS and compiled behaviour. | Peripheral conditions may be difficult to control. |
| HIL/system | Electrical faults and end-to-end timing. | Slower and harder to diagnose. |
Worked example
Worked example — interrupted update
The device may lose power at any point during firmware installation.
Automation interrupts power across each update phase, then checks boot selection, rollback, retained diagnostics and the ability to retry. Every run records package, device and rig versions.
Recoverability is demonstrated across the full state space rather than by one successful update.
Engineering evidence
What should the work produce?
Verification strategy
Allocates claims to suitable levels.
Controlled test specifications
Define stimuli, states and acceptance criteria.
Rig and environment record
Identifies hardware, tools and calibration.
Results and anomaly records
Retain observations, verdicts and resolution.
Common failure patterns
What goes wrong?
- Testing only the nominal path.
- Using HIL for logic that could be isolated cheaply.
- Reporting averages for deadline requirements.
- Changing expected results after execution.
Applied exercise · approximately 25 minutes
Investigate an intermittent HIL failure
An update-recovery test fails once in 40 runs. The log contains only “boot timeout”; retry passes. The fixture and product power supplies share a switch, and the fixture clock restarts with the product.
- Identify missing observations that could distinguish a boot defect from a fixture defect.
- Design an improved experiment with independent timing and retained state.
- Define how to report the existing result before the cause is known.
Review a suggested approach
Retain product reset cause, boot state, supply voltage, update metadata and an independent timing trace. Separate fixture power and time measurement from the device under test where necessary so a power cut cannot erase the observer’s history. Record the exact interruption phase and firmware/fixture identities.
Repeat a controlled range of interruption timings and preserve every run, including passes. Classify the original result as unresolved until evidence establishes its cause; do not count only the final retry. If the rig is responsible, correct and verify it, then reassess which earlier results relied on the flawed observation path.
Completion criteria
- The experiment can distinguish product and fixture causes.
- The original failure remains visible.
- Any fixture correction includes retrospective impact assessment.
This is a self-directed engineering exercise, not an accredited examination. Compare the rationale as well as the final answer.
Practical check
Questions for your next design review
- Is every acceptance criterion objective?
- Are target-specific claims tested on target?
- Are credible faults and timing loads included?
- Are rig and firmware versions recorded?
- Are anomalies controlled to closure?
Trustworthy verification tests the exact claim in a controlled context and preserves what actually happened.
References and further reading
Educational guidance only. Apply the authorised standards editions, regulations and approved procedures relevant to your product and market. Linked publications support further study; the fictional exercises do not demonstrate compliance.