← Back to the learning library
Before you begin: Read TFA-109 on unit, integration and HIL testing and TFA-303 on requirements. Bring a test result whose meaning depends on its environment.
Reading time and practice time are estimates. Examples use fictional products and illustrative engineering values, not approved product limits.
Publication basis: ISO/IEC/IEEE 29119-2:2021 and IEEE 1012-2024. Other parts of the 29119 series may support documentation and test techniques; assess the relevant editions before making a formal conformance claim.
Learning objectives
By the end of this module, you should be able to:
- Distinguish testing from the wider verification and validation effort.
- Allocate claims to suitable evidence and environments.
- Define reproducible test conditions and acceptance criteria.
- Assess anomalies, independence and release evidence proportionately.
Lesson 1
1. Understand the two frameworks
ISO/IEC/IEEE 29119-2 describes processes for governing, managing and implementing software testing across lifecycle models. IEEE 1012 addresses verification and validation for systems, software and hardware, including firmware and interfaces. Its scope includes activities beyond execution tests, such as reviews, analysis and assessment, with lifecycle requirements related to integrity levels.
Use these publications to structure an assurance approach appropriate to the product and governing framework. They do not assign a safety classification to a controller merely because its tests are automated. For a fictional actuator controller, define what must be shown about behaviour, timing and intended operation before deciding how many unit tests to write.
Lesson 2
2. Write claims before selecting environments
Separate claims that can be established in isolation from those that depend on physical integration. A host test can exercise state transition logic and boundary conditions quickly. It cannot automatically establish interrupt latency, memory-mapped peripheral behaviour or the delay between a command and actuator motion.
Create a claim-to-evidence plan. For each claim, identify the relevant requirement, method, environment, acceptance criterion and limitations. Include non-test evidence where it is stronger or necessary: a review may expose an inconsistent interface contract, while analysis can establish a bound across conditions that are impractical to sample exhaustively. A simulator is useful when its assumptions and differences from the target are understood.
Lesson 3
3. Plan tests that could reveal a wrong implementation
Choose cases from the requirement and risk, not only from code branches already written. Examine boundaries, invalid inputs, state transitions, sequencing, resets and resource pressure. A test should fail for a plausible incorrect implementation. Repeating nominal operation a thousand times may miss a defect that occurs only when a timeout and a completion interrupt coincide.
For the actuator controller, test the stop request while motion is starting, while a sensor is stale and while the command queue is full. State the intended result before execution. Avoid modifying expected results merely to match observed behaviour. If the requirement is ambiguous, resolve it through the requirements process rather than letting the test author silently become the product decision-maker.
Lesson 4
4. Make measurements and fixtures credible
Record stimulus, observation point, sampling method and timing reference. Suppose an illustrative response limit is 100 ms and a measured result is 99 ms with a bounded observation uncertainty of ±3 ms. That measurement alone cannot establish the true response is at or below 100 ms. Decide in advance how uncertainty is handled and improve the method where necessary.
A fixture can introduce latency, quantisation, electrical loading or stale state. Verify the aspects that affect the claim. Record fixture hardware and software versions alongside the firmware build. If a simulator reports a sensor failure that real hardware cannot produce in the same way, describe the difference and plan additional integration evidence rather than quietly treating the two situations as equivalent.
Lesson 5
5. Make traceability bidirectional and meaningful
A test-to-requirement link should explain what part of the requirement it assesses. A single test may cover nominal timing but not restart behaviour. Conversely, one system claim may depend on a requirements review, an architectural argument and several target tests. Describe this combination explicitly.
| Claim | Primary evidence | Residual question |
|---|---|---|
| Queue rejects excess commands | Boundary and sequence tests | Does the field configuration use the same capacity? |
| Physical stop is timely | Target measurement at the actuator endpoint | Is uncertainty and worst-case load covered? |
| Recovery preserves valid settings | Interrupted-write and migration tests | Are all supported data versions represented? |
Review uncovered claims, obsolete links and tests whose acceptance criteria changed. Traceability is an assessment aid, not a substitute for reading the evidence.
Lesson 6
6. Handle independence and anomalies deliberately
Determine required independence using the adopted framework, assigned integrity and project obligations. Fresh eyes can reveal assumptions shared by the implementer and test author, but an external laboratory is not automatically independent in every relevant sense. Record who specifies, executes, reviews and approves each important activity.
When a test fails intermittently, preserve the failure and investigate its cause. A later pass does not erase it. Distinguish product defects, test defects, environment failures and unresolved results. Each category needs a disposition and an assessment of affected evidence. If fixture software was wrong, identify earlier results that used it; fixing the fixture does not retroactively validate those runs.
Lesson 7
7. Assemble an evidence-based release conclusion
The release summary should state the assessed configuration, claims, methods, results, deviations, limitations and open anomalies. Explain why the evidence supports the intended use at the relevant level. A firmware verification conclusion alone should not be relabelled as complete system validation when usability, physical operation or operational context has not been assessed.
Keep the evidence stable and retrievable. Record exact binaries and the environments used rather than pointing only to the most recent pipeline run. Where some supported hardware combinations are represented by equivalence reasoning, preserve that argument. The release decision should remain understandable after tools, staff and dashboard contents have changed.
Decision guide
Which evidence best supports this claim?
| Claim type | Useful method | Important limit |
|---|---|---|
| Pure calculation or state logic | Host unit tests and review | Target representation can differ |
| Peripheral behaviour or scheduling | Target integration tests and analysis | Observation must not distort timing |
| Correctness of a safety argument | Structured review and corroborating evidence | Passing execution tests does not replace the argument |
| Intended system use | Validation in a justified operational context | Firmware-only evidence covers only its contribution |
Worked example
Worked example — a passing HIL report with an unresolved stop delay
A HIL report says every stop-response test passed. The limit is 100 ms, the worst recorded result is 99 ms, and the fixture polls at 10 ms intervals. The test observes a software status flag rather than physical de-energisation.
The team reviews the claimed endpoint and measurement uncertainty. It separates the firmware status change from output-stage response, selects appropriate physical instrumentation, establishes a justified uncertainty treatment and repeats the relevant worst-case sequences. The original report is retained with its limitations rather than deleted.
The revised evidence supports a narrower and better-defined claim. If the total response still cannot be demonstrated within the limit, the issue remains open for design or requirement allocation review; it is not resolved by rounding the measurement.
Engineering evidence
What should the work produce?
Assurance and test plan
Allocates claims to reviews, analysis and execution environments.
Controlled specifications and results
Record stimuli, expected outcomes, configurations and observations.
Fixture assessment
Explains measurement credibility, limitations and version control.
Anomaly and release summary
Retains failures, impact assessments and the final evidence-based conclusion.
Common failure patterns
What goes wrong?
- Counting tests without examining the claims they support.
- Using a status flag as an unqualified proxy for physical behaviour.
- Erasing an intermittent failure after a successful rerun.
- Assuming host coverage proves target timing.
Applied exercise · approximately 25 minutes
Assess an intermittent regression result
A new communication driver passes all host tests. One target run misses an acquisition deadline under heavy traffic; the next three runs pass. The team wants to mark the failure as a fixture glitch.
- Identify evidence needed to distinguish a fixture problem from a firmware defect.
- Specify a focused investigation that preserves the failed configuration.
- Draft a release disposition while the cause remains unknown.
Review a suggested approach
Retain the failing binary, build options, fixture version, traffic pattern, timing records and target conditions. Check whether the observation method can miss or delay events, and whether interrupt activity, queue growth or shared resources could delay acquisition. Instrument both sides of the relevant boundary where possible without materially changing it.
Repeat a defined stress sequence and compare diagnostic evidence, rather than counting unstructured successful reruns. Until the cause and impact are understood, classify the result as unresolved and apply the project’s release criteria. Any decision to release needs an explicit risk and evidence assessment by the appropriate authority. Calling it a fixture problem requires supporting evidence.
Completion criteria
- The original failure remains part of the record.
- The investigation examines both product and measurement hypotheses.
- The release conclusion accurately states unresolved evidence.
This is a self-directed engineering exercise, not an accredited examination. Compare the rationale as well as the final answer.
Practical check
Questions for your next design review
- Are acceptance criteria defined before execution?
- Do observation points and environments match the claims?
- Are uncertainty, fixture limitations and configuration identity recorded?
- Are all failures and incomplete claims explicitly dispositioned?
Verification confidence comes from credible evidence for defined claims, not from the size or colour of a test dashboard.
References and further reading
- ISO/IEC/IEEE 29119-2:2021 — test processes
- IEEE 1012-2024 — system, software and hardware V&V
- TFA-109 — Unit, Integration & HIL Testing
- TFA-204 — Verification & Test Engineers
Educational guidance only. Apply the authorised standards editions, regulations and approved procedures relevant to your product and market. Linked publications support further study; the fictional exercises do not demonstrate compliance.