← Back to the learning library
Before you begin: TFA-106 and familiarity with automated builds. Read TFA-105 before changing production signing or installation policy.
Reading time and practice time are estimates. Examples use fictional products and illustrative engineering values, not approved product limits.
Learning objectives
By the end of this module, you should be able to:
- Make firmware builds reproducible and traceable.
- Integrate analysis, tests and hardware rigs into delivery.
- Protect signing and release authority.
- Produce complete release and rollback evidence.
Lesson 1
1. Treat the pipeline as production infrastructure
Pin toolchains, dependencies, generators and configuration. Separate source-controlled inputs from generated outputs, and make build variants explicit. A binary should be explainable from one controlled invocation.
Lesson 2
2. Build layered quality gates
Run compiler warnings, static analysis and host tests quickly. Add target integration and HIL stages with managed devices, fixtures and calibration. Quarantine unreliable rigs rather than normalising intermittent failures.
Lesson 3
3. Protect release authority
Separate ordinary builds from approved releases. Restrict signing keys, require reviewed inputs and retain signing events. Never expose production secrets to untrusted pull-request code or general-purpose runners.
Lesson 4
4. Publish a complete release record
Record source commit, toolchain, dependencies, configuration, test results, anomalies, SBOM, binary hashes, signatures and approval. Define retention, rollback and field-support access.
Lesson 5
5. Build a trustworthy source-to-artefact chain
The delivery pipeline should identify the source, dependencies, toolchain, generator inputs and configuration used for every output. Pin immutable versions or digests where possible and retain the build invocation. A container tag such as latest is not an immutable environment description.
Separate reproducibility from provenance. Reproducibility asks whether controlled inputs produce the same output. Provenance explains where an output came from and which process produced it. A reproducible malicious build is still malicious; a detailed provenance record is useful only if its generation and verification are trustworthy.
For firmware, include linker scripts, memory maps, board definitions, feature options, generated protocol code and post-link processing. Signing, encryption or package construction may create additional outputs whose identities must remain linked to the verified executable image.
Lesson 6
6. Separate untrusted execution from release authority
A pull request may modify scripts executed by the build. Treat that code as untrusted until reviewed, even when the repository is private. General validation jobs should not gain production signing authority, deployment credentials or unrestricted access to manufacturing systems.
Promote the tested artefact into the release process rather than rebuilding an apparently equivalent image after approval. If release processing changes bytes, document exactly which changes occur and how integrity and verification applicability are established. Do not sign an arbitrary pathname without checking its manifest and approved hash.
| Pipeline stage | Trust and evidence boundary |
|---|---|
| Change validation | Build and test without production secrets. |
| Candidate creation | Pin reviewed source and supported configuration. |
| Verification | Associate results with exact artefact hashes and rig state. |
| Approval | Resolve required gates and recorded anomalies. |
| Signing and publication | Restrict authority; verify approved inputs; retain audit record. |
Require controlled approval for exceptions. An emergency fix may use an accelerated workflow, but it still needs an identified binary, risk-based verification, signing control and a documented release decision.
Lesson 7
7. Engineer reliable hardware automation
Maintain a device pool with board revision, bootloader, firmware and calibration identity. Restore a known state between tests and detect failed flashing, stale serial sessions and fixture disconnection. A job must not report the previous run’s log as its own result.
Separate test scheduling from product deadlines. Long HIL queues affect delivery time; they do not justify skipping required evidence. Use fast host gates for early feedback and reserve scarce target capacity for the claims that require it. Track infrastructure availability and intermittent failures with owners.
Exercise disaster recovery: restore the toolchain, build an old supported release, recover archived test results and prove access to approved signing/recovery procedures. Tool licensing, expired certificates and unavailable dependencies can make a nominally supported firmware line impossible to maintain.
Decision guide
What belongs in the release gate?
| Gate | Purpose | Failure response |
|---|---|---|
| Reproducibility | Confirms controlled inputs and toolchain. | Stop and reconcile provenance. |
| Verification status | Confirms required evidence and anomalies. | Block or obtain explicit risk acceptance. |
| Signing approval | Protects production trust. | No unsigned or unauthorised release. |
Worked example
Worked example — multi-variant firmware
Debug and production images are built from the same source but differ in keys, diagnostics and memory layout.
The pipeline uses reviewed variant manifests, isolated production signing and separate artefact identifiers. Tests execute against each releasable configuration and the release record lists every input and hash.
Variants cannot be confused and a shipped image can be reconstructed and assessed.
Engineering evidence
What should the work produce?
Pipeline definition
Controls build, analysis, test and release stages.
Tool and dependency lock
Supports repeatability and impact analysis.
Signing audit trail
Protects release authority and key use.
Release manifest
Identifies inputs, outputs, evidence and approvals.
Common failure patterns
What goes wrong?
- Building production images on an uncontrolled workstation.
- Using mutable tool or dependency tags.
- Letting flaky HIL tests pass on retry without investigation.
- Storing signing keys in repository secrets accessible to every job.
Applied exercise · approximately 25 minutes
Repair a build-and-sign workflow
A CI workflow runs on every pull request, downloads an unpinned compiler, executes repository scripts with a signing credential, and rebuilds the image after test approval.
- Identify the trust and provenance failures.
- Sketch a revised sequence that promotes an exact tested artefact.
- Specify the release manifest and a recovery rehearsal.
Review a suggested approach
Remove signing authority from untrusted change jobs and pin the compiler and dependencies. Create a candidate from reviewed source, run the required tests against its identified outputs, and require approval before a separate restricted signing stage accepts the approved hash and manifest.
Retain source, toolchain, board/variant, unsigned and signed hashes, test identities, anomalies and signing event. If a packaging step changes executable content, it must not inherit test approval without an equivalence or additional verification argument. Rehearse restoration of a historical supported release from retained inputs and demonstrate that its evidence and authorised signing path can still be used.
Completion criteria
- Untrusted scripts cannot access production authority.
- The signed release is traceable to the tested candidate.
- Maintenance is demonstrated through an actual recovery exercise.
This is a self-directed engineering exercise, not an accredited examination. Compare the rationale as well as the final answer.
Practical check
Questions for your next design review
- Are all build inputs versioned?
- Can each variant be identified unambiguously?
- Are rigs and target versions controlled?
- Are signing rights isolated?
- Does the release manifest identify exact binaries and evidence?
A firmware pipeline is trustworthy when it preserves provenance from reviewed source to signed field image.
References and further reading
- SLSA — Supply-chain Levels for Software Artifacts
- TFA-106 — Configuration & Version Control
- SLSA specification — provenance and supply-chain assurance
Educational guidance only. Apply the authorised standards editions, regulations and approved procedures relevant to your product and market. Linked publications support further study; the fictional exercises do not demonstrate compliance.