To make the standard concrete, this post follows Ridgeline, a hypothetical 15-person company that builds a customer web portal, through each stage of building software. Ridgeline isn’t a real company or a client of ours. The standard is the one we recommend. It is DevSecOps in practice: the security gates live in the same pipeline that ships the code, so security keeps pace with development instead of waiting at the end.
At a glance: six stages, each with a gate
Each stage has a gate that has to pass before work moves on, and what is learned in production feeds the next design. The table lists every gate, the first thing to do about it, and where it maps in NIST SSDF, OWASP SAMM v2, the OWASP Top 10 CI/CD Security Risks and the CSSLP exam domains.
| Stage | Gate before work moves on | Do now | NIST SSDF | OWASP SAMM v2 | OWASP CI/CD risk | CSSLP domain |
|---|---|---|---|---|---|---|
| 1. Design | Every feature answers the four threat-modeling questions; each risk becomes a written requirement | Add the four questions to the feature template | PO.1, PW.1 | Threat Assessment; Security Requirements | No direct entry | 3 Requirements; 4 Architecture and Design |
| 2. Build | Fast checks pass on every change; new dependencies confirmed real | Run unit and regression tests on every change | PW.4, PW.5, PW.7 | Secure Build | CICD-SEC-3 Dependency Chain Abuse; CICD-SEC-6 Insufficient Credential Hygiene | 5 Implementation; 8 Supply Chain |
| 3. Review | Someone who didn’t write the change approves it; checks are required | Protect the release branch | PS.1, PW.7 | No direct practice | CICD-SEC-1 Insufficient Flow Control Mechanisms | 2 Lifecycle Management |
| 4. Verify | The full gate passes on the exact code, and every check has proved it can fail | Plant a defect in each test harness | PW.8 | Requirements-driven Testing; Security Testing | No direct entry | 6 Testing |
| 5. Release | Saved, signed and reversible; a failure undoes itself, then fixes itself within limits | Back up before every change and test a restore | PS.2, PS.3 | Secure Deployment | CICD-SEC-9 Improper Artifact Integrity Validation | 7 Deployment, Operations, Maintenance |
| 6. Operate | Watched after release; silence alerts; outsiders can report bugs | Alert when a job processes nothing | RV.1, RV.2, RV.3 | Incident Management; Operational Management | CICD-SEC-10 Insufficient Logging and Visibility | 7 Deployment, Operations, Maintenance |
WHAT
The failure to design against
Ridgeline’s nightly import job stopped importing in week one of a quarter. It still exited cleanly, the dashboard stayed green, and nobody noticed for three weeks, until a customer asked why their numbers hadn’t moved.
That’s the most expensive kind of bug: one that breaks nothing loudly. An empty to-do list and a broken one look the same from the outside. Everything in this standard comes back to two rules.
- A result has to prove itself. No job reports success on a path that did nothing. A check that couldn’t run is reported as incomplete, never as a pass. A missing value is recorded as unknown, never filled with a plausible-looking default.
- A change has to be able to undo itself. If it can’t be put back, it isn’t ready to change.
NOW WHAT
1. Design: decide how it could be abused before building it
Do now: add the four threat-modeling questions to the feature template, and write each real risk down as a requirement.
Skip if: the change is cosmetic and touches no data, login, permission or outside input.
What
When Ridgeline plans a customer file-upload feature, the ticket doesn’t move to “ready” until it answers the four questions from the Threat Modeling Manifesto:
- What are we working on? A one-line data flow: who sends what, where it’s stored, who can read it.
- What can go wrong? Walk the flow with a simple checklist such as STRIDE: spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege.
- What are we going to do about it? Each real risk becomes a written security requirement, for example “uploads are scanned, size-capped and stored outside the web root.”
- Did we do a good enough job? Each requirement becomes a test that the Verify stage runs.
So what
Tests prove the code does what the spec says. They can’t tell anyone the spec forgot how a feature could be misused. A risk named in design costs a sentence; the same risk found in production costs an incident.
Now what
- Plan the test data now. Tests and logs use synthetic data only. Real customer data never goes into a test fixture, a log line or a screenshot.
- Write down what can’t be undone. Any change that can’t be reversed, such as sending an email or deleting data at a vendor, is named in the design and gets its own approval step.
- Start from a published requirement set. OWASP’s Application Security Verification Standard (ASVS 5.0, May 2025) lists verifiable security requirements for web applications and services; pick the ones each feature touches instead of writing them from scratch.
The principle: a risk named in design costs a sentence. Maps to SSDF PO.1 (define security requirements for software development) and PW.1 (design software to meet security requirements and mitigate security risks). OWASP SAMM: Threat Assessment and Security Requirements. CSSLP: Domains 3 (Secure Software Requirements) and 4 (Secure Software Architecture and Design).
2. Build: fast checks on every change, and real packages only
Do now: run unit and regression tests on every change, and confirm every new dependency is a real, established package before adding it.
Skip if: no code is written in-house.
What
Rigor and speed only conflict when every check runs on every keystroke. Ridgeline’s developers make dozens of changes an hour, so they work in two tiers.
- Fast check (every change, seconds, never skipped): all unit and regression tests, a secret scan and a security code scan of the changed files, a dependency check, and the end-to-end tests the change can reach. A change to a path the tool doesn’t recognize runs all of them. This tier is narrowed to what the change can affect, never weakened.
- Full gate (once per release): everything in the Verify stage, on the exact code being shipped.
- Plain reporting: every result says which tier produced it and what it didn’t cover. A quick pass is never reported as “done.”
AI writes the tests; it doesn’t grade them. Ridgeline uses an AI assistant to draft hundreds of tests from the spec in minutes, and people review whether they check the right things. Writing tests is now the cheap part. Keeping them honest is the expensive part, and that is what the Verify stage is for.
Secrets never live in code. Keys and passwords come from a secrets manager or the operating system’s keychain at run time. The secret scan is the backstop, not the plan.
So what
AI-written code adds a new risk: packages that don’t exist. In a study of 576,000 code samples from 16 code-generating models, about 19.7% of recommended packages were hallucinated (made up): on average, at least 5.2% for commercial models and 21.7% for open-source ones, across 205,474 unique made-up names (Spracklen et al., USENIX Security 2025; The New Stack). In repeat tests, 43% of the made-up names came back every time. That repeatability is what makes “slopsquatting” work: an attacker registers the made-up name and waits. OWASP files this family of attacks under CICD-SEC-3, Dependency Chain Abuse.
Now what
At Ridgeline, a new dependency is added only after a person confirms it exists in the official registry, has a real release history and a known publisher, and is the package they meant. A waiting period on new releases doesn’t cover a brand-new package name, so this check stands on its own.
The principle: be fast while building and strict before shipping. Maps to SSDF PW.4 (reuse existing, well-secured software when feasible instead of duplicating functionality), PW.5 (create source code by adhering to secure coding practices) and PW.7 (review and/or analyze human-readable code to identify vulnerabilities and verify compliance with security requirements). OWASP SAMM: Secure Build. CSSLP: Domains 5 (Secure Software Implementation) and 8 (Secure Software Supply Chain).
3. Review: someone who didn’t write it approves it
Do now: protect the release branch so nothing reaches it without a pull request approved by someone other than the author.
Skip if: never. A one-person team can use a separately run AI reviewer, with the person signing off on the risky changes named in design.
What
- Protected release branch. Nobody pushes straight to the branch that ships, including the people who own the repository. Changes arrive through a pull request.
- Independent approval. Every pull request needs approval from someone who didn’t write it: a second person on a rotation, or an AI reviewer run separately from the one that wrote the code.
- Signed commits. Each commit is signed, so the history shows who actually made a change.
- Required, not advisory. The fast checks and the Verify gate are required status checks. A red check blocks the merge; nobody clicks past it.
So what
When one developer and an AI assistant write both the code and the tests, the author is also the approver. That’s the gap an auditor finds first, and it’s cheap to close. OWASP ranks it first among CI/CD risks: CICD-SEC-1, Insufficient Flow Control Mechanisms.
Now what
Turn on branch protection with required reviews and required checks, then try to push straight to the release branch once to prove it’s refused.
The principle: the person who wrote the change shouldn’t be the only one who approved it. Maps to SSDF PS.1 (protect all forms of code from unauthorized access and tampering) and PW.7, and to the change-management criterion (CC8.1) in the AICPA Trust Services Criteria behind SOC 2. CSSLP: Domain 2 (Secure Software Lifecycle Management).
4. Verify: the release gate, and tests that can fail
Do now: plant a deliberately broken “canary” in each test harness, and fail the run when any canary isn’t caught.
Skip if: never. A check that has never caught anything hasn’t been tested.
What
Every check below has to pass on the exact code being released. A check that can’t run is marked INCOMPLETE, and INCOMPLETE blocks the release. A pass writes a stamp tied to a fingerprint of every file; the ship command refuses code without a matching stamp, and any later edit voids it.
| Check | In plain terms | What it catches | NIST SSDF |
|---|---|---|---|
| Unit and regression tests | Test each small piece, plus one test for every bug ever fixed | A rule that quietly changed; an old bug coming back | PW.8, RV.3 |
| Design-requirement tests | One test for each security requirement written in design, drawn from OWASP ASVS where it fits | A risk named in design that the code doesn’t handle | PW.1, PW.8 |
| Integration and end-to-end tests | Test the pieces together, and drive the app the way a user does | Parts that no longer agree; a flow that broke | PW.8 |
| Mutation testing | Make small deliberate changes to the code and confirm the tests fail | Tests that run but don’t actually check anything | PW.8 |
| Static analysis (SAST) | Read the source for security flaws | Injection, unsafe file paths, weak crypto | PW.7 |
| Secret scan | Search the full git history for keys and passwords | A credential committed today or years ago | PS.1, PW.7 |
| Software composition analysis (SCA) with an SBOM | List every package, container and tool shipped or built with, then check each for known flaws | A vulnerable library several layers deep | PW.4, RV.1 |
| Fuzzing | Feed every parser of outside input huge numbers of malformed inputs | Crashes, runaway regular expressions (ReDoS), slowdowns that grow with input | PW.8 |
| Dynamic testing (DAST) | Attack the running app from outside | Login bypass, cross-site request forgery, Host-header and DNS-rebinding tricks, path traversal, missing headers, oversized requests | PW.8 |
| Load and resilience | Push traffic and slow clients at it | Missed response-time targets, errors, memory leaks | PW.8 |
| Real-browser and accessibility checks | Check the interface in an actual browser | Broken layouts; screen-reader and keyboard failures | PW.8 |
| License checks | Confirm everything shipped may be shipped | Incompatible licenses, copied code | PW.4 |
So what
A suite that can’t fail is worse than none, because it gives false confidence. A security scanner that silently checks nothing is the most dangerous tool in the pipeline.
Now what
- Planted defects. If a scanner or test suite misses a canary, the run fails. Mutation testing applies the same idea across the whole codebase, and a mutation score says more than a coverage percentage.
- Behavior, not intent. Every fix ships with a test that runs the code and compares real output. Searching the source for the line just added passes forever and protects nothing.
- Tests come from the spec, ideally written by a reviewer that never saw the implementation and so can’t write a test it already knows will pass.
- Keep it local. The tools run locally with telemetry off, so code and package lists don’t leave Ridgeline’s machines to be scanned.
The principle: a release is a claim, and the gate is the evidence. Trust a check only after watching it catch something. Maps to SSDF PW.8 (test executable code to identify vulnerabilities and verify compliance with security requirements). OWASP SAMM: Requirements-driven Testing and Security Testing. CSSLP: Domain 6 (Secure Software Testing).
5. Release: signed, reversible, and able to fix itself
Do now: back up before every change, test a restore once, and make the deploy undo itself automatically when its checks fail.
Skip if: the app has no database and redeploys in seconds; the undo still applies, the backup step gets lighter.
What
At Ridgeline, the program itself saves what it needs, proves the change, undoes it on failure, and then works on a fix. Nobody has to follow a runbook at 2 a.m. The rule covers code deploys, configuration, database migrations, infrastructure, containers and any settings the program manages.
- Baseline first. Run health checks before touching anything. If the system is already broken, stop with nothing changed; a failure afterward would prove nothing.
- Save before changing. Save everything needed to put it back exactly: files, configuration, a database backup, the previous release, infrastructure state. Check free space first. Fingerprint (hash) the original and the copy, and keep the copy only if they match. Store it where the thing being changed can’t reach it.
- Prove the change. Each change carries its own proof that it did what it claimed, and health must not drop below the baseline.
- Undo automatically on failure. All or nothing, never half-restored. Verify the undo. The previous release keeps serving, so nothing is rushed.
- One-click undo for 30 days. Undo states exactly what it restores and what newer work would be lost, then asks to confirm. Nothing is deleted: expired restore points are archived.
- Keep the previous release runnable side by side (blue/green, versioned folders or feature flags). Every database schema migration is reversible and backed up first.
- Log every change in an append-only, hash-chained log, so a tampered restore point is caught and refused.
- Rehearse on the real target. Make a harmless, marked change, undo it, and prove it’s gone byte for byte before trusting the mechanism.
So what
An undo protects production, but the work still isn’t shipped, and an alert alone hands the job back to a person. The fix loop finishes the job without lowering the bar.
Now what: fix it, within limits
- Sort the failure first. A temporary fault, such as a network blip or a slow service, is retried with a growing wait between tries. A real defect, such as a failing assertion, goes to the pipeline agent to fix.
- The fix is a new change. It starts again at the baseline and goes through the full pipeline, including independent review. Nothing skips a gate because a machine wrote it.
- The agent can’t move the goalposts. It may not edit tests, thresholds, gate settings or the review rule. Otherwise the easiest fix for a failing test is a weaker test.
- Capped tries and a timer. The loop stops after a fixed number of attempts or when its time limit runs out, whichever comes first. Then it escalates to a person with a loud alert and the evidence: what failed, what was tried, and what each try changed.
- Healed is not passed. Every retry and fix is logged and reported. A failure the pipeline fixed on its own is reported as fixed, never as if it never happened.
- Sign what ships. Each release carries signed build provenance, a record of what source it came from and how it was built, generated by a hosted build platform rather than a developer’s laptop. That’s SLSA Build Level 2 (SLSA v1.0). The SBOM ships with the release. Together they close OWASP CICD-SEC-9, Improper Artifact Integrity Validation.
The principle: if it can’t be put back, it isn’t ready to change. Maps to SSDF PS.2 (provide a mechanism for verifying software release integrity) and PS.3 (archive and protect each software release), and to the change-management (CC8.1) and recovery (A1.2, A1.3) criteria behind SOC 2. OWASP SAMM: Secure Deployment. CSSLP: Domain 7 (Secure Software Deployment, Operations, Maintenance).
6. Operate: watch it, respond to it, and let outsiders report it
Do now: alert when a scheduled job runs and processes nothing, and publish a security.txt with a working contact.
Skip if: nothing is reachable from the internet and nothing runs on a schedule.
What
- A watch window after every release. For a set period after each deploy, Ridgeline compares error rates, response times and key business counts (such as records imported per night) against the baseline. A drop past a set threshold triggers the same undo and fix loop as a failed deploy.
- Alerts on silence, not just on errors. A nightly job that ran and processed nothing raises an alert.
- Security logging that someone reads. Logins, permission changes, admin actions and failed checks go to a central log with a retention period, and the alerts reach a person who knows what to do with them.
- An incident playbook, rehearsed. A short written plan says who decides, who communicates, how to contain, and how to preserve evidence. The team runs through it as a tabletop exercise at least once a year.
- A front door for bug reports. Ridgeline publishes a
security.txtfile at/.well-known/security.txtwith a contact and an expiry date (the two required fields in RFC 9116), plus a short disclosure policy that tells researchers how to report safely. - A periodic outside review. Once a year, or before a major launch, someone who didn’t build the system runs a supervised break-in test. It finds what the team’s own tests are blind to.
So what
A clean release is the start of the job, not the end. Some failures only show up under real traffic, days later. The silence rule alone would have caught Ridgeline’s three-week import failure on its first night.
Now what
Every incident and every reported bug ends with a root cause, a fix and a regression test, so it can’t quietly come back. What was learned goes back into the next design.
The principle: watch for the job that did nothing as closely as the job that crashed. Maps to SSDF RV.1 (identify and confirm vulnerabilities on an ongoing basis), RV.2 (assess, prioritize and remediate vulnerabilities) and RV.3 (analyze vulnerabilities to identify their root causes), and to OWASP CICD-SEC-10, Insufficient Logging and Visibility. OWASP SAMM: Incident Management and Operational Management. CSSLP: Domain 7.
Underneath every stage: self-healing
A fault should repair itself on the next run, with no one having to remember it happened. If the fix needs a person to run a cleanup command, it isn’t fixed.
| What goes wrong | What the system does on its own |
|---|---|
| A change is cut off by a crash, a sleeping laptop, a deploy or a timeout | The next run finds the half-finished change and either completes it or undoes it. |
| A cache, saved session or staged file gets corrupted | State is checked when it’s read, not only when it’s written, so bad entries from before a check existed are dropped and rebuilt. |
| The network is down, a file moved, a key is missing | The job reports what happened in plain words and never produces a wrong answer. A failed download retries on the next run without raising a false alarm. |
| Someone else edits the file mid-change | Edits target content (keys, names, patterns), not line numbers, and the version is checked before writing. |
| A restore point is altered outside the program | The hash-chained log catches it, and the restore point is refused. |
| A job hangs or loops | Every job, scheduled or not, has a timeout. Long jobs can be resumed, are safe to re-run, and keep their partial progress. |
| A scheduled job keeps failing | It backs off on its own. A person who names the job can override the backoff; an unattended job never can. |
| A release fails its checks | It undoes itself, then the pipeline agent tries a fix as a new change through the full pipeline. After a capped number of tries or its time limit, a person gets the alert and the evidence. |
Underneath every stage: keep every part current
Every piece Ridgeline runs updates on a schedule: dependencies, runtimes, base images, tools and operating-system packages. Each update goes through the same save, prove and undo cycle as a code change.
- Verified releases only. Install a release only after it matches the publisher’s checksum or signature. Anything that can’t be verified is reported, never installed.
- A 72-hour wait. New releases sit for three days, which gives the ecosystem time to catch a hijacked or broken one. Security fixes skip the wait.
- Stay on the supported line. Follow the current major or long-term-support version. A jump to a new major version is a deliberate human decision.
- Side by side, with a way back. The new version installs next to the old one; switch over, run its checks, and switch straight back on failure.
- Refresh the inventory. After every update, rebuild the SBOM and rerun the vulnerability check. A published security fix that isn’t installed yet counts as an open finding.
- Nothing is skipped silently. A component that needs an admin password, or has no verifiable release, is reported every cycle until it’s handled.
Maps to SSDF PW.4 and RV.1, OWASP SAMM Environment Management, and OWASP CICD-SEC-3 and CICD-SEC-8 (Ungoverned Usage of 3rd Party Services).
Copy this standard
Nobody needs all of it on day one. A few fundamentals, done right, do most of the work. Start with what stops silent failures and unreviewed changes, then add depth.
| When | What to put in place | Skip if |
|---|---|---|
| Now (first 30 days) | Fail loudly: no job reports success on a path that did nothing, and every job has a timeout. Alert when a job processes nothing. | Never. |
| Now | Back up before every change, and test a restore once. A backup that has never been restored is a hope, not a backup. | Never. |
| Now | Protect the release branch and require approval from someone other than the author. | Never. |
| Now | Scan the full git history for secrets, rotate anything found, and move secrets into a secrets manager. | Never. |
| Now | Run unit and regression tests on every change; every bug fix adds a regression test. | No code is written in-house. |
| Fast follow (30–90 days) | Plant a defect in each test harness and fail the run if it isn’t caught. | Never. |
| Fast follow | Keep an SBOM, check it for known flaws on every change, and confirm every new dependency is real before adding it. | There are no third-party dependencies (rare). |
| Fast follow | Publish security.txt and a short disclosure policy; write a one-page incident playbook. | Nothing is reachable from the internet. |
| Fast follow | Make deploys reversible: keep the previous release runnable and every migration reversible. | The app has no database and redeploys in seconds. |
| Next (next quarter) | Threat model each new feature with the four questions, and turn each risk into a test. | Never. |
| Next | Split checks into a fast tier and a stamped release gate; add a post-release watch window and the capped fix loop. | The whole suite already runs in under a minute. |
| Next | Fuzz every parser of outside input, and run dynamic tests against every network-facing surface. | Nothing built takes outside input. |
| Later (when budget or size allows) | Signed build provenance (SLSA Build Level 2), mutation testing, load testing and a yearly outside review. | The software isn’t distributed to others or exposed to the internet. |
How this is put together
Ridgeline is hypothetical, used to make each control concrete; any resemblance to a real organization is coincidental. The hallucinated-package figures are measured results from the cited study. Every NIST SSDF practice ID was checked against the practice titles in the full SP 800-218 text. The SOC 2 criteria references were checked against a secondary summary of the AICPA criteria, labeled as such below. Tool and vendor choices are left out on purpose: the gates matter more than the brands that run them. The OWASP mappings use SAMM v2, ASVS 5.0 and the Top 10 CI/CD Security Risks; the CSSLP domains follow the ISC2 exam outline effective 15 September 2023. Where a stage has no direct entry in a framework, the table says so rather than forcing a fit. For the supply-chain side of DevSecOps pipelines, NIST SP 800-204D (February 2024) is the deeper reference. Mapping a control to a framework shows where it fits; it is not a compliance claim.
Bottom line
Make failure loud, make every change undo itself, and make sure nobody approves their own work, including an AI assistant and the pipeline agent that fixes a failed release. Start with the four fundamentals this month; add the rest a stage at a time.
Sources
- NIST, SP 800-218, Secure Software Development Framework (SSDF) Version 1.1, final, February 2022. csrc.nist.gov
- NIST, SP 800-218 full text (PDF), practice titles PO.1 through RV.3. nvlpubs.nist.gov
- NIST, SSDF project page: practice groups and the SP 800-218A profile for generative-AI development. csrc.nist.gov
- Threat Modeling Manifesto: the four key questions. www.threatmodelingmanifesto.org
- SLSA v1.0, build levels. slsa.dev
- IETF, RFC 9116, A File Format to Aid in Security Vulnerability Disclosure (security.txt), April 2022. www.rfc-editor.org
- Spracklen et al., “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs,” USENIX Security 2025. arxiv.org
- The New Stack, “Slopsquatting: the newest threat to your AI-generated code” (repeat-run and per-model figures). thenewstack.io
- Open Security Architecture, SOC 2 Trust Services Criteria (secondary summary of the AICPA criteria CC8.1, A1.2, A1.3). opensecurityarchitecture.org
- OWASP, Top 10 CI/CD Security Risks (CICD-SEC-1 through CICD-SEC-10). owasp.org
- OWASP, Software Assurance Maturity Model (SAMM) v2: business functions and security practices. owaspsamm.org
- OWASP, Application Security Verification Standard (ASVS) 5.0.0, 30 May 2025. asvs.owasp.org
- ISC2, CSSLP Certification Exam Outline, effective 15 September 2023: eight domains. www.isc2.org
- NIST, SP 800-204D, Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines, February 2024. csrc.nist.gov
Provenance: written by Fezz with Claude (Anthropic) on 23 October 2026; figures and framework references re-verified against the linked sources that day. Threat Vector Management is not affiliated with any company operating under the name “ThreatVector.”