DevSecOps Breach Response Plan: Poisoned Dependencies, Compromised Pipelines and Leaked Secrets
A software supply chain incident has three questions: where did the bad code run, what could it reach, and did anything you shipped change? When a dependency is poisoned, find every lockfile, build, container image and developer machine that installed an affected version during the exposure window, and treat each of those environments as compromised. Many recent malicious packages steal tokens and environment variables, so rotate secrets in order of what they let an attacker do: publishing and signing credentials first (package registry tokens, CI tokens, signing keys, deployment credentials), then credentials that reach data, then the rest, from a clean machine. Check your own releases against known-good hashes and build provenance. If a tampered artefact reached customers, publish an advisory with affected versions and the fix.
A software supply chain incident comes down to three questions: where did the bad code run, what could it reach, and did anything you shipped change? Answer them in that order. Most recent malicious packages are built to steal what is lying around in a build: registry tokens, cloud keys, CI secrets and environment variables. So the first hours are about finding every place the code ran and rotating secrets in the right order. This guide is the DevSecOps playbook in the master plan and playbooks structure; the incident response hub has the rest.
What counts as a software supply chain incident?
- A poisoned dependency: a legitimate package publishes a malicious version, often after its maintainer’s account is taken over, or a typosquatted package is installed by mistake.
- A compromised pipeline: a CI/CD system, runner or workflow is changed to run attacker code.
- Leaked pipeline secrets: tokens exposed in logs, artefacts or a public repository.
- A tampered release: something you published does not match its source.
Some recent package worms do all four: they steal a maintainer’s publish token from a build and use it to publish poisoned versions of that maintainer’s other packages. Our coverage of the npm worm campaigns and the LiteLLM compromise shows the pattern.
What do you do first when a dependency is poisoned?
- Declare under the master plan and name an owner for the DevSecOps workstream.
- Pin the facts: affected package and versions, the publish and removal times, and what the payload does, from the advisory or your own analysis.
- Find every install. Search lockfiles across all repositories, CI build logs, container images and registries, and developer machines. A software bill of materials (SBOM) for each build makes this minutes instead of days.
- Treat each environment that ran it as compromised. A CI runner that installed the package exposed every secret available to that job.
- Block the versions in your package proxy or registry and pin known-good versions.
- Rotate secrets in the order below.
The DevSecOps Breach Response Checklist has each step with owners and sample answers. The npm supply chain tracker lists current malicious package campaigns.
In what order should you rotate secrets?
Order by what the secret lets an attacker do, not by how easy it is to rotate:
- Change what you ship: package registry publish tokens, source control tokens and deploy keys, CI/CD tokens, code-signing keys, and credentials that deploy to production. These turn one incident into a supply chain attack on your own customers.
- Reach your data or infrastructure: cloud access keys, database credentials, and API keys to payment or customer systems.
- Everything else the job or machine could read.
Rotate from a clean machine, not the one that ran the package. Confirm each new secret works before revoking the old one, so you do not lock yourself out mid-incident. Check each secret’s usage logs for calls you do not recognise; a token used from an unknown address means the incident has already moved on. The Secret Rotation Order Planner turns your list of exposed secrets into an ordered plan, and the key rotation planner handles the scheduled side afterwards.
How do you know if your own release was tampered with?
- Rebuild and compare. If your builds are reproducible, rebuild the tagged source in a clean environment and compare hashes with what you published.
- Check signatures and provenance. Signed artefacts and build provenance attestations (for example SLSA provenance) show which workflow built what, from which commit.
- Review the registry’s publish history for versions you did not release, and who published them.
- Review workflow changes. Changes to pipeline definitions, build scripts and runner configuration in the exposure window are the usual tampering point.
What if the CI/CD system itself is compromised?
Treat it as identity and infrastructure at once. Review who changed workflows and when; rebuild self-hosted runners from clean images rather than cleaning them, because they persist between jobs; review the trust policies that let pipelines assume cloud roles through short-lived tokens, which an attacker may have widened; and check for new deploy keys, webhooks and integrations. If the pipeline could deploy to the cloud, open the cloud playbook too.
Do you need to tell customers?
If a tampered or malicious artefact you published reached anyone, yes. Publish an advisory with affected versions, what the code does, how to detect it and how to fix it, and request an identifier (a CVE or a GitHub security advisory) so scanners pick it up. Customers will judge you less by the incident than by how quickly and specifically you told them. If personal data was exposed along the way, the data breach playbook’s clocks apply as well.
How do you make the next one smaller?
- Short-lived, narrowly scoped credentials for pipelines instead of long-lived secrets.
- Separate jobs that install dependencies from jobs that hold publish or deploy secrets.
- A package proxy with a cooling-off period for brand-new versions.
- SBOMs stored for every build, so “where did this run?” is a query.
- Branch protection and required review for workflow changes. See the controls that stop GitHub supply chain attacks and the CI secret hardening guide.
- Regular secret scanning of repositories and build logs with the secret scanner.
Checklist: DevSecOps breach response
- Incident declared; affected package, versions and window confirmed.
- Every install found: lockfiles, builds, images, developer machines.
- Affected runners and machines isolated or rebuilt.
- Bad versions blocked; known-good versions pinned.
- Secrets rotated in order: publish and signing, then data, then the rest.
- Usage logs of exposed secrets checked for unknown callers.
- Own releases verified against hashes, signatures and provenance.
- Workflow and runner changes in the window reviewed.
- Advisory published if a tampered artefact reached users.
- Cloud and data breach playbooks opened where reach requires it.
Rehearse the decisions in the Poisoned Dependency Lab.
Frequently asked questions
What should I do first if a dependency I use was compromised? Find out which versions were malicious and when they were published, then search lockfiles, build logs, container images and developer machines for those versions. Every environment that installed one during the window should be treated as compromised: isolate it, find out what the payload did, and rotate the secrets it could reach. Pin or block the bad versions so nothing installs them again.
In what order should secrets be rotated after a CI/CD compromise? Rotate first the secrets that let an attacker change what you ship: package registry publish tokens, source control and CI tokens, code-signing keys and deployment credentials. Then rotate secrets that reach production data, then the rest. Rotate from a clean machine, and revoke the old secret only after the new one is confirmed working.
How do I know whether my own release was tampered with? Rebuild from the tagged source in a clean environment and compare hashes with what you published, if your builds are reproducible. Check signatures and build provenance attestations, review the registry’s publish history for releases you did not make, and review changes to workflow files and build scripts during the exposure window.
Do we have to tell customers about a supply chain incident? If a tampered or malicious artefact you published reached customers or users, tell them: publish an advisory naming affected versions, what the malicious code does, how to detect it and how to fix it, and request an identifier such as a CVE or a GitHub security advisory. If personal data was exposed, the data breach notification rules also apply.
Ready to practise the decisions these articles describe?
Run your first War Room free →One run free with a free account · more runs with Pro