Chapter 12 of 14 · Months 2–3
Measure time to fix, not findings count
- How do I show I am making a difference?
- Is the number of findings a good metric?
- What should I report to my manager?
Around month two, someone will ask how AppSec is going. The tempting answer is a big number: findings raised, scans run, tickets opened. Those numbers go up when you turn on a noisy tool and down when you tune it, and neither change says anything about risk.
Measures that mean something
| Measure | What it tells you | Watch out for |
|---|---|---|
| Time to fix, by severity | How long real risk stays open | Closing tickets as will not fix to look faster |
| Open serious findings on internet-facing services | Exposure right now | Arguing about severity instead of fixing |
| Share of services with a named owner | Whether anyone can act | Owners who left the company |
| Share of critical services with checks in the pipeline | Coverage | Checks that run but block nothing and alert nobody |
| Features threat modelled before launch | How early you are involved | Models done after launch to tick a box |
| Repeat findings of the same type | Whether fixes teach anything | Small samples that swing a lot |
Ask the developers too
Some of the most useful signals are not in any tool. Do teams invite you to design reviews? Do engineers report suspicious things to you? Do they argue less about findings than they did in month one? Ask a few of them, every month or two, what is working and what is in their way.
The question you might not ask out loud
My numbers look worse than last month. Should I worry?
Not necessarily. Adding a service to the inventory or turning on a new check often makes numbers look worse, because you are now seeing what was always there. Say that plainly in your report. Trends over a few months matter more than one month.
Reporting upward
- Lead with a sentence on exposure: what is most at risk right now, and what you are doing about it.
- Show two or three measures as trends, not snapshots.
- Name one thing you need: a decision, an owner or time from a team.
- Keep it to one page. Detail goes in an appendix nobody has to read.
Rule of thumb:Every measure can be gamed. Pick a few, explain how each could mislead and look at them together rather than one at a time.
Agree the definitions with the teams before you publish anything. When does the clock start: when a scanner flags something, or when a person confirms it? What counts as fixed: merged, or running in production? Settling these early stops arguments later and makes the numbers something engineers recognise rather than something done to them.