A defensible measure is not the most sophisticated number available. It is a number another person can recompute, challenge and use for the decision it was designed to support.
This guide complements the Verification Measure Definition Sheet.
Industry basis and IO addition
NIST and ISO methods already expect organisations to monitor, measure and evaluate performance. IO adds an operating discipline: write the measure before execution, capture the baseline at the accountability event, preserve the same source and population, and retain results that do not support success.
Completing this method does not prove causation or satisfy an external control. It produces a transparent before-and-after finding with stated limitations.
Start with the decision
Bad starting point: “What data do we have?”
Better starting point: “What decision will change if this measure moves?”
Decision: ____________________________________________
Decision owner: ______________________________________
Material change required: _____________________________
If no decision would change, the measure may be informative but should not drive risk movement or maturity progression.
Write the measure as a complete sentence
Among [population], the [numerator] divided by [denominator], observed from [source] over [window], should move [direction] by more than [noise threshold] following [intervention].
Example:
Among active workforce identities in the pilot population, confirmed restricted-data submissions to unsanctioned AI services per 1,000 active users, observed from the IO Browser™ source over 30 complete days, should decrease by more than the established normal variation following the approved browser intervention.
The sentence exposes missing parts before the work begins.
Distinguish four thresholds
| Threshold | Question |
|---|---|
| Data-quality threshold | Is the observation complete enough to use? |
| Noise threshold | Is the movement larger than ordinary variation or measurement error? |
| Decision threshold | Is the movement material enough to change a decision? |
| Risk-band rule | Does an eligible verification move observed residual under the current model? |
These must not be collapsed. A statistically or operationally detectable change may be too small to matter, and a material-looking change may come from an unreliable sample.
Define the denominator before the numerator
Most misleading measures are denominator problems. Record who or what could have generated the event, which exclusions are legitimate, how population changes are handled and whether the intervention itself changes eligibility.
Never allow a lower numerator caused by lost telemetry, removed population or a shorter window to verify an intervention.
Preserve comparability
- Same source class and vendor-specific source.
- Same query or method version, or a documented bridge test.
- Same population definition.
- Equivalent observation windows and reporting delay.
- Stable classification rule.
- Source health and completeness recorded for both observations.
If comparability fails, the result is Inconclusive rather than Unverified.
Look for rival explanations
List concurrent campaigns, policy changes, seasonal effects, incidents, population changes, source changes and external threat shifts that could move the value. Where practical, use a comparison group, interrupted time series or additional observation windows. Where that is not practical, constrain the language rather than pretending the rival explanation does not exist.
Add a guardrail
The target measure can improve while the system becomes worse elsewhere. Examples:
- fewer risky prompts but more use from unmanaged devices;
- fewer MFA fatigue events but more help-desk recovery abuse;
- faster closure but more reopened incidents;
- fewer supplier exceptions but delayed business operations.
Choose at least one measure that would reveal displacement or unacceptable cost for material interventions.
Test the measure before acceptance
- Two people independently compute the same baseline.
- Zero qualifying events is distinguishable from collection failure.
- The source can support the full re-measurement window.
- The eligible population is available at both observations.
- The result cannot be guaranteed merely by completing the activity.
- The decision owner understands what the measure does not establish.
Use precise result language
Verified: the declared measure moved beyond the noise threshold in the intended direction following the intervention.
Unverified: it was validly re-measured and did not move materially.
Regressed: it moved beyond the threshold against the intended direction.
Inconclusive: the comparison cannot support one of those conclusions.
Do not write “the intervention caused” unless the evaluation design supports a causal claim.