Example Scope Leakage Audit — Synthetic Data
A transparent review of quoted and actual tester effort, built from synthetic engagement data for demonstration only.
Executive summary
18 of 27 engagements (66.7%) exceeded the agreed tolerance on the high side. This is a retrospective review, not a statement of lost revenue or a prediction of future results.
Estimate accuracy
| Measure | Result |
|---|---|
| Weighted percentage error | 17.6% |
| Mean absolute percentage error | 19.4% |
| Under-scoped | 18 (66.7%) |
| Over-scoped | 3 (11.1%) |
| Within tolerance | 6 (22.2%) |
Tester-day leakage
Gross positive overruns were 25.5 days. Gross underruns were 2.3 days. 14.4 days was explicitly marked confirmed unbilled across 14 engagement(s).
Financial impact
Additional delivery cost is attributable to positive effort overruns on rows with supplied cost/day. Observed revenue and gross-profit change use only rows with quoted price, final revenue, and cost/day; they include all recorded commercial outcomes and are not proof that a change order caused any result. Confirmed unbilled days and confirmed change-order revenue are shown only where explicitly supplied.
| Fully paired financial subset | Result |
|---|---|
| Coverage | 24 of 27 engagements |
| Quoted gross profit / margin | $270,320 / 77.9% |
| Realized gross profit / margin | $263,312 / 74.2% |
| Observed net gross-profit change | -$7,008 |
| Observed revenue change | $7,500 |
| Confirmed change-order revenue | $7,500 |
Patterns and highest-risk categories
| Dimension | Segment | n | Average error | Net variance | Evidence | How to read it |
|---|---|---|---|---|---|---|
| assessment type | API Pentest | 9 | 20.8% | 11.2 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| assessment type | Web Application Pentest | 11 | 15.5% | 10.0 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| assessment type | External Network Assessment | 7 | 5.9% | 2.0 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| complexity | complex | 11 | 24.7% | 17.6 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| complexity | typical | 11 | 13.5% | 6.5 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| complexity | simple | 5 | -4.2% | -0.9 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| authenticated roles | 4+ roles | 8 | 27.1% | 14.7 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| authenticated roles | 2–3 roles | 9 | 16.3% | 6.9 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| authenticated roles | 0–1 roles | 10 | 3.5% | 1.6 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| multi-tenancy | multi-tenant | 11 | 27.8% | 18.9 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| multi-tenancy | not multi-tenant | 16 | 5.8% | 4.3 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| API endpoint band | 76-150 | 3 | 28.6% | 4.8 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| API endpoint band | 151-300 | 2 | 32.0% | 4.8 days | Insufficient | Too few engagements to infer a recurring pattern. |
| API endpoint band | 26-75 | 3 | 8.2% | 1.2 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
| API endpoint band | 1-25 | 1 | 13.3% | 0.4 days | Insufficient | Too few engagements to infer a recurring pattern. |
| retest included | retest included | 20 | 21.8% | 24.7 days | Descriptive pattern | Descriptive only; contextual review is still needed before changing a scoping practice. |
| retest included | no retest included | 7 | -5.1% | -1.5 days | Small sample | Small sample; inspect individual engagements before changing a scoping practice. |
Patterns are descriptive and may reflect client mix, scope changes, delivery process, or record quality. They do not establish statistical certainty.
Recommended adjustments
- Operational priority 1: before the next quote, review “retest included” (retest included) (n=20; net +24.70 days in this sample). This is a review priority, not a causal coefficient or automatic multiplier.
- Operational priority 2: before the next quote, review “multi-tenant” (multi-tenancy) (n=11; net +18.90 days in this sample). This is a review priority, not a causal coefficient or automatic multiplier.
- These priority groups can overlap: one engagement may appear in more than one group. Do not add their net variances together or treat them as independent effects.
- Make retest reserve an explicit scoping decision and record it separately from testing and reporting days.
- Require a role/authorization review for multi-role applications before approving effort; capture role count and privilege boundaries in the estimate record.
- Use the findings to ask better scope questions and preserve human approval; do not treat this sample as an automatic pricing rule.
Methodology and data limits
Scoping variance is actual tester-days minus quoted tester-days. Percentage error is variance divided by quoted days. Weighted error uses aggregate days; unweighted error averages engagement-level error. Supplied total days are never added to components. Component-only rows need all components or an explicit completeness declaration, preventing double count or missing work being treated as zero.
- This audit describes the supplied engagement sample. It does not establish a market benchmark or a causal explanation for any variance.
- An effort overrun is not automatically unbilled work or lost revenue. Change orders, scope changes, contractual terms, and billing treatment must be verified separately.
- Rows are classified outside tolerance only when variance exceeds both the practical absolute threshold and the percentage threshold. Change the thresholds before comparing reports if the firm uses a different materiality rule.
- Patterns with fewer than three records are displayed only for transparency. Three to nine records are a small sample; ten or more are descriptive only. None establish statistical significance or justify an automatic pricing change.
- ScopeMargin should retain only sanitized estimation metadata. Do not provide client names, targets, vulnerabilities, credentials, secrets, or testing evidence.
Preventing recurrence
ScopeMargin helps firms turn standardized scope into an explainable tester-day range, preserve assumptions and human overrides, and compare approved estimates with delivery actuals. It supports the feedback loop; it does not replace technical judgment or silently set a final price.