CTO Toolkit
AI Engineering Scorecard
Four measurement categories for engineering organisations where humans and agents build software together.
How to use this
Review quarterly alongside the Engineering Health Scorecard. DORA remains the spine of Flow and Quality; the last two categories exist because an organisation can improve DORA numbers while spending more to get less.
Flow — where does work wait?
Signal
Score (1–5)
Evidence
Lead time for changes
____
____________________
Cycle time from ready to merged
____
____________________
Review time and review queue depth
____
____________________
Change size distribution
____
____________________
Time spent waiting on decisions rather than implementation
____
____________________
Quality — what does speed cost?
Signal
Score (1–5)
Evidence
Change failure rate
____
____________________
Escaped defects reaching customers
____
____________________
Rework rate on merged changes
____
____________________
Security findings by severity and age
____
____________________
Rollback rate
____
____________________
AI effectiveness — is delegation working?
Signal
Score (1–5)
Evidence
Agent task completion rate without human takeover
____
____________________
Human intervention rate per delegated task
____
____________________
Retry rate before an acceptable result
____
____________________
Acceptance without rework
____
____________________
Defects attributable to agent-generated changes
____
____________________
Verification failure rate on agent output
____
____________________
Economics — did this create value?
Signal
Score (1–5)
Evidence
Cost per accepted change
____
____________________
Inference and tooling spend against delivery outcomes
____
____________________
Engineering hours reclaimed for architecture and difficult problems
____
____________________
Rework avoided
____
____________________
Time to value for the customer
____
____________________
Anti-patterns
- Celebrating volume of generated code as an outcome
- Reporting adoption metrics — seats, prompts, tool usage — as engineering performance
- Measuring individuals rather than the system
- Improving Flow while Quality and Economics quietly degrade
This is an original, generic framework for reuse. It is not proprietary employer material.