What we said
Intervention rate is useful but incomplete. A company can reduce interventions by narrowing the operating domain, slowing throughput, or moving edge cases to human supervisors, rather than by improving the underlying autonomy. If our hypothesis is right, buyers whose intervention rates have fallen should be able to name those mechanisms in their own deployments.
How we tested it
To test our hypothesis, we looked at four measures:
- What buyers with falling intervention rates say contributed to such decrease.
- How confident they are that the fall in intervention rates reflects real autonomy.
- Whether the range of tasks and conditions the system handles widened or narrowed over the same period.
- Whether buyers accept the general principle that demonstrations are not a reliable indicator of production readiness.
What we found
Of the 65 buyers whose intervention fell, 88% had narrowed what the system attempted, 75% had started routing harder cases to people first and 71% had slowed the system down for reliability. Buyers also accept the general point that intervention rates are useful but incomplete: 69% agree a demonstration does not tell you a system is ready for production. But 89% are confident their own drop reflects real autonomy rather than a smaller job, and nobody in any group chose “not at all confident.”
88%
of the robotics buyers whose human intervention had fallen name narrowing the range of tasks the system attempts among the three things that contributed most to falling intervention rates
The 65 buyers who reported human intervention falling over the past 12 months, out of 85 · Buyers ranked up to three, so the shares add to more than 100%
| Category | In top three (%) |
|---|---|
| We narrowed the range of tasks / conditions the system attempts | 88% |
| We route harder or unusual cases to people before the system tries them | 75% |
| We slowed things down to make the system more reliable | 71% |
| We added more human supervision around the edges | 63% |
| Other | 3% |
The 65 buyers who reported human intervention falling over the past 12 months, out of 85 · Buyers ranked up to three, so the shares add to more than 100%
See the full benchmark89%
of the robotics buyers whose intervention rate had fallen are confident the fall reflects a genuine improvement in the system’s autonomy, rather than a narrower set of tasks or more human handling
The same 65 buyers who reported intervention falling · single-select, so the shares add to 100%
| Category | % of the 65 |
|---|---|
| Very confident | 23% |
| Somewhat confident | 66% |
| Not very confident | 11% |
| Not at all confident | 0% |
The same 65 buyers who reported intervention falling · single-select, so the shares add to 100%
See the full benchmark69%
of robotics buyers say the range of tasks and conditions the system handles has widened over the past 12 months
All qualified buyers, n=85 · single-select, so the shares add to 100%
| Category | % of buyers |
|---|---|
| It handles a materially broader range of tasks / conditions | 12% |
| It handles a slightly broader range | 58% |
| About the same | 24% |
| It has narrowed slightly | 4% |
| It has narrowed materially | 0% |
| Too early to tell | 4% |
All qualified buyers, n=85 · single-select, so the shares add to 100%
See the full benchmark69%
of robotics buyers agree that demonstrations are not a reliable indicator of production readiness
All qualified buyers, n=85 · single-select, so the shares add to 100%
| Strongly agree | Somewhat agree | Neither agree nor disagree | Somewhat disagree | Strongly disagree | |
|---|---|---|---|---|---|
| All buyers | 27% | 42% | 16% | 13% | 1% |
All qualified buyers, n=85 · single-select, so the shares add to 100%
See the full benchmark