Evaluation Discipline Is as Important as the Model

In physical AI, the ability to measure task success, degradation and real-world robustness is often as important as the model itself. If a company cannot measure the failures that occur in real environments, or detect whether changes improve those failures over time, it will struggle to show genuine learning. If that is right, measurement capability should be one of the things separating companies in production from companies stuck in pilots.

We looked at three measures:

• How well buyers say they can actually measure four different things about the systems they are closest to: whether a deployment is succeeding or failing, whether performance is degrading over time, whether a change actually improved real-world performance, and how the system handles edge cases it wasn’t built for.
• Which operating metrics they track consistently today.
• How often evaluation comes up when buyers are asked what the hardest part of making physical AI work is.

Few buyers can reliably tell whether their systems are getting better. Only 20% of buyers say they can reliably measure whether a change they made improved real-world performance. 19% say the same about how their system handles conditions it was not built for. The metrics buyers do track lean toward keeping the plant running: 69% track uptime and 65% track safety incidents, while intervention rate is tracked by 42% and recovery success by 26%.