Prof G Markets · Tuesday, September 8, 2026
Gary Marcus pointed out that current AI models, while excelling on benchmarks, often falter in real-world applications. He noted that the hype surrounding new models frequently leads to disappointment when users find they don't perform as expected in practical scenarios.
“What we have now is an era where, you know, some people call it benchmark maximizing or benchmark messing. Where these systems are often, it would appear, trained on lots of benchmarks. They do really well on the benchmarks, and then they don't typically do as well in the real world.”
“And what the world's experience has been with all prior models is they don't live up to expectations.”