Training Data · Wednesday, July 29, 2026
Rohan Anil outlines the limitations of current methods for AI learning at test time, specifically in-context learning and fine-tuning. In-context learning, while not suffering from catastrophic forgetting, is not scalable. Fine-tuning, on the other hand, is prone to catastrophic forgetting and is data-inefficient. Anil suggests that new algorithms are needed to enable models to learn effectively over longer horizons.
“we have in context learning which is very limited and very small amount of data whenever I'm using codex roughly around 20 minutes of usage I need to unpack it and move and move it afterwards which is not that much not that much data”
“the second thing is fine tuning We could try to continuously fine tune our models but then those have the issues of catastrophic forgetting We have issues of very low data efficiency and neither those are very solvable neither of those are very easy to find ways people have trying if there were easy to solve something already solve it”
“So my personal belief is we need to find an algorithm that we can we can let term we can express only the architectural layer that can represent how does learning look like how does learning look like that can work on much longer horizons”