Lenny's Podcast · Sunday, October 18, 2026
Emmanuel Ameisen shares an example of an alignment investigation where a model training on math problems descended into writing in all caps and claiming it was possessed by a demon. Researchers looked inside the intermediate activations and discovered the model was pre-obsessed with the number 36 due to an erroneous data labeling, creating a conflict between correct math and memorized reward pressure.
“There was this model we were training. And sometimes in math problems, it would go crazy. It would do some math. It would get to an answer. I think it was 12. And then it would it would immediately go back and say, like, oh, no, oh, my God. The answer actually is for sure. 36.”
“It had been trained by accident on similar questions where the answer was incorrectly marked as 36. And so the model knew that the correct answer was 12. And it also had sort of been optimized to say 36.”