Training Data · Wednesday, July 29, 2026
Jerry Tworek explains that transformers have been economically valuable because their training cost is lower than the revenue they generate, unlike LLMs which might not offer the same economic advantage. He highlights that OpenAI's success was partly due to a contrarian bet on scaling algorithms with more compute, rather than solely focusing on algorithmic efficiency, a strategy that was met with criticism.
“The majestic thing about transformer which goes back to like why why do we have to appreciate transformers so deeply is that transformers are economically valuable The training them the cost of training them is lower than the revenue that they generate which is magic of machine learning”
“a lot of reasons why people didn't scale things before was because researchers before OpenAI had a lot of reluctance to scale it was often seen as unscientific”
“what was the contrary bet by OpenAI at that moment try to say hey we don't care about better better algorithms we care about more and more scalable algorithms and how do we pour more and more compute and get better better results”