How to Money · Friday, October 2, 2026
A comprehensive study by the AI and technology firm Saturn found that 18 AI models, including paid versions, provided incorrect financial answers 57% of the time. Joel expresses concern over users giving sensitive financial information to chatbots, despite Meta's AI, Muse, offering potential benefits like identifying unused subscriptions.
“And this is, man, I'm going to read this verbatim. This is the most in-depth benchmarking exercise of its kind to date. They tested 18 AI models across 121 different money-related questions with each question run five times to account for variability. In total, more than 10,000 responses were assessed.”
“Wrong answers, 57% of the time, which is like mind-blowing. Like this is a research firm that really went through and painstakingly did this. And even still, less than a coin flip, was it giving you the correct answers, which is not what you want to see.”
“I am just worried. I'm less worried about AI strangling me in my bed, but I am worried about if I give it extra information that it doesn't need, instead of asking it questions, And using that thoughtfully to parse through how I react and what I choose to do. Instead of doing that, people are just like saying, here's everything. Check out my life, my calendar, my email, all my financial stuff.”