How accurate are AI money assistants?

There’s a lot of commentary these days about how easy it is to use an LLM to manage your finances. Just give it copies of your bank statements, or better still, link it directly to your bank accounts, and away it goes. There are even specialist tools being released to do exactly that, or you can use one of the new assistants or AI agents. Just ask Claude.

As a risk-averse banker, I look at all of these claims with a sceptical eye. Despite the obvious attraction of having something do all that analysis for you, my mind goes straight to privacy and security, and then to reliability and accuracy.

Let me talk about the second point.

Earlier this year, when I was researching how other companies build AI money assistants, I came across an article from Cleo (the US and UK money app) on how they test their accuracy against others.¹ I have to say, it was a pretty sobering read.

They took 129 real transactions from one month and asked six ordinary budgeting questions: what did I spend on bills last month, what’s my most expensive subscription, how much did I spend at each of three big stores, that sort of thing. They put the same questions to three general AI models and to Cleo.

Of the 21 answers the questions called for, Gemini 2.5 Flash got six right, Claude Sonnet 4 got seven and GPT-4o got 13. Cleo, which designed the test, got 17.

The standout for me was Gemini, which said the month’s bills came to $28,813. The right answer was $3,047. That’s a lot of electricity or rent.

Cleo’s own figures show how. $28,813 is 96% of everything that went out that month, so Gemini had counted nearly every payment as a bill. In fact, all three general models put the month’s income at about $30,000, when it was really about $6,000.

Cleo themselves say one-off transfers, like someone paying you back, were hard for every system they tested, their own included. And OpenAI warns on its help page that a spending breakdown can look wrong if transfers, card payments, reimbursements or duplicate pending transactions are counted as spending.²

Now, your first response to this might be to say ‘that’s last year’s models, and things have moved on’. And they have, to a point. But here are two recent results:

• OpenAI’s own test this year, marked by more than 50 finance professionals, gave its best model 82.5 out of 100 for the quality and accuracy of its answers to personal-finance questions.³
• Digits, an accounting software company (with a system of its own to sell), put 2,000 real small-business transactions to 11 of the latest AI models in October. The best sorted 79.5% into the right category. With the business’s past records to go on, the best model reached 88.9%.⁴

Accurately working out the details of money is harder than it sounds. Take these simple examples:

• How much did you spend last month? Say you move $500 into your savings account every month. It leaves your everyday account, but it’s still yours. Is it spending?
• Paying off your credit card takes money out too. But you’ve already counted what you bought on the card, so counting the repayment doubles it.
• A flatmate pays you back for the electricity. Is that income? Is the whole bill your spending, or just your share?
• Same with a refund. You’ve already spent that money. Or have you, now that it’s come back?

Some of these need a rule about what counts. Others come down to what the money was for, and a bank’s transaction description won’t help you much with that.

When you give an AI assistant your bank statements, the model ingests the data and has to decide for itself what each transaction is. When it decides wrongly, the answer comes out just as confident as a right one. That’s what people call model hallucination.

Only you know whether a particular transfer was for groceries or hardware. A careful person doing your books by hand would stop and ask: what about this one? It could be this, or it could be that. Or they’d tell you what they’d assumed, so you could put them right. They’d still get some wrong, but the misinterpretations would be transparent.

Solving these issues is how I spend my life at the moment. With Lucie Money, the aim is an assistant that behaves like that careful person, asking when it can’t be sure what you mean or telling you what it has assumed. And the figures themselves are designed to be worked out from your bank’s records, the way a spreadsheet adds up a column.

Some of this is built and some is still to come. But given the choice, I’d take an assistant that asks me a question over one that’s confidently wrong.