Academics warn financial firms against blind belief in AI ‘magic eight balls’

Banks and financial firms deploying artificial intelligence (AI) need to understand how the technology actually works and avoid viewing large language models (LLMs) as ‘magical beings’, Sandra Wachter, professor of technology and regulation at the University of Oxford, warned the Treasury Committee recently.

Giving evidence to the committee’s inquiry into the use of AI in financial services, Wachter said that while LLMs were often presented as “magical beings, or magic eight balls that always tell the truth”, in reality all they actually do is “just try to predict the next word”.

“This is why [computational linguistics expert, Emily] Bender and her colleagues call it stochastic parrots: it is just parroting back what it has heard,” Wachter said.

Academics submitted the largest number of written responses to the committee’s inquiry into the use of AI in financial services. In all, academics from 26 universities submitted evidence. The next most vocal group were financial services trade associations — 22 of which submitted evidence — followed by seven consulting groups, six tech firms, six fintechs and five traditional financial services groups.

While many of the academics highlighted opportunities for the financial sector from AI, all of them raised concerns about its unrestrained adoption.

AI adoption

According to the Bank of England and the Financial Conduct Authority (FCA), 75% of UK-authorised financial institutions are already using AI in their operations and a further 10% intend to deploy it in the next three years.

Data from European regulators suggests a similar adoption rate. The European Banking Authority (EBA) said 86% of European Union-based banks used AI in 2024, while separate research from the European Insurance and Occupational Pensions Authority (EIOPA) found that 80% of insurers would be using AI by 2027.

According to regulators, financial firms are using AI to optimise internal processes, mitigate external cyber treats, increase fraud detection, and improve customer support.  The Bank of England/FCA research found that 17% of AI use consisted of so-called foundational models. This is were an AI model is trained on vast amounts of data which serves as a base that can be adapted to perform a wide range of tasks.

Herding fears

Neil Lawrence, DeepMind professor of machine learning at the University of Cambridge, said lawmakers needed to address the very real risk of “placing excessive confidence in AI”, and he advised the committee to listen to “honest brokers”, including independent scientists rather than “issue advocates” who might have a financial interest in the deployment of AI.

Galina Andreeva, a professor at University of Edinburgh Business School, agreed there was a real problem with people becoming over-reliant on AI agents.

In written evidence to the committee, Queen Mary University of London academics Daniele D’Alvia and Rosa Lastra referenced the case of DeepSeek vs Nvidia to highlight the vulnerabilities associated with AI-driven investment strategies.

“The over-reliance on AI models for investment decision-making can lead to market distortions, unintended herding behaviour and increased market volatility. In the DeepSeek case, AI-driven investment models fuelled excessive speculation in Nvidia stocks, amplifying market momentum and creating a fragile investment cycle susceptible to sudden reversals. Such episodes underscore the need for robust regulatory mechanisms to mitigate AI-induced financial instability,” the pair said.

In his written evidence, Nader Virk, an associate professor in finance at Manchester Metropolitan University’s Business School, also pointed out that “herding” could be exacerbated by AI. This could “create and inflate asset bubbles — triggering market crashes — that will pose a significant challenge to traditional risk management frameworks” of asset managers, he said.

Independent testing

Internal validation and testing of AI models is essential, Andreeva told the committee. Credit institutions were accustomed to putting internally developed models through robust challenge processes, she said, adding that similar independent assurance systems should be adopted for testing AI agents.

Regulators should provide guidance of explainability and transparency, according to Simon Weidenholzer, professor of economics at the University of Essex, who has conducted research into robo-advice models.

“A possible downside of robo-advice is the lack of transparency. Many robo-advisers operate as ‘black boxesʼ, making it difficult for consumers and regulators to understand how decisions are made. Regulation should thus be aimed at increasing transparency, which may also ease investor scepticism towards this new technology,” he said in his written submission.

The FCA said earlier this year that it did not plan to write new regulations regarding the use of AI by financial services firms. It said its existing rule book on the Senior Managers Regime and the Consumer Duty  gave it sufficient tools to oversee the sector.

In contrast, the EU AI Act has designated certain financial services — credit and insurance pricing — as being sufficiently high risk as to require separate supervision.

Rollback

Not all use cases for AI have proved successful for financial firms. Klarna, the buy now, pay later (BNPL) provider, recently announced it was re-employing humans in its customer support functions. The decision was a major about-turn for the company, which claimed very publicly last year that its AI chatbot did the work of 700 humans.

The Treasury Committee inquiry will publish its conclusions and recommendations in the autumn.