Three Tips We Learnt from Benchmarking Claude as a Quant Analyst

In this piece for Nesta, author Seth Caldwell looks into the reliability of AI coding agents as quantitative analysts, examining whether they can move beyond building technical infrastructure to independently directing and executing research. Caldwell finds that these tools perform extremely well on closed, clearly defined, and verifiable tasks, such as cleaning datasets, standardizing administrative records, and building classification models. When the research plan and success criteria are clear, AI agents can effectively handle the implementation.

However, the benchmarks also reveal significant weaknesses on messy, open-ended problems requiring domain knowledge and judgment. Agents frequently produced clean and confident but incorrect datasets when confronted with changes in reporting methodologies, with success rates improving only when the relevant methodological shift was explicitly specified. Similar problems arose in data matching, cleaning, aggregation, and other tasks where understanding the underlying context was essential, leading Caldwell to conclude that AI agents can fail silently when analysts do not already understand the problem well enough to identify errors.

Caldwell also highlights concerns about methodological decision-making. AI agents may select different statistical approaches based on programming language or available libraries rather than the research question, while alternative modelling choices can substantially change results. In some tests, agents failed to identify violated assumptions and presented questionable findings as reassuring or inconclusive. These results suggest that although AI agents are capable builders, they are not yet sufficiently careful or reliable methodologists to take control of complex analytical work.

As a result, Nesta is using coding agents primarily for well-scoped analytical tasks, software engineering, testing, code review, and quality assurance while keeping humans responsible for methodological decisions. Caldwell recommends that organizations develop their own benchmarks based on the complex work they actually perform and continually test new models and tools against them. He also suggests using AI to challenge and reproduce human analysis rather than replacing human judgment, while exploring whether structured workflows, specialized skills, and multiple collaborating agents can eventually improve AI performance on more demanding analytical tasks…

CONTINUE READING…