newsfilter
← Latest News
Lightspeed Venture Partners·

Introducing Financial Audit Bench: Evaluating Audit Capabilities of AI Agents

Summary

  • Modus Labs released the open-source Financial Audit Bench (FAB) to evaluate 11 frontier AI models on multi-step financial audit procedures.
  • While models passed over 85% of individual checks, the top model achieved a strict pass rate of only 44.4% when required to succeed in all eight runs of a procedure.
  • Specific task performance varied widely, with journal-entry completeness checks at 94.5% but revenue testing at 8% \& accounts receivable at 4%.
  • The benchmark utilized six synthetic company audits across 45 workpapers to prevent client data exposure while mirroring staff-level auditor fieldwork.
  • Key failure modes included agents stopping searches too early \& trusting incorrect evidence sources, such as using client cash-receipts reports instead of bank statements.
  • Modus recommends deploying AI agents as co-pilots rather than autonomous preparers due to current consistency limitations.
  • The strict pass rate for leading models improved from below 50% to 69.2% during the benchmark development phase.
  • Future evaluations must measure not just correctness but also whether the AI's documentation allows another auditor to review \& follow the work.
  • Modus Labs has published the paper, code, dataset, \& live leaderboard to enable industry-wide verification \& model iteration.

As more audit firms are putting AI agents to work, the question they keep running into is simple: which parts of an audit can an agent do well? Until now, there was no shared way to answer it.

Keywords
AI