AI Benchmarks Reality Check: Finance and Legal Tasks Score Zero
Researchers at SemiAnalysis released GDPx on November 2, a new test that puts leading AI models through long, realistic assignments in eleven high-value fields. The headline result is humbling: the top performer clears only 12% of the work.
If you are choosing AI tools for your company, that number matters. It follows the AI hype cycle conversation with real data, and it shows why a vendor’s favorite chart may say little about your daily workload.
What GDPx Actually Measures
GDPx spans eleven fields, from semiconductors, hardware and materials work to security, biology, trading, software, law and finance, plus tests of alignment and of models that improve themselves. The authors deliberately skipped sectors such as government, real estate and insurance. Their view is that early AI revenue will cluster where frontier intelligence pays off fastest.
Each task runs inside a rich simulated workplace, and the default time limit is eight hours. Scores use pass@1 across three attempts, and every domain carries equal weight. Fourteen specialist partners supplied the tasks, and SemiAnalysis says it only accepted partners whose work passed its quality checks on the first try.
Why Finance and Legal Scores Stayed at Zero
The finance example shows how demanding the test is. Models land inside a mock private equity deal in which a fund buys roughly 62% of a snack company for $880 million. Then they must dig through about 2,000 files to find the facts, much like a junior associate would.
| Item | Reported figure |
|---|---|
| Best overall model score | 12% |
| Finance and legal scores for nearly every model | 0% |
| Specialized data partners | 14 |
| Default time limit per task | 8 hours |
For founders, the takeaway is practical. Accounting, tax and contract work often feels ripe for automation, yet this test suggests human review should stay in the loop for now. Treat AI output in those areas as a first draft that a qualified person checks before anything reaches a customer or an investor.
Legal work faced a similar wall. Because of that, the early wave of legal AI startups deserves a careful reading: strong demos do not guarantee strong results on messy, multi-file matters.
Where Older Benchmarks Fall Short
SemiAnalysis argues that OpenAI’s GDPval, released on September 25, 2025, was impressive for its time but predates modern agents. Each GDPval task hands the model every needed detail in the prompt. Real jobs rarely work that way, because gathering context is a large part of the effort.
This gap matters for buyers. A tool that shines when everything is spoon-fed can stumble once it must search email threads, spreadsheets and old PDFs, which is exactly how most small companies store their knowledge.
The team also points to grading problems. Human experts agreed with each other only 71% of the time on a gold subset, according to the report. In addition, its own quality screen flagged nearly half of the tasks in another popular coding benchmark as broken.
How to Test AI Vendors Against Your Own Work
You do not need a research lab to apply the lesson. First, collect five real tasks from your business, such as a pricing memo, a contract review or a month-end reconciliation. Next, give each vendor the same messy files you would give a new hire, and score the results against your own standard.
Then compare cost as well as quality. Prices can shift quickly, as the story on LLM API pricing showed, so rerun your tests whenever a plan changes. For a simple way to document risks and results, the NIST AI Risk Management Framework offers a free structure that small teams can borrow.
Finally, watch for safety filters. SemiAnalysis notes that classifiers in some models lowered their scores in cyber, biology and alignment tasks. A model that looks weaker overall may still be the best fit for your narrow use case.
Questions Founders Ask About AI Benchmarks
Does a low score mean AI is useless for my business?
No. A 12% headline reflects very hard, eight-hour expert tasks. Many everyday jobs, like drafting emails or summarizing calls, are far simpler than those challenges.
Which model won?
SemiAnalysis names Opus 5.5 the strongest overall, ahead in trading plus chip and hardware design. Gemini 4 Carbon took biology and software engineering, and GPT-6 Astra took materials science and the self-improvement tests.
Next Moves as Models and Tests Evolve
Expect more specialized benchmarks built with domain partners, and expect vendors to cite them in launch posts. When they do, ask who designed the tasks, how graders were checked and whether the work resembles yours.
Also watch finance and legal scores. A first model to move off zero would signal that agents are ready for higher-stakes office work, and it would change how you plan hiring and tooling. Until then, treat benchmark claims as a starting point and your own pilot as the final exam.
