Title: Agents' Last Exam
Executive summary:
The Problem: Today’s AI systems are acing academic benchmarks, yet businesses are struggling to translate these technical wins into real, deployable economic value. The root cause is a testing gap: widely used benchmarks measure performance on short, isolated prompts rather than the long-horizon, complex workflows that human professionals execute every day.
The Breakthrough: Agents' Last Exam (ALE) is a new, rigorous benchmark designed specifically to test AI agents on real-world, economically valuable jobs with verifiable outcomes. Co-developed with over 250 industry experts, ALE maps directly to the U.S. federal occupational taxonomy (O*NET / SOC 2018). It spans over 1,000 professional tasks across 55 sub-fields and 13 industry clusters. Furthermore, it operates as a "living" benchmark that will continuously evolve as new industries and workflows are onboarded.
The Reality Check: Despite the hype, ALE reveals a stark truth about the current state of AI: the hardest professional tasks remain entirely unsolved. Across mainstream agent frameworks and foundation models, the average full pass rate on the top-tier tasks is currently below 1%.
Business Impact: For enterprise executives, ALE provides a much-needed baseline - a way to verify if an AI agent can actually complete a multi-step job before you invest in deploying it. For AI builders and startups, that sub-1% pass rate highlights a massive, untapped market opportunity. By shifting the focus from academic trivia to commercial-grade workflows, ALE acts as a blueprint for building AI that delivers true, GDP-relevant impact.
Generated by Gemini