Google Opens 85-Task Engineering Benchmark and Agent Competition

Google AI & Infrastructure announced on November 3, 2026 that it was releasing 29 public tasks, grading tools and an evaluation container for Physical-Engineering’s Last Exam (PLE), an 85-task engineering benchmark developed with Mercor, together with an open agent competition and leaderboard.

PLE asks an agent to complete professional assignments in mechanical, electrical, civil and structural engineering rather than answer textbook-style questions or make an isolated tool call. The benchmark contains 30 mechanical, 27 electrical, 18 civil and 10 structural tasks, with success measured through deliverables that must satisfy engineering codes, physical constraints and, for design work, cost or performance objectives.

The paper’s arXiv record lists an initial submission on October 21, 2026 and a revision on October 30, 2026. The new development is the public release of tasks and evaluation tooling and the opening of the competition, rather than the paper’s first publication.

Project Files and Production Deliverables

Each assignment runs in a containerized project environment built on Mercor’s Archipelago. Tasks average 61 files and total 5,170 across the benchmark, spanning PDFs, spreadsheets, emails, 2D CAD drawings, 3D BIM models and superseded revisions. An agent must decide which documents and requirements govern, perform calculations and issue files such as engineering memos, calculation workbooks, equipment schedules and drawings in formats intended for licensed review.

Practicing engineers built the 44 expert-authored tasks end to end, including their project folders, prompts and weighted grading rubrics. For the 41 co-authored tasks, engineers supplied the initial briefs; an adversarial Creator–Solver pipeline constructed and tested the environments and executable graders from real licensable materials, after which experts validated the tasks and increased their difficulty. The technical paper says no task includes proprietary or client-confidential information: expert-authored environments were written from scratch, while co-authored files use public-domain, permissively licensed or newly generated material.

Grading reflects the same emphasis on completed work. Expert-authored tasks use weighted criteria tied to evidence in the submitted files. Co-authored design tasks combine constraint compliance with an economic or performance objective, and a failed hard safety or feasibility gate prevents the affected design from earning objective credit. The objective component addresses a limitation of rubric-only scoring: two wire schedules can satisfy the same code checks while differing in installation cost.

Baseline Results Show a Compliance Gap

Google evaluated 14 models with seven publicly available agent harnesses, running each task three times per configuration. The solver containers had no internet access and allowed six hours of agent time per run; rubric judging and executable grading took place outside the solver’s container, keeping graders and reference deliverables away from the agent.

On the 56-task private test set, Google reports that every model scored below 50% on its native harness, led by Claude Opus 5.5 in Claude Code at 48.3%. The competition leaderboard ranks models using those hidden-task results.

The paper’s separate evaluation across all 85 tasks reports a high of 50.6% for Claude Opus 5.5 in Claude Code, averaging 27 million tokens and 63 minutes per task and costing $17.18 per task at list prices. The two leading figures therefore describe different evaluations: one covers the hidden test set with native harnesses, while the other covers the complete benchmark and the strongest model-harness configuration.

The more revealing result sits inside the co-authored tasks. Google reports that the strongest configuration satisfied 91% of individual constraints, yet only 41.3% of its deliverable items passed every hard safety and feasibility check required to earn objective credit. In practical terms, the agents often got most requirements right without producing a fully compliant design—the exact gap PLE’s hard-constraint gate is designed to expose.

Google reports an approximately 50-point spread across models, against a median three-point spread across harnesses for the same model, although harness choice still changed scores and costs. Individual task results also varied substantially, with an average three-run range of 15.4 points for a task-harness cell. That variability supports the authors’ use of repeated-run averages rather than single attempts.

Competition Tests the Whole Agent Setup

The official repository divides PLE into 29 public development tasks and 56 hidden test tasks. An entrant submits its agent harness—the software that drives a selected model through each task—along with the model choice and artifacts produced on the public tasks. Organizers grade those artifacts, rerun the harness on sampled public tasks to check reproduction, then run it three times on the hidden tasks to determine the leaderboard score.

Entrants can use 13 Google-sponsored models free on the evaluation server, provide a commercial model API key or connect a self-hosted model server. The Apache-2.0-licensed repository includes starter entries for custom harnesses and adapters for existing agents, giving participants several starting points for building a complete submission.

The competition therefore evaluates more than a model’s final answer. A submitted system must navigate realistic project files, reproduce its public results and produce complete deliverables that survive hidden safety, feasibility and quality checks. The baseline results make full-package compliance the central challenge: satisfying most individual requirements is still insufficient when one hard-constraint failure can deny objective credit to an otherwise economical design.