Vals AI Legal Aid Benchmark Finds Models Fall Short on Local Help

NextLadder Ventures and Vals AI announced the Access to Justice Benchmark on November 10, 2026, reporting that none of 23 AI models reached its 80% threshold for a reliable first point of contact across 252 legal-aid scenarios.

The benchmark results and methodology show a sharp divide between explaining law and finding practical help. Vals AI reports that models satisfied 78% to 87% of expert criteria for general legal explanations, but only 30% to 50% for identifying specific resources such as a local legal-aid program, court self-help center or particular form. Twenty-one models named a national directory, including LawHelp.org, the ABA’s Find Legal Help or 211, in at least 70% of their final answers instead of consistently identifying the relevant local program or office.

The joint announcement puts Qwen 3.8 Max in first place with 76.4% of criteria met, followed by Claude Sonnet 5.5 at 74.9% and Wonder Full at 74.7%, with all three within one another’s margin of error. That clustering means the numerical order does not establish a clear leader.

Meghana Kotcherlakota, a research scientist at Vals AI, contrasted the project with legal AI benchmarks focused on law firms and corporate clients: “We built the Access to Justice Benchmark to show how well these models serve the people who need help most, problem by problem, and to track whether they get better.”

The release frames the results against the Legal Services Corporation’s 2022 Justice Gap study, which found that low-income Americans received no or insufficient legal help for 92% of their civil legal problems. Among those who did not seek help, 46% cited cost. The research drew on a survey of approximately 5,000 adults by NORC at the University of Chicago and data from legal-aid organizations funded by the corporation.

How the Benchmark Scored First-Contact Help

The benchmark casts each model as a first point of contact similar to an intake worker or court self-help center, providing general legal information rather than legal advice. Two legal practitioners supplied real-world queries, after which six legal-aid experts adapted them to assigned states: California, Louisiana, Virginia, New York, Maryland and Texas. The experts wrote opening questions, background facts withheld unless the conversation elicited them, difficulty ratings and rubrics averaging 15.4 independent criteria.

The full dataset contains 504 scenarios, divided evenly between the public test set and a private validation set. Each target model could search the web, open pages and question a GPT-5 simulated client before Decider v1.2, an automated judge, graded the complete conversation. Only information requested by the target model was supplied, and the simulated client neither added other details nor challenged faulty assumptions. A model’s score was the pooled share of rubric criteria met across the test scenarios.

Conversation and Source Choices Exposed Gaps

Statutes, court rules and case law accounted for 42% of the pages models read, while legal-aid and nonprofit self-help sites accounted for 10%. Vals AI links that pattern to the split in scores: reading statutes can support a general summary, but it does not supply the name of the organization or court office a user should contact next.

Vals AI’s conversation analysis found an average of about 3.5 exchanges, with most models delivering a full answer in their first reply before asking questions, if they asked any. GPT-6.1 Sol ended its first reply with a question 88% of the time but ranked 18th at 68.1% on elicitation, the rubric’s measure of whether a model asked for required facts. The contrast shows that frequent questions did not necessarily surface the facts specified by the experts. Models that did not ask sometimes assumed details such as the user’s state or whether a court order already existed.

Vals AI’s topic results put housing and landlord-tenant scenarios highest at 75.4% across all models, while public benefits scored lowest at 57.3%, although that category contained only five scenarios. Family law supplied 110 scenarios and averaged 71.2%; within it, visitation scored 61.3% and kinship caregiving involving child protective services scored 63.0%.

Two examples in the announcement show how those weaknesses affected guidance. In a Louisiana small-claims case involving a verbal repayment promise, only eight models asked whether the debt involved real estate, four warned that expiration of the 10-year filing deadline would prevent recovery, and three directed the user to the court clerk’s website for claim and fee-waiver forms. In a New York City security-deposit scenario involving a tenant who had moved out, 14 models were marked down for recommending an out-of-court demand-letter step that the expert rubric warned against, and only four referred the tenant to the city’s Right to Counsel program.

Free-Tier and Premium Model Scores

In a separate comparison of major chatbot providers, Vals AI’s estimated free-tier group averaged 69.7%, versus 70.4% for premium models. Claude Sonnet 5.5 scored 1.2 percentage points above Claude Opus 5.5, and only xAI’s premium model showed a clear advantage over its provider’s free-tier model. The comparison’s tier assignments are estimates, not confirmed provider mappings, and the identical API tests cannot fully capture consumer chatbot behavior; open-weight models were excluded.

The organizations said they are collaborating on a fuller analysis and an additional brief intended to help frontline practitioners understand the remaining gaps. For legal-aid organizations evaluating these systems, the practical implication is that explaining the law is only one part of first-contact performance: the benchmark also grades whether models uncover case-changing facts and supply a specific local next step.