Protege Launches Independent Healthcare AI Tests as Disease Odyssey Exposes Diagnostic Gaps

Protege is launching independent evaluations of healthcare AI, putting models through tests on patient cases they have not seen before. Its first benchmark, Disease Odyssey, offers an early indication of why that matters: according to the company, leading models identified conditions such as sleep apnea and anemia in only 53%–69% of cases when working from patient histories.

The October 21 launch also includes evaluations with Xlue AI and Standard Model Biomedicine. Together, the results address two questions that matter to hospitals and developers: whether a model can extract useful clinical signals from real records, and how much it costs to produce those predictions.

Protege’s proposition is to move more of that assessment outside the organizations building the models. It designs, runs and scores evaluations using privacy-protected patient records and clinician-verified answers, giving developers a way to identify weaknesses before making broader claims about performance.

A Different Test From Answering Medical Exam Questions

A medical question with a neatly stated diagnosis is different from a patient history spread across several visits. Symptoms develop over time, records can be incomplete, and the decisive clue may be buried among information that is less relevant. Disease Odyssey focuses on recognizing illness before diagnosis from these longitudinal histories.

The reported 53%–69% range applies to the cases and conditions tested, rather than to healthcare AI as a whole. Protege’s announcement does not provide enough detail about sample sizes, the exact comparison models or uncertainty around the results to support a broader ranking. What it does show is that substantial room for improvement remains on the company’s initial benchmark.

“Healthcare AI is moving quickly, but the way we evaluate it hasn’t kept pace,” said Bobby Samuels, Protege’s co-founder and CEO, in the announcement.

How Protege Separates Testing From Development

Protege describes its process as double-blind: model builders do not see the patient data or questions ahead of time, while Protege evaluates the systems without knowing how they were built. Here, the term describes the evaluation arrangement, rather than a randomized clinical trial.

Keeping test material out of development matters because exposure to benchmark questions can inflate apparent performance. A model that has encountered an answer during training may look more capable than it is when faced with a genuinely new case.

The company’s healthcare evaluation platform adds several practical controls. It describes protected, held-out test cases, versioned tasks and scoring, and analysis of failure patterns and affected populations. Versioning makes comparisons between model releases more meaningful: a higher score is easier to interpret when the task and scoring rules have not quietly changed.

The platform draws on linked information including electronic health records, clinical notes, imaging, pathology, claims and outcomes. Its Data Lab translates clinical questions into evaluation tasks and reference answers. That combination is intended to test how models use clinical context, rather than just whether they reproduce isolated medical facts.

Specialized Models Compete on Performance and Cost

The accompanying evaluations examine different tasks, so their results should be read separately from Disease Odyssey.

In work with Standard Model Biomedicine, Protege evaluated predictions involving breast and lung cancer survival and the time until a patient’s next treatment. The company reports that the specialized model performed competitively with leading systems while costing 36–170 times less per prediction.

A separate evaluation of Xlue AI’s HealthSAW model examined future risks involving cancer, cardiovascular disease and neurological conditions. Protege says HealthSAW outperformed leading models on 14 of 16 tests, at less than 0.1% of their cost per patient.

These are task-specific comparisons reported by Protege, not a demonstration that one model is best for every clinical setting. The announcement does not spell out the full cost methodology. Lower prediction costs could make repeated screening or large-scale analysis more practical, but deployment costs also depend on integration, oversight and the work created by incorrect results.

Evaluation Needs to Keep Up With Model Updates

Protege’s broader business supplies data for AI development, making the separation between development data and evaluation data an important part of its approach. Its materials emphasize provenance, controlled access and refreshed patient cases as ways to preserve the usefulness of testing over time.

The company plans to expand the evaluations into areas including pathology, dermatology, oncology and administrative workflows. Its aim is a repeatable process that can be rerun as models change, rather than a single score attached permanently to a product.

That is useful for both buyers and builders. A hospital needs to know whether an updated model still works for its population and workflow. A developer needs more than an average score: it needs to understand which cases fail and whether an improvement in one area introduces weaknesses elsewhere.

Independent testing can make those conversations more concrete, but retrospective benchmark performance is not the same as demonstrated benefit in clinical practice. Prospective studies, performance across patient groups, false-positive and false-negative rates, and the effects on clinicians’ workloads remain important pieces of the evidence.

Protege’s launch puts a sharper question in front of healthcare AI companies: how does the system perform when the cases are unfamiliar and someone else sets the test? The value of the new service will depend on how clearly its methods and results help organizations answer that question—and how consistently the tests expose problems that polished demonstrations miss.