º£½ÇÉçÇø

Title: 

Journal: New England Journal of Medicine AI (September 25, 2025)

Background

Large language models (LLMs, the technology behind tools like ChatGPT) are increasingly used in clinical decision support, yet common evaluations—such as medical licensing exams—rarely assess whether decisions are revised appropriately when new or uncertain information appears.

To address this gap, we used Script Concordance Testing (SCT), a long-standing method in medical education that quantifies how new data should shift diagnostic or therapeutic judgments under uncertainty, and we evaluated leading LLMs using purpose-built questions. These items do not fully mirror real-world care, but they target a core element of clinical reasoning: knowing when evidence should change your mind and by how much.

Methods

We created a public benchmark of 750 SCT questions drawn from 10 international datasets across multiple specialties, nine of which are newly released. Each question presents a clinical scenario and asks how additional information affects the likelihood of a diagnosis or management option, scored against expert-panel responses. We compared the performance of 10 state-of-the-art LLMs with more than 1,500 human participants, including medical students, residents, and attending physicians.

Results

The top model had scores comparable to medical students and junior resident physicians, but they lagged behind senior resident physicians and attending physicians. Concerningly, models were overall quite overconfident in their responses, and recent developments in the technology have actually made this problem worse.

Even when the extra information was completely irrelevant, more advanced models convinced themselves that it was important. They shifted their judgments too far in response to weak or equivocal information and were less likely to appropriately indicate that the new information is irrelevant to the question at hand.