Is AI getting better at law school exams? Maybe not, new study shows
The latest version of OpenAI’s reasoning model might not be much better at taking law school exams than last year’s version, according to a new study. (Illustration by inkoly/Getty Images)
The latest version of OpenAI’s reasoning model might not be much better at taking law school exams than last year’s version, according to a new study by the University of Maryland Francis King Carey School of Law.
Last year, OpenAI’s o3 reasoning model took the professors’ final exams, and its answers were graded on the same curve as students. It impressively received grades ranging from A+ to B, according to the authors.
This year, when the researchers repeated the experiment with the updated model, GPT-5.5, it didn’t noticeably outperform last year’s model, and in fact, it may have done worse, according to the study.
“This apparent plateau, despite a better model and using extra-high inference-time compute, may reflect trends observed in other legal benchmarks for AI and makes theoretical sense,” they concluded, adding that they plan to continue the experiment in future semesters.
The TaxProf Blog had coverage.
See also:
Artificial intelligence can be used for grading law school exams, but should it be?
Some law profs move to old-school assessment methods to combat AI
Write a letter to the editor, share a story tip or update, or report an error.
