Audit of California bar exam disaster blames poor planning, contract failures and rushed timeline
A rushed timeline, poor planning, contract failures and a host of questionable decisions—including the use of generative artificial intelligence to help write some questions—are to blame for California’s disastrous launch of a proprietary bar exam in February 2025 that led to $5.1 million in costs plus millions more in lost revenue and legal costs, according to a report from the state auditor released Thursday.
Ironically, the move to launch the exam that allowed test-takers the option of taking it in person or remotely was an attempt by the cash-strapped State Bar of California to save money in the face of an anticipated $3.8 million deficit.
“The state bar did not achieve cost savings in its administration of the February 2025 bar exam,” the 61-page report stated, noting the launch of the new test cost $5.1 million, excluding ongoing legal costs.
Unlike the widely used Uniform Bar Examination and its components, which are administered and developed by the National Conference of Bar Examiners, the hybrid test written by Kaplan Exam Services and deployed by Meazure Learning was designed to be taken remotely and at test centers.
The audit noted that the state bar signed a contract with Meazure Learning without fully verifying its ability to effectively administer the exam, which was then plagued by software glitches, crashes, error messages and unhelpful proctors. For instance, the report found that half of the nearly 2,700 post-exam survey respondents reported computer crashes and freezes, and nearly 1,200 respondents reported error messages during the multiple-choice portion of the exam.
The state bar was aware of potential technical problems related to Meazure Learning’s exam administration as early as November 2024, the audit added.
In addition, the haste in launching the exam amplified problems. The audit notes that the bar signed a five-year, $8.25 million contract with Kaplan to develop multiple-choice questions in August 2024, just six months before the exam. And despite a problematic test run in November 2024, the bar went ahead with the exam without Meazure Learning ensuring it fixed those issues, the report stated.
The report also revealed that only one subject expert was employed to review 200 multiple-choice questions, and 23 questions that made it to the exam did not receive any review. In comparison, the National Conference of Bar Examiners uses committees composed of legal experts jointly determine questions’ appropriateness and employs two independent subject matter experts per subject to develop questions for the Uniform Bar Exam.
Meanwhile, 20% of the exam’s 200 multiple-choice questions had performance issues, and six raised legal accuracy concerns, the report found, with 13 of the 29 exam questions generated with the help of AI raising performance concerns.
“We fully acknowledge the unacceptable experiences exam-takers faced leading up to and during that exam. We agree with all of the audit’s recommendations, which are consistent with the changes we have already implemented or are in the process of implementing, and the direction we have set for exam development and contracting,” said José Cisneros, chair of the State Bar of California Board of Trustees, in a statement.
The report comes at a key moment for the bar exam in California. After the February 2025 exam, the California Supreme Court ordered a return to the Multistate Bar Examination until the NCBE sunsets the Uniform Bar Exam in July 2028.
Now the state is under the gun to decide which exam to use. Under a new California law aimed at allowing candidates and law schools enough time to prepare for a new exam, there must be at least a notice of two years before the state can make substantial changes to how the exam is administered. That means bar officials must make a recommendation to the California Supreme Court with enough time to make the final decision by July 2026. The contract with Kaplan remains in force, and there has been discussion about Kaplan creating a future California-specific test.
Write a letter to the editor, share a story tip or update, or report an error.
