California’s recent bar exam disaster casts doubt on the feasibility of off-site exams
Is technology ready for the bar exam?
California thought so. The cash-strapped state bar, citing financial concerns, chose to develop its own test designed to be taken remotely and at test centers, a move that was projected to save $3.8 million.
But the February launch of the test, written by Kaplan Exam Services and administered by Meazure Learning, was a disaster. Much of the tumult stemmed from tech troubles, including glitches cutting and pasting answers, uncertainty if and when answers were saved, and being kicked off the internet. The savings were a pipe dream. Instead, the move online resulted in scoring errors, finger-pointing lawsuits, resignations, furious candidates and projected costs of approximately $5.6 million.
The debacle has also led to questions about whether technology is capable of doing what California had been hoping to do. According to individuals interviewed by the ABA Journal, the answer is far more nuanced than go/no go.
“Which tech are you talking about?” says Greg Sarab, CEO of Extegrity, a testing software provider.
To administer a bar exam, several types of software are used, each handling specific tasks—ranging from registering for the exam, delivering the exam securely, proctoring and monitoring remotely, offering IT support and collecting answers, experts say.
Some systems operate fairly seamlessly, but others are trickier, particularly those involved in delivering the high-stakes exam remotely, like those that maintain the test’s security and remote proctoring.
“I definitely worry about the tech given California’s experience,” says Deborah Jones Merritt, a professor emerita at Ohio State University Moritz College of Law who has worked with several jurisdictions on updating their bar exams.
Bad precedent
Over the years, the bar exam, which is designed to ascertain a candidate’s minimum competency to protect the public, has faced its share of tech difficulties. The woes of the February administration of California’s new exam echo those during the COVID-19 pandemic, when distancing mandates forced state bars to experiment with remote exams.
In July 2021, 30 jurisdictions had remotely administered exams and 25 had in-person tests, according to the National Conference of Bar Examiners. The quick move to use new technology to create a remote exam was forgivable, some sources say.
“It was the pandemic. You had to do something,” says Dean Barbieri, director for examinations for the State Bar of California between 2001 and 2010. “It was the best way to do it in fairness to those applicants who had hundreds of thousands of dollars in student loans and needed to get on with their lives.”
Extegrity’s Sarab says he considered getting involved at the time. But he found that the test centers lacked adequate space and machines, and at-home testing brought up a host of proctoring issues, so he opted out. And despite the technological advances of the past four years, Sarab says he has no regrets about sitting out this time.
“I’d do it again,” he says.
High stakes
Some other high-stakes tests can be taken at home or a chosen location. For instance, the Medical College Admission Test, a standardized multiple-choice exam, is offered at testing centers.
“The only time technology today is not going to be a problem at all at a large-scale exam is the remote testing of something that is easy for machines to evaluate,” says Aron Solomon, chief strategy officer at Amplify and a journalist. “Multiple choice is a really, really good example.”
Even so, tech troubles have emerged at other high-stakes exams. During the March SAT college entrance exam, which is taken remotely and consists of mostly multiple-choice questions, 12.8% of international and .06% of domestic test-takers had their tests automatically submitted before the test time was up, according to the College Board. Others may have lost some testing time due to monitors asking them to reboot their devices to fix and prevent the auto-submit issues, a College Board release says.
For the bar exam, security of the test questions—no matter how they are delivered—is a sticking point because questions are often reused.
Currently, the Uniform Bar Exam’s multiple-choice questions are delivered to all jurisdictions simultaneously via paper booklets to keep questions from being leaked. As many as 50,176 examinees have taken the UBE at one time, according to the NCBE.
“All it takes is one person to compromise the security of the exam, and they have to burn all of the 200 questions,” Barbieri says, “To draft a sufficient number of multiple-choice questions and pretest them to make sure they perform the way they’re supposed to perform takes a long time.”
But security is even trickier for essay questions delivered electronically, Sarab says.
“Essay exams must be event exams, with everyone starting at the same time,” he says. As a result, remote is never going to be suitable for highstakes essays.”
What’s next?
While the NCBE plans to deliver its NextGen UBE exam entirely via the examinees’ personal computers, the new test launching in 2026 will not be conducted remotely but will be administered only at designated locations with in-person proctors.
But a fully remote bar exam brings different levels of concern, sources say. With in-person exams, even those delivered electronically, proctors sit in the testing center, keeping an eye out for cheating or examinees who have special issues. But with remote exams, that’s not possible.
“My sense is that remote proctoring is the most glitchy part of any computer- based testing,” Merritt says. Conducting proctoring via video camera or installing software on personal computers would seem like easy solutions but aren’t, sources say.
“There’s far too much technology working on both ends to expect that cameras or software that’s being installed onto your computer are going to be able to catch everything,” Sarab says. “What are we looking for? What’s that camera showing you, my face? If I scratch my nose, have I cheated?” he says. Sometimes second monitoring devices have been added, but “it adds a layer of complication,” Sarab says.
When Texas administered the bar remotely during the pandemic, “we couldn’t look at their video during the exam. We could look at it later,” says Susan Henricks, executive director of the Texas Board of Law Examiners from 2015 until her retirement in December 2020.
Watching hours of footage from hundreds of candidates, however, would take a great deal of time.
Even having live remote proctors for individual candidates can still pose problems, sources say. “If they say they have to go to the bathroom, you can’t say no, and they can have study materials there,” Henricks says.
And the proctors can’t control the examinee’s test location. “Anything could be happening in the room, whether it’s [artificial intelligence] or just chatting with your pal who’s got a JD,” Sarab says.
Several certification organizations for other professions have stepped back from remote proctoring, including the Society for Human Resources Management and the Behavior Analyst Certification Board. Technology, specifically AI, would need to evolve before it could keep a careful eye on candidates to ensure they are not cheating and hit the standard of consumer protection the bar exam stands for, Sarab says.
“Until you have a robot with artificial general intelligence sitting next to you while you take your exam—basically an actual proctor sitting next to you—they can’t hit the standard,” he adds.
The problem with remote proctoring, “like so many problems with technology, is it’s a good idea that works at a small scale,” Sarab says. “I would love to make it work. I’ve thought about it a lot, and I just don’t see how it can scale.” For California’s February exam, the state bar hired Meazure Learning to handle remote proctoring plus hired proctors on site at test centers. But the state bar has since sued Proctor U, Meazure Learning’s parent company, for fraud, negligent presentation, false promise and breach of contract.
Both the California and pandemic-era attempts prove that rushing the development of technology to administer the test can create big problems.
“There are so many moving pieces in administering a bar exam that really just bring the tech problems to bear, right?” says Greg Bordelon, assistant professor at Suffolk University and former executive director of the Louisiana Committee on Bar Admissions. But most don’t believe using tech for the bar exam is an all-or-nothing proposition.
“That’s the way the California bar exam was looking at it, and they got burnt because they turned the switch on before they showed that it was ready,” Solomon says.
Months later, California’s state bar continues to consider remedies for the nearly 3,900 bar candidates who completed the February exam, even after it lowered the pass score and made other scoring adjustments in addition to offering provisional licenses to some of the candidates who didn’t pass. And in May, the state supreme court mandated that the July administration be held in person.
Solomon says what tech and how it is used for the bar must be thoughtful decisions.
“It’s not like we’re using tech or we’re not using tech,” Solomon adds. “It’s about how mission- and situation- appropriate the technology is. I think we iterate, which is what technology does.”
Still, many believe in tech’s promise for the bar exam.
“We watch space shots to the moon,” Barbieri says. “I don’t see any reason why tech should not be utilized to the greatest extent possible.”
Write a letter to the editor, share a story tip or update, or report an error.
