Software provider pulls out of remotely proctored bar exams because of technology concerns
Image from Shutterstock.com.
The National Conference of Bar Examiners has a remote proctoring requirement for states using its testing materials in October online bar exams. However, according to one of three bar exam software providers that recently pulled out of the online exam, the mandate may not be possible to carry out.
Greg Sarab, the founder and chief executive officer of Extegrity, says his primary concerns about a bar exam with remote proctoring include reliable internet connections being required for live remote proctored exams, and that the requirement of simultaneous start times comes with significant technological and procedural burdens. He also says there hasn’t been sufficient development time or product testing for the technology.
“Could it work? I had to make a business decision based on my perception of risk. My assessment of the risk was that I couldn’t advocate for it. So far, what you’d want to see is every single test run and exam day going off like clockwork, and they haven’t,” says Sarab, an attorney who started developing laptop exams in 1995. He co-founded SWA Software, which is now known as ExamSoft, in 1996, and formed Extegrity in 1998.
The NCBE in an email told the Journal that “each jurisdiction is working diligently with its selected vendor to ensure the October exam goes smoothly.” Hulett H. “Bucky” Askew, who chairs the organization’s board of trustees, on Aug. 14 told the council of the ABA’s Section of Legal Education and Admissions to the Bar that 30,000 people are expected to take the October online test.
ExamSoft and ILG Technologies are the other companies involved in bar exam software, according to the NCBE. Joseph Figo, executive vice president of ILG Technologies, did not respond to an ABA Journal interview request.
Nici Sandberg, an associate director of marketing content and communications at ExamSoft, in an email to the Journal wrote that if jurisdictions choose to move forward with remote exam delivery, her company will be ready to serve them.
“As an innovative, solutions-oriented software company, our role is to tackle difficult problems through technology,” she wrote.
Pilar Margarita Hernández Escontrías, a founder of United for Diploma Privilege, says the coalition has raised concerns about the reliability of bar software with remote proctoring.
“In an era where companies are rarely straightforward and forthcoming, Extegrity’s decision and subsequent statement [regarding] its decision is unusual and demonstrates integrity and ethics in the midst of very troubling times for bar applicants. How often do companies actually release statements against interest? This says a lot and we should be listening,” Hernández, a 2020 University of California Irvine School of Law graduate, told the Journal in an email.
Michigan delivered a remote bar exam, using ExamSoft software, on July 28. Some test-takers had a password glitch, and anyone who was affected got extra exam time, John Nevin, communications director of the Michigan Supreme Court’s office of public information, told the ABA Journal in July.
On Aug. 5 ExamSoft in a Tweet described the problem as a “DDOS cyberattack,” and stated that it had requested an official investigation from the FBI and the U.S. Department of Homeland Security.
Sandberg told the Journal on Aug. 18 that ExamSoft has spoken with the agencies, and “will follow the lead of those departments in pursuing this matter as they investigate.”
Meanwhile, Nevin told the Journal on Aug. 19 that the Michigan Supreme Court is consulting with its state board of law examiners and ExamSoft as they review bar exam information, “including input from law enforcement as they work to identify and prosecute the perpetrators.”
Nevada and Indiana, two states that initially planned to have online bar exams on July 28, both decided the week before to postpone the tests after users experienced delays when typing during practice tests. Both jurisdictions used software from ILG Technologies.
On Aug. 16, Florida announced that its online exam, scheduled for Aug. 19, had been canceled and would be rescheduled for sometime in October. The jurisdiction had planned to use ILG software for the August exam.
“Despite the board’s best efforts to offer a licensure opportunity in August, it was determined that administering a secure and reliable remote bar examination in August was not technically feasible,” the news release states.
The Nevada exam, which the state supreme court in May announced would be open-book, was administered Aug. 11 and Aug. 12. Indiana’s exam was given Aug. 4, with the state switching to an open-book format at the last minute and no remote proctoring.
Usually, nine jurisdictions use Extegrity software for bar exams, and Sarab says he was at various discussion stages with them for the remote exam when his company pulled out. The remote proctoring was to be provided by a separate provider, he adds, which Extegrity worked with for procedural and technical integration.
“We talked to experts, and the clear message was that for an ‘event’ type of exam, which the bar exam is, that wasn’t what remote proctoring was envisioned for,” he says.
While some remote proctoring involves watching test-takers during the exam, other offerings need test-takers to video themselves while taking the exam and then upload the video.
“If you’re recording a video, that uses quite a lot of system resources. The software running on the machine, the power of the machine, the quality of the broadband connection, all these would impact the ability of the computer to handle the workload and upload a file,” says Sarab.
He adds that some bar exam decisions he’s seen don’t seem to have sound technological underpinning.
“The bar administrators we talk to understand the technology, but it may be that their courts have taken over some decision-making and don’t have the same knowledge or experience,” he says.
Gregory Bordelon, who previously served as the executive director of the Louisiana Supreme Court Committee on Bar Admissions, told the ABA Journal he has concerns about the validity and reliability of a bar exam with remote proctoring because he’s not sure remote proctoring will do a good job protecting test security.
“It gives me pause that one of the three vendors has decided to pull out, and another of the three vendors has had significant problems,” he says, mentioning ILG.
Bordelon, now an associate professor and director of academic success at the University of Maine School of Law, adds that state supreme courts, which for the most part have the final say for how the bar exam will be handled, could think that because they are paying a company to handle remote proctoring, that takes care of any problems.
“Bar admissions people are tireless civil servants working very hard on behalf of courts, and sometimes they’re working with very limited resources. There’s the expectation from jurisdictions to say, ‘Our mission is to go out and protect the public.’ But oftentimes, it’s a challenge with resources, particularly with the circumstances we’re seeing now,” Bordelon says.
Bruce Schneier, a security technologist who is a fellow at Harvard University’s Berkman Klein Center for Internet and Society, says building technology for government entities is often difficult. Also, he thinks that when people discuss policy in technology, they frequently talk at each other rather than engaging in active listening to find the best solutions.
“It’s not a matter of just plugging in the system and saying, ‘Go.’ It’s hard, and it needs to be thought about. To the extent it hasn’t been thought out, that’s kind of a recipe for disaster,” says Schneier.
Many universities now use online proctoring for tests, but Schneier says the systems are “kind of mediocre at everything.” He’s not sure the offerings will improve much and wonders if jurisdictions may want to consider moving away from proctored remote bar exams during the pandemic, and instead, replace them with take-home tests.
“How do we define success? If it’s online proctoring and cheating, and I don’t detect you cheating, it’s a success, right? This is hard,” Shneier says.
Sarab has some ideas for alternatives to a remotely proctored bar exam. He’s suggested unproctored remote exams, similar to take-home tests, and in-person exams given in hotel rooms, which he says could be “exam pods.”
The latter, he adds, would allow for a Uniform Bar Exam, with a portable score as well as transferable multistate bar exam scores. Texas is planning to administer a September in-person bar exam at four different Hilton hotels in the state, and Sarab says he’s spoken with other jurisdictions about using hotel rooms for the February 2021 bar exams.
“To take the exam, people need a desk, an Internet connection and a place to plug in a laptop. Now add more space, good ventilation, their own bathroom and meals delivered. Hotels have that kind of space and those amenities, and many have a lot of empty rooms,” Sarab says.
Updated Aug. 18 at 6:03 p.m. to correct information about Texas and its September in-person exam. Updated Aug. 19 at 10:30 a.m. to provide additional information about Michigan and its online bar exam in July.
