NCBE runs remotely proctored test to audition NextGen UBE questions

The National Conference of Bar Examiners is using proctored remote testing to test-drive potential questions for upcoming administrations of the NextGen Uniform Bar Examination, including some questions generated with support from artificial intelligence.
In late August, 56 people completed a full section of NextGen content from individual locations using ProctorNow software through Internet Testing Systems, which also handles the exam’s delivery and scoring platform, says Kara Smith, the chief product officer for the NCBE. The three-hour test included 40 multiple-choice questions, two integrated question sets and one performance task—with AI-assisted monitoring backed by human oversight and review.
The pretest is part of the NCBE’s quality control.
“We require a certain number of individuals to pretest items so we can measure the psychometric properties” before using them on a bar exam, Smith says.
Questions undergo years of review and multiple testing opportunities.
Despite the trial and plans for more pretesting in early 2027, the NCBE is not considering remote proctoring for the NextGen exam, a legacy exam or any other bar exam, Smith says. Nor does it plan to put AI-supported questions on operational forms anytime soon.
Instead, the pilot sought to determine whether the technology could prevent potential questions from leaking, help build a bank of clear and accurate items that meet psychometric standards, and provide a workable experience for candidates.
Kara Smith is the chief product officer for the National Conference of Bar Examiners. (Photo courtesy of the National Conference of Bar Examiners)
Tech test
With the pilot, the NCBE was more concerned about content theft than cheating.
“Our biggest concern is if somebody’s going to screenshot the question or take a picture of the question, put it on Reddit,” Smith says.
No such breach happened, she adds.
To create a secure remote environment, examinees sat in a room of their choosing, entered a registration ID, then checked audio and two cameras, says Pat Ward, the president of Internet Testing Systems.
“While you’re taking the test, these cameras are watching you. They’re listening to you. They’re transcribing what you say,” Ward says. “They’re making sure that you’re not reading questions out loud,” which would constitute a security breech.
If ProctorNow’s AI detected a potential infraction, such as an examinee wearing headphones during the test, it alerted a trained human proctor who reviewed the video and made the call.
“You need humans in the loop,” Ward says.
Without telling Internet Testing Systems, an NCBE employee deliberately displayed prohibited behaviors, including eye and head movements that could be construed as cheating, to see whether the system responded. Smith says the reactions were appropriate and showed that an AI-enabled remote pilot could scale.
The pilot also underscored the drawbacks of high-stakes remote testing. Rigid physical restrictions created enough anxiety to affect performance, Smith says, reinforcing research suggesting that the technology is not ready for high-stakes assessments.
Questioning the questions
Pilot participants had taken the bar exam but were not scheduled for either of the next two NextGen UBE administrations. They evaluated whether questions were clear and accurate enough to use on an actual exam, helping the NCBE build a bank of “bullet-proof” questions.
The need for questions is great. Each NextGen UBE test contains 120 multiple-choice questions, six integrated question sets and three performance tasks. Each administration has a unique set of questions.
In the past, candidates who took the legacy bar exam, which will sunset in 2028, would answer experimental or piloted questions that would be randomly sprinkled through the test and would not be scored. Instead, these questions would be evaluated for possible useon future tests.
But that no longer works. The new exam, more focused on skills than memorization, has questions with different formats than those in the Multistate Bar Examination, forcing the audition process to change, Smith says.
Also, as the NextGen exam rolls out, the number of candidates taking the exam is much smaller than the number taking the legacy exam, meaning that there are not enough responses to the trial questions to fully vet the new questions.
Although the NCBE does not plan to use AI-supported questions on actual tests, remote pilots offer a low-risk way to evaluate them. Leaked content can be discarded; items that remain secure and perform well can enter the usual process of human review, editing and revision, Smith says.
Examinees are told that some questions have AI origins. Two days after the August exam, they were surveyed and asked whether they could pick out those AI questions, as well as to provide feedback for the test designers, Smith says.
Did the pilot fly?
Remote testing also allows flexible nationwide recruitment, including more candidates from racial groups that historically have not performed as well on the bar exam. That broader pool can help the NCBE investigate score gaps and assess whether questions are equitable across populations, Smith says.
“You’ll historically see on the bar exam that there is a gap between performance of individuals who report being African American and performance of individuals who report being white,” she adds, allowing the NCBE to “really try to dig into the why and how we can make sure our questions are equitable for all populations.”
Although August was a pilot, the organization ultimately wants remote pretests that candidates can take at any time.
Traditionally, the bar exam is given twice per year, but adding more administrations is being considered, she says, creating a need for even more questions.
“If we had four administrations, we would need four forms. If we had 12 administrations, we would need 12,” Smith adds.
More remotely proctored pretests will help fill that pipeline with auditioned questions.
“Remote proctoring is a reliable way to pilot test early-stage items in development,” Smith says. “So now, it’s really about making sure the operation can maintain, and we can continue to improve.”
Write a letter to the editor, share a story tip or update, or report an error.

