AI software is more accurate, faster than attorneys when assessing NDAs
Shutterstock.com.
In a new AI-versus-lawyer matchup, the lawyers were bested when trying to accurately spot issues in non-disclosure agreements.
Tasked with spotting issues in five real NDAs from companies including Cargill and Pacific Gas & Electric Co., the software from LawGeex outperformed the attorneys with an average of 94 percent accuracy, according to their study. The attorneys had an average accuracy rate of 85 percent.
The attorneys assessed all five NDAs in an average of 92 minutes, while the software took only 26 seconds.
“Participating in this experiment really opened my eyes to how ridiculous it is for attorneys to spend their time (as well as their clients’ money) creating or reviewing documents like NDAs which are so fundamentally similar to one another,” Grant Gulovsen, a participating attorney, said in a statement. “Having a tool that could automate this process would free up skilled attorneys to spend their time on higher-level tasks without having to hire paralegal support.”
The competition pitted software from LawGeex, a contract review software company, against 20 U.S.-trained attorneys. The software and the attorneys were asked to issue-spot topics like confidentiality, indemnification and various exclusion clauses.
This event looked to build on previous lawyer-versus-machine competitions.
CaseCrunch’s competition last October in London featured over 100 lawyers pitted against their software to predict the outcomes of historic insurance misselling claims. The software outperformed the lawyers, however, the competition was criticized for giving the computer the complete dataset and using attorneys without subject matter experience, creating an unfair advantage.
LawGeex hoped to avoid similar criticisms. In the report, the company states that the structure of the competition “ensured significant expertise of the lawyers in the exact area of law it was assessing.” Further, the software was not given access to the NDAs used in the competition beforehand.
LawGeex’s AI was created through a combination of supervised and unsupervised learning, explained Yonatan Aumann, professor of computer science at Bar-Ilan University in Israel and LawGeex advisor, in the report. This meant that the AI algorithm was trained on numerous legal documents to learn legal terms and phrases, and then people helped the software recognize concepts in those terms. Using sentiment analysis, the software is able to understand concepts even if it encounters new words.
While the takeaway of this research could be that the robots are coming for attorneys’ jobs, the legal advisors on this project say AI is complimentary to a lawyer’s existing work.
“Not only should use of the AI provide consistency and predictability in a client’s contracts, thus providing better client protection,” said Erika Buell, director of the Program in Law & Entrepreneurship at Duke Law, in the report, “but it also should allow lawyers to focus on the highest and best use of their time.”
The report ends on a note of caution regarding AI’s current place in the legal field.
“The reality of this powerful technology is that it is not meant to be, nor indeed is it currently capable of being, used as a standalone tool,” the report reads. “Lawyers must play a vital role in strategic legal work for the foreseeable future, and use technology to become even more competitive and impactful.”
