‘Man versus Machine’ contests are always popular. We have already seen such contest where Artificial Intelligence (AI) systems defeat human experts at chess, Jeopardy, and Go. And in recent months, robot lawyers have been giving human lawyers a run for their money, too. In the headlines this week is a story of an Artificial Intelligence system outperforming human lawyers in reviewing non-disclosure agreements. Some months ago, in October 2017, there was a similar story about an AI system that was better at predicting the outcome of cases about Payment Protection Insurance than human lawyers were. Let’s start with the latter.

Case Cruncher Alpha

One of the first occasions where robot lawyers beat their human counterparts happened in October 2017, in the UK. The Artificial Intelligence system taking on the human lawyers was Case Cruncher Alpha, developed by Casecrunch. Casecrunch is the current name for the company that previously had developed LawBot, Lawbot-X, and Divorcebot. It was started by a group of Cambridge University Law students. (See the article on Legal Chatbots). Case Cruncher Alpha is designed to study existing legal cases in order to predict the outcome of similar cases.

In the contest, CaseCruncher Alpha competed with 112 lawyers. All of them were commercial lawyers from London. None of them were experts in the specific matter the cases dealt with.

The purpose of the contest was to predict the outcome (success or failure) of cases dealing with Payment Protection Insurance (PPI). The topic is well known in the UK, where several banks were ordered to pay damages to customers to whom they had sold insurance products that they didn’t require. The contestants were given real cases that had been handled by the UK Financial Ombudsman Service. The lawyers were permitted to use all available resources.

The result was a staggering defeat for the lawyers: they achieved an average accuracy score of 62%, while Case Cruncher Alpha scored 86.6 %. (No data were provided on how individual lawyers did, or on whether there were any that had scored better than the AI system).

Richard Tromans from artificiallawyer.com rightfully points out that evaluating the results of this contest is tricky. They do not mean that machines are generally better at predicting outcomes than lawyers. What they do show is that if the question is defined precisely enough, machines can compete with, and sometimes outperform human lawyers. At the same time, this experiment also suggests that there may be factors other than legal factors that contributed to the outcome of the cases. But what the contest undoubtedly made clear was that legal prediction systems can solve legal bottlenecks.

Ian Dodd was one of the judges. He believes AI may replace some of the grunt work done by junior lawyers and paralegals, but that no machine can talk to a client or argue in front of a High Court judge. In his view, “The knowledge jobs will go, the wisdom jobs will stay.”

LawGeex

Another occasion where robot lawyers got the upper hand was in a contest in February 2018, in the US. In this case, the Artificial Intelligence system was developed by LawGeex, the company that was started in 2014 by commercial lawyer Noory Bechor (cf. the previous article on AI and contracts). Noory Bechor had come to the realization that 80% of his work consisted of reviewing contracts and was highly repetitive. He believed it could be done faster, cheaper and more effectively by a computer, and that’s why he started LawGeex.

In this challenge, the AI system competed with 20 lawyers in reviewing Non-Disclosure Agreements. All 20 lawyers were experts in the field. The LawGeex report explains: “The study asked each lawyer to annotative five NDAs according to a set of Clause Definitions. Each lawyer was given four hours to find the relevant issues in all five NDAs.” In that time, they had to identify 30 legal issues, including arbitration, confidentiality of relationship, and indemnifications. They were given scores reflecting how accurately they identified each issue.

Once again, the AI system did better than the human lawyers. It achieved an accuracy rate of 94%, whereas the lawyers achieved an average of 85%. There were, however, considerable differences between how well the lawyers did. The lowest performing lawyer, e.g., only scored 67%, while the two best performing lawyers beat the AI system and achieved, depending on the source, either 94 or 95% accuracy. For one specific NDA, one lawyer only identified 55% of the relevant issues, while for another one the AI system reached a score of 100%, where the best human lawyers reached 97% for that one.

The human lawyers were no competition for the AI system when it came to speed. The AI system cleared the job in 26 seconds, while the lawyers took 92 minutes on average. The longest time one lawyer took, was 151 minutes, while the shortest time was 51 minutes.

Gillian K. Hadfield is a Professor of Law and Economics at the University of Southern California who advised on the test. She says, “This research shows technology can help solve two problems: both making contracts faster and more reliable, and freeing up resources so legal departments can focus on building the quality of their human legal teams.” In other words, the use of AI can actually help lawyers expedite their work, and free them up to focus on tasks that still require human skills.

Sources:

• bbc.com/news/technology-41829534
• artificiallawyer.com/2017/10/28/ai-beats-human-lawyers-in-casecrunch-prediction-showdown/
• com/2018/02/26/ai-beats-humans-at-contracts/
• legalfutures.co.uk/latest-news/ai-beats-average-legal-mind-not-best-performing-lawyers
• artificiallawyer.com/2018/02/26/lawgeex-hits-94-accuracy-in-nda-review-vs-85-for-human-lawyers/