Ari Kaplan recently spoke with Nick West, a partner and chief strategy officer at Mishcon de Reya.

They discussed the law firm’s approach to innovation; the metrics it captures related to its use of Legora; how it measures the return on investment of Legora and other tools; how law firms should think about the return on their technology investments; and the new Legora ROI report, titled Measuring the impact of AI on law firms.

Ari Kaplan: Tell us about your background and your role at Mishcon.

Nick West: I’ve been in law for 25 years. I originally trained as a lawyer at Linklaters and have held a number of roles across legal services-related businesses, including LexisNexis, Axiom and, most recently, Mishcon. My role sits at the intersection of macro strategy, technology and innovation, really shaping the future of law. It’s a thrilling ride.

Ari Kaplan: How would you describe Mishcon’s approach to innovation?

Nick West: We’ve always wanted to be at the forefront of innovation in law. We are a business that’s more of a challenger than a follower. It starts with technology changing the world, so it’s going to change the legal landscape. The best way to understand it is to get your hands on it, to play with it, to experiment with it. We built MDR Lab, the first industry legal tech incubator. It has spawned a number of well-known legal tech companies, including Laurel, DraftWise, Orbital and Thirdfort. We helped those companies get started, understand product-market fit, and clarify the problem they were trying to solve. That’s always been our approach. We have had a data science team since 2017. We’ve always wanted to get in early, and sometimes that means making a few false moves, but you learn from them.

Ari Kaplan: You announced a firmwide rollout of Legora in July of last year. How has it changed how your professionals practice?

Nick West: It’s changing very quickly. We have associate daily active use up in the late 60s to 70%. That’s a pretty astonishing number for legal technology, and people are clearly getting value from it. We are now clear that doing a task this way, using a Legora workflow, a Legora prompt or chaining Legora tasks together, is better than doing it in a non-AI-enabled way. It’s changing because people want to use it, and we are making people use it because we can get high-quality outcomes faster and do more. Therefore, it’s definitely changing the way law is practiced.

Ari Kaplan: What metrics do you capture related to the use of Legora and your other legal technology applications?

Nick West: We look closely at usage and adoption. We always have across different tools because ultimately that is one component of the ROI question. Part of it is adoption, but we also consider the impact of usage on lawyer utilization, throughput and the speed at which work gets done. We look at usage and adoption across different levels of the firm, lawyer types and departments. We are beginning to be quite methodical in how we track ROI, but it’s early. Adoption isn’t ROI, and ROI is hard, even in some cases, to describe what we mean by ROI, let alone track it.

Ari Kaplan: I was fortunate to interview law firm leaders in 14 countries to start the conversation about the ROI of Legora and technology and to try to develop some metrics. Is there any way you measure the ROI of Legora or other tools at this point?

Nick West is a partner and chief strategy officer at Mishcon de Reya, a law firm in London.

Nick West: We have to define ROI. We have to be able to put a number on the value created, and some of those numbers are easier than others. In a fixed-fee matter, the value to a client is probably about increased speed. Maybe we can increase quality, but let’s hope we were giving great quality anyway. Value might be that we can do something faster or do more for the fixed fee. Value for us is happy clients but also probably a higher margin. If we can price it at a hundred, and it used to cost us 50 of input costs to deliver a hundred, we might make 50, making the math simple. If we can do it for 30 and make 70, that’s clearly beneficial for us. The obvious question is whether we should be sharing that upside with the client. How does the price go? I don’t know the answer to that question at the moment. So I think about all of those components. Does it increase the speed of delivery? Does it allow us to do more? Do we make more margin on matters? That’s definitely one basket of ROI tracking that we do on fixed-fee matters. When we think about time and materials, value to the client is trying to fix and make more certain the cost of something. Ultimately, no one wants an open-ended check, but it’s very difficult to say this piece of litigation will cost X. There are so many different paths that the litigation can take. So can we make certain elements more certain because we have AI tools and therefore be more certain about how much time it will take or how we can reduce it? Are there things we can now do that we wouldn’t have been able to do before to increase quality, and can we make our clients happier with the work we do? The value to us, other than happier clients and more work, is more complicated than with a fixed fee, obviously, because if we’re compressing the time taken, in theory, we’re not able to charge as much as we did. So you really think about focusing that work on the tasks we do that we find very difficult to bill for, that clients won’t pay for or won’t pay fully for. How do I really focus these tools around that stuff, so that I can improve the margin on that work? Some of those things are easier to track, and others are harder. Margin on a fixed-fee matter is very easy to track. The quality of a piece of litigation and the service we deliver to a client is much harder to track, other than asking clients how they feel about the work. These are hard to do. So I don’t have answers to all the questions, but I’m determined that over the next year or two, we build much better ways of measuring that stuff and linking together time recording data, client satisfaction data and tool usage data and trying to be able to sit at the board and say, of course, we track revenue and we track profit, and we track hours, but I’m tracking the ROI on some of these investments, and I report on them. That’s the aspiration.

Ari Kaplan: Many of the leaders I spoke with revealed that data has a hierarchy, with some types easier to capture than others. Are you seeing interdisciplinary data sharing to develop a further understanding of ROI?

Nick West: We have always thought carefully about how some of our data scientists use multiple data sources, but there are gaps in the data available. If you’re someone who uses very sophisticated AI timekeeping tools, you get better-quality data about who is doing what with which tools. Some provide better telemetry about what people are doing on their platform, while others provide less information. You can choose to buy some of those tools, and if you do, you have better data and can generate more insight. If you choose not to buy those tools, you have to do a lot more of the joining of the dots yourself, which is harder. If you have time data without exact timestamps, it’s really hard to match it to a tool’s usage data. But equally, if the tool’s usage data lacks the fidelity to show exactly which workflow the lawyer used at this point, you are also struggling. If you’re serious about this stuff, you have to push providers hard. You have to be very demanding. And then you have to start thinking a little bit about your own. There’s a bunch of data in your own enterprise that you have to work quite hard to make sense of. Some individuals are working 10 hours a day on the same task, while others are constantly context-switching between many tasks. You have to figure out their work, how they context-switch, and how to allocate ROI to individual matters. That is clearly a much harder task, and you have to be well organized in how you look at the data. Interestingly, those people are quite attractive to follow because we’re seeing early evidence that people whose practice looks like that can actually work across more matters when they use AI. So if somebody’s working on two or three things at once, I’m starting to see them able to work on four or five at once because each component of their day takes a little less time, giving them some extra capacity to take on more work. That comes before you even start worrying about tool usage. We just try to spot those trends, which is a form of ROI for us anyway. We’ve got more capacity in the organization. We’ve got to win more work, but we can do more work with the same number of people. So you have to think quite hard about what you’re trying to measure and how you join up different datasets. Like any data problem, you’ve got to be able to triangulate data correctly. ROI measurement is a nontrivial task, but I think it’s worth working hard at it to ensure you can get real insight and report on it. I’m not one of those people who think it’s a bit like Microsoft Word. We don’t measure the ROI of that. That’s nonsense. If we are investing large six- or seven-figure sums in these tools, you have to be able to turn round and say, “This is what it does for our business.” So I think we have to work quite hard to join all those datasets. In some cases, you can make it easier for yourself by choosing the tools that you use. In other cases, you just have to work hard to make sense of the data you already have.

Ari Kaplan: How long do you think it will be before law firms are able to capture the metrics necessary to have a more meaningful conversation about the return on their investments?

Nick West: Until I have incredibly granular data on what users are doing with these tools every minute they’re in them, my ability to translate that into hard ROI is limited. Equally, I’ve got to work hard to get all users working in a particular way because one of the challenges of measuring ROI is that you might be looking at a sample size that’s too small. You’re just looking at edge cases, not the whole. Like any data task, you’ve really got to think about the sample you’re taking and the statistical sample you’re taking. It takes a while. And whilst we’re still very much in the adoption curve, some of the data is hard to believe until you reach a certain usage level. It’s very complicated and very messy. We’re in a better position now than we were three months ago. That’s the way I think about a lot of this stuff. When things are hard, maybe I can’t get there, but I can get here, so let’s just go here. And then when I’m here, the world looks a bit different. So now what do I need to do? And I realize that I’m trying to do too much, so I narrow it down and focus. That’s the way I think about innovation, experimentation and R&D. Here, we are saying, “I’d love to be able to put a single ROI number on my spend with Legora, but maybe that’s very difficult.” So what I can do is say X. What I can do is say Y. What I can do is show how, on work that we do in real estate, which is on a fixed fee, this is the impact of it. The work we do in our litigation or arbitration practice is typically on a time-and-materials basis, so this is the value we can create. This is how using Legora Portal and engaging clients directly, bringing them directly into Legora’s portal tool, is generating new work, opportunities and revenue for us. All of those things, I’ve taken the big ROI idea down and broken it into smaller chunks. In some of those, we are doing well on reporting, and in others, I’ve got a long way to go, often because I haven’t got enough usage to really report on.

Ari Kaplan: How should law firms generally approach the return on their investments in technology?

Nick West: They should be pretty disciplined about it. It’s an increasing line item on a budget. It doesn’t make it easy to track. It’s quite hard to translate all of those dollars back into a single ROI number. It’s quite hard to express the total value of an individual lawyer. You can see their hours billed, but what about all the other things? So it’s the same with technology. A lot of technology enables the practice of law, but I think we should try hard to do so. I don’t think it’s acceptable to turn around and say, “We don’t measure the value of email or Word, so why should we measure the value of Legora?” I think it has to be: Is it improving quality? Is it improving speed? Is it improving the margin? Is it generating new revenue through things you couldn’t do or services you can now build? They’re all sources of value. Some of those are easier to manage and measure. Some of those are easier to allocate direct revenue and cost to, but these are not impossible questions to answer. You just have to work quite hard at them. And I think you should because I have a responsibility to my fellow partners to be able to turn around and say, “When you give us more money to spend on more tools, I ought to be able to tell you how that’s generating return for us as a business.”

Listen to the complete interview at Reinventing Professionals.

Ari Kaplan regularly interviews leaders in the legal industry and in the broader professional services community to share perspective, highlight transformative change and introduce new technology at his blog and on Apple Podcasts.