Charting the AI Frontier

Tom Godden:
Hello. Welcome to the Executive Insights Podcast, brought to you by AWS. My name is Tom Godden. I'm a Director of Enterprise Strategy here with AWS, and today I'm joined by Matt Fitzpatrick.

Matt, thanks for joining us here today.

Matt Fitzpatrick:
Thank you for having me.

Tom Godden:
Really appreciate it. Could you give us a little bit of an introduction? Explain to us a little bit more about your role at McKinsey and please talk to me more about Quantum Black Labs. Love the name.

Matt Fitzpatrick:
So you can think of McKinsey Engineering broadly as we have a couple thousand engineers that are at client sites, building models, building technology, and Quantum Black Labs is the kind of technology development group that supports all of that.

So whether that's a retention model or a recommendation engine. Anything where you can think of it as a software interface that has a model that's leading to a decision. In a Gen AI context, that might be something like a chat-bot, that's what our Quantum Black team is doing.

And then Quantum Black Labs is all of the tools and product development that supports those forward deployed engineers.

Tom Godden:
So you're seeing a ton going on with AI right now at McKinsey. Our customers always benefit a lot from the insights. I follow McKinsey a lot to gain insights. What are you seeing, top of mind right now, in the AI space?

Matt Fitzpatrick:
Yeah, I think the last two years have been interesting. Obviously AI has become more top of mind for a lot of people. It’s on the top, it's the front page of every newspaper these days.

Tom Godden:
It's the cool kid in town.

Matt Fitzpatrick:
But I think the reality of it has been more challenging than people expected. I think that a lot of people expected it to be more like software where you'd install a new ERP system and it would work. And I think it's actually a lot more about experimentation and figuring out-

Tom Godden:
Training.

Matt Fitzpatrick:
Training. And so that process by which a lot of more traditional non-technology organizations who've only used really software and then maybe some modeling here and there have tried to pivot to being more of technology development groups. That means something very different. Nobody was going to build custom software 25 years ago with their own mainframe, it would've been pretty hard. But I think in this context, as you think about leveraging all the different technologies that you can combine to then leverage, let's say a Gen-AI model, that's a very different process than most enterprise Fortune 5000 companies that have never really built before. The data we see would say that about 92% of models these days don't make it to production.

Tom Godden:
Really?

Matt Fitzpatrick:
So only about 8% of what's being built today is being used. And I think that's been a hard adjustment process for most traditional companies.

Tom Godden:
So let's double-click on that for a minute, what are some of the hurdles? What's holding people back from turning that to 80% instead of 8%?

Matt Fitzpatrick:
Yeah, I think it's a couple different things. I think the most pronounced one is hallucinations in a Gen AI context. So machine learning, it's a little bit easier. You can say, “Here's my dependent variable, there's some level of accuracy I need” and you can get comfortable with that. In a Gen AI context, if you decide to build a model that doesn't necessarily have a clear definition, like let's say a chat-bot to have discussions about what type of coffee someone would like to order. Defining what good looks like is actually pretty hard.

Tom Godden:
Yeah. Sounds easy.

Matt Fitzpatrick:
And so I think a lot of organizations, if you think if they've got a hundred pilots going on, they're not really sure what the mark is of, “Okay, this moves to production, this works.” And so, I think that definition of saying, “The KPI I'm going to measure is this, and here I'm going to bound it and test it and make sure that I'm comfortable, that there's no risk happening.” I think that's been the biggest one. And you have seen some really high-profile public issues with that where organizations have released a chat-bot that did something pretty embarrassing. So I think that's been by far number one.

Tom Godden:
I think that's causing people to rightfully worry, but overreact a little bit. Because rightfully so, they don't want to become that company that's being talked about on social media or on the front page of the Wall Street Journal.

Matt Fitzpatrick:
Completely agree. And by the way, the interesting thing about that is there are ways to do that with fairly high levels of conviction that you have managed the risk. It means you put guardrails around things that can't say things around bias, things around toxicity. You can put guardrails and you can say, "Look, this is the outcome I'm looking for is this bound of recommendations." Or if you need to make a recommendation like suggesting the purchase of this product or a price, you can actually have that governed by a very auditable, clear, transparent machine learning model and then have the LLM be the wrapper that is the conversational kind of wrapper around that.

And in that sort of a paradigm, there's very little risk. You know what your recommendation's going to be and you know you can put guardrails around so it doesn't say anything particularly scary. But again, that's in the context of organizations that have never had to build anything like that before. And so I think that learning curve has been hard.

I think the other components of it are it requires a lot of organizational change to get comfortable with “test and learn.” I think that's the other hardest part about this. I think most organizations that build technology want to know that in six months it will be at something where they know with certainty how it works.

Tom Godden:
And this is unique because a living breathing thing, it's going to continually evolve and change as the data that's feeding it and interacting with it changes. And I think that mindset, I think you hit on it, is just something that's so foreign to so many people and rightfully so. I mean it is understandable, but then I think they freak out a little bit on it and don't know where to go.

Matt Fitzpatrick:
Completely. I think if you've operated in a paradigm of software that works or models that are highly predictable, like basic statistical models, and then you suddenly have to move to test and learn, that is not easy.

Tom Godden:
Blow your mind.

Matt Fitzpatrick:
And think about even something that let's say is going to empower someone on your front lines, a Salesforce rep or a call center rep, and the idea that person's got to make a decision around something that might not be perfect right away. Again, that's hard. And I think what has been interesting is if you get comfortable with that motion and you do the proper training and testing before you push it out live, you can get to a really good outcome, but it's just not a muscle most organizations have.

Tom Godden:
So I'm going to bring you back a little bit. We talked about a few of the challenges, we'll talk a little bit more about it, but let's talk about the opportunities. So what are you seeing companies leaning in and doing? What's some of the exciting things? And the other one, Matt, where's the easy layup? I mean, give me the win. What's the win?

Matt Fitzpatrick:
I think we're going to look back in five to 10 years and say that the first two years were much harder than people expected, but the change in 10 years from now is going to be much bigger than people expect in a lot of ways. And I think if you look across a couple different dimensions, coding, software development, I think-

Tom Godden:
Such a great use case to go after.

Matt Fitzpatrick:
Yeah, I mean think about, let's say five years ago, eight years ago, if you wanted to start a company and build an app, you had to find somebody to build you a web page, which was actually painful to do. The idea of text to HTML, text to SQL, like all of these things are going to dramatically democratize access to new technology development.

And I think that will be a really positive change for society, for anyone who wants to start a company. But it'll also make engineers much more effective. I mean, we already see many of our engineers already using a lot of Gen AI when they're developing it. It is more efficient in a lot of ways than just a code repository that you have to search through. And so I think we're already seeing we're in inning three of software development improvement. But I think that's a huge...That'll probably be the area in my opinion, that has the biggest impact over the next couple of years.

Tom Godden:
One of the things I like about that is not only the immediate value that you're going to get for it, but it gets the mental gears going for the developers. And start as they use generative AI for coding assistant to go, "Well, I wonder if I could also use that in customer service in this kind of way." And it also shows that we're talking about augmenting individuals not replacing them. And so I think embodying all those things and you get the coding benefit and value of it. I mean, I'm like, “Oh my gosh, go!”

Matt Fitzpatrick:
A hundred percent. Well also, think about the several trillion of legacy software that exists across the world. And think about the pain for those organizations if you have an old system that's 20 years old, written on an old legacy code base-

Tom Godden:
Could I upgrade from this version of dot-net to the next?

Matt Fitzpatrick:
And it's a horrible experience for your customers. It's a horrible experience for your employees. And so imagine-

Tom Godden:
And it's non differentiating, right? It's just work you got to get done so you can get onto the real stuff.

Matt Fitzpatrick:
And that's really, I think if you think about kind the age of digital, that's what's prevented a lot of companies from really moving into that is just these huge legacy code bases.

So I think if you can move to a world where you can modernize those more efficiently and start to allow your organization to become more digital first, that's going to have huge implications for making more old-school businesses feel a lot more like technology companies. So I do think that's going to be a real step change in a positive way for the way businesses function. Because anyone who's got a thirty-year-old system that they're trying to run their business off of knows that is an incredibly painful process to refactor.

The other area that I think is material is customer experience. So anything that's around customer outreach, call centers, outbound emails, outbound texts. Right now, that's a fairly painful process.

I mean, we've seen, for example, the experience of call centers is like the MPS scores on most call centers are not very good. Most people do not like the experience of waiting on hold for 25 minutes. So imagine a world where rather than a routed directional flow that says you can only talk about billing or service, you can have a conversation and sort your issue. That's going to be, again, you're going to take the amount of time you spend on hold at call center is down very materially.

And your ability to outreach to your customers to talk to them, that will be a really big change. Right now if you send an outbound email, most of the time it's a form letter about an event. But imagine if that outbound email knows things about the person's account, it can really give them a customized offer. I think that's a really material one.

The other one that I think is very material is what I would call knowledge management. So think of any organization that has enormous amounts of data, whether that be, let's say a claims processing business or a auto repair business, but it has lots of information that sits across it, that's really not organized or structured in any way. And so a lot of the time the wheel is reinvented every time anything's done. And there's no institutional memory I guess I would say. I think that's one we see a lot of interest in that will be very transformative.

Tom Godden:
There's a great example on the AWS website about a company —elevators, escalators, that type of thing — and they've built a great AI solution on top of that on AWS to go back into the history of all of the service records that have gone on. So as you're there trying to repair this escalator or this elevator at a place, instead of reinventing the wheel, why can't you go back into the history of all the service calls that have ever existed and say what's the way to be able to triage that? And that's just one example and I think there's so many to leverage for that.

Matt Fitzpatrick:
I think service calls is one of the best examples of this. I mean, if you think about how almost every organization on earth functions now with service calls, unless they have a really sophisticated call center system, they have an individual go make a decision, make an assessment, which may or may not be right, and then that is not stored or used in any way. And so we all as a society deal with a lot more pain around incorrect diagnoses at any point of service. And I think that will be a very positive thing as well.

Tom Godden:
So we lived through in 2023 and 2024 a little bit maybe of the hype of generative AI, the excitement, but a little bit of hype that came with it. I believe, and I think we believe here strongly at AWS that over the fullness of time we're going to see really almost every application augmented by AI and by generative AI. But given that as a backdrop, how do we still help leaders rationalize as they bring forward business cases on generative AI so they don't oversell it? So that you find that correct value. How do you advise people to move forward in that?

Matt Fitzpatrick:
So this goes a little bit to why only 8% are making it right now. And I do think the uncertainty of success is a tricky part in the business case development process here. What I mean by that is let's say that you want to install a new software system to do any process like expenses right? Right now you would build a business case and you'd know exactly the workflow it's going to do and you'd have very high confidence. And so it’s a very simple process to build a business case for that. But imagine your business case rests around 12 different things where you could use your organization-

Tom Godden:
Solve for N, yeah.

Matt Fitzpatrick:
Where five of them might work and seven of them might not work. And I think you actually have to change the way business case development works to look a lot more like venture capital. I don't think that means that only one in ten work. I think it means that you've got to be comfortable experimenting, trying 10, 12, 14 different things and over time you'll get in the first batch you'll get five that work. And the next batch you'll get 10 that work. You also won't be investing huge amounts to develop each one.

The way I think of it is, take knowledge management, the same data you use for that service call as an example, you can probably use for contact centers, you can probably use for document production about how the call went, all this kind of stuff. And so you'll end up with 5, 6, 7 use cases linked to the one you built that worked.

Tom Godden:
I love that. And I advise people all the time on it, get that document summary use case maybe to work, but then use it in HR, use it in finance. You're going to have to tweak it and train the model a little bit, but you're 80% or 90% of the way there.

Matt Fitzpatrick:
This has been the hard thing about business cases. You're not taking a paradigm where it's going to take you two years and it's one monolithic business case that at the end it's done. You're actually more of a paradigm of “I need to get four different data components for this use case. All of those data components are modular, I can use them for four other use cases. And so actually my business case development is, ‘does this seem worthwhile, where I'm going to build enough capabilities and I can justify getting started because it'll be useful for things repeatedly?’” And over time on a three, five year basis, you're going to make multiples of your investment. It just may not be on the first one.

Tom Godden:
So Matt, let's pretend that I'm a CIO, as a former CIO, so not a hard thing to pretend. And let's say I'm looking at a circumstance where I want something to be very unique, customized to me, but I also want it to be inexpensive, right? Isn't that always the paradigm? How is McKinsey, how are you approaching advising people build versus buy? When you build it, you get to customize it. It costs a lot more. When you buy it, theoretically less expensive, less customized. I as CIO, I'm stuck in the middle now. Help me out. What advice do you give?

Matt Fitzpatrick:
Yeah, you know, I have I have this view that actually the definition of build versus buy has become very skewed in how we think about it as any technology organization. Here's what I mean by that. 10 years ago, the definition of build versus buy meant, buy: I take something off the shelf and it works and it cost me a certain amount. And in build case, to build, I would literally have to stand up a mainframe computer. I have to build all of my code largely from scratch, often in, maybe this is 15 years ago, but-

Tom Godden:
Build it all.

Matt Fitzpatrick:
You are really building something from scratch and your investment is going to be massive.

Tom Godden:
We appreciate you saying that because that's a little bit AWS's strategy. Help take that undifferentiated lifting.

Matt Fitzpatrick:
Well if I take a broad view, not with any particular technology provider, today, “building” means I spin up a cloud instance. I use various modular components. I pull GitHub code, code repositories with information. I pull from six or seven different off-the-shelf components that allow me to stand something up for a fraction of the cost that we used 10 years ago that's really useful and customized to my organization.

And so three years ago I was working with a player that was debating buying an off-the-shelf, it was a large asset manager and they needed to rebuild their credit system. And their debate was do I buy an off-the-shelf credit platform or do I build one? And no, this is not a technology company, 10 years ago the idea of building a credit platform would've seemed absolutely insane.

Tom Godden:
Alarm bells are going off.

Matt Fitzpatrick:
But when they ran through it, what they ended up realizing was, okay to buy this, it's going to take me a very material investment to customize the data schema to match to the off-the-shelf credit platform. And then I'm going to need screens that look like...They had a homegrown system they were trying to replace when they did this, it was going to take a ton of investment to customize the off-shelf system to look like what they wanted. It was going to take a ton of investment to map the data to it. And so the time they were done-

Tom Godden:
What's the end game?

Matt Fitzpatrick:
They were basically building a new system on this kind of off-the-shelf system. So their other option was take all the modern tooling that exists, modern data tooling, cloud infrastructure, all that, and just stand up a system. And that actually ended up crazy enough not being more expensive than using it.

And we're seeing that more and more. I think Gen AI is going to really accelerate that because one of the things I've loved about the Gen AI ecosystem is how interoperable all the different applications are. Nobody is building this to say, "You can only use ours." Everyone is saying you can participate in best of breed. And so any new tech comes out you're going to be able to participate in it. And so the question then becomes if you create a new credit application today and you buy something and then some new interesting Gen AI tool comes along, you can't use that. You've actually got more tech debt by using the ten-year-old off-the-shelf credit platform than you would if you built a modern Interoperable evergreen stack and allow you to use all the modern components.

I will say the number of companies that have gotten comfortable with building, as you might call it today is way higher than it was five years ago. And it's all the investment cloud and infrastructure allows it to be so much faster.

Tom Godden:
So Matt, you've done a lot of transformations, you've led a lot, you've seen a lot. McKinsey talks a lot about rewiring organizations to be able to do this. What are you seeing is successful models for how people are approaching that cultural aspect to help succeed as we move into this generative AI?

Matt Fitzpatrick:
I think a couple of different things have been key for that. One is being very clear on what use cases actually matter and can move the needle. I think again, if you take a 10 year ago view where you might have your tech organization not really talking to your business organization, that is surefire failure. You need a clear view of what's my vision? What are the 10 use cases I'm going to try out, how are my tech teams and business teams going to work together to test and learn? That whole process is a new muscle. I think the skill set we're seeing the most interest in these days is what you might call a translator skill set or somebody who's kind of digitally knowledgeable but also understands the business. I've worked with a bunch of real estate clients, for example, and somebody who understands both tech and the real estate industry is way more valuable than someone who understands one or the other.

I think you need to have the link between the tech organization and the business teams so that what's being built is workable and relates to what the business users want. So I think the translator skills gets really important.

I also do think you have to think a lot about re-skilling, or at least the technical skill set of your engineering organization. Like if you have an engineering organization that doesn't know how to use Python or even things like Rust that are a little bit newer, it's going to be harder for you to take advantage of a lot of the modern Gen AI tooling. And so what that may lead is kind of retraining, re-skilling, things like that or new hires, but you're going to have to augment your traditional engineering organization.

Tom Godden:
So you talked a little bit earlier, about 10 years from now we're going to look back on these first two years a little bit harder. Where are we in 10 years? Where are we going?

Matt Fitzpatrick:
Here's a positive view that I think there's obviously been a lot of controversy in some ways around what it could look like in 10 years, but I'll give you the positive spin.

So we all spend a ton of time right now on our phones. If you think about the amount of time you spend on your phones and then the prevalence of reports and documents that everyone spends their life on, it's enormous, right? Everyone has a thousand customer reports, spreadsheets, all this kind of stuff paired with one to two hours of screen time per day usually looking down and not at the real world. Now imagine a world where that gets replaced. And by the way, a lot of that relates to the prevalence of different apps or tools you use to manage flight bookings or any different component of your life. You have a different app to do it.

Now imagine a world where you don't need a thousand apps, spreadsheets. You have let's say some sort of glasses or things like that where you're actually in the real world walking around, you start to see alerts on those glasses. By the way, these already exist, so I'm not actually saying anything that revolutionary, but then you're paired that with the virtual assistant in your ear. And now let's say you're a CEO in that world, rather than walking through 17 different reports every day that show different things and all that, you're just asking your assistant-

Tom Godden:
Summarize it. Net it out for me.

Matt Fitzpatrick:
You're saying, "Show me my sales in Europe and how they split by customer and these five cuts." And you see that inside of your glasses and then you walk by and you say, "Oh, that's an interesting ad. Who made that?" You can actually start to interact with the world. Science fiction is often the best predictor of that and there various indications of that you see in the science fiction world-

Tom Godden:
That or the Simpson's right?

Matt Fitzpatrick:
Exactly. Exactly. But in science fiction, you have various kind of librarians or characters that you interact with that actually just start to tell you the information you need whenever you need it. I think that's the optimistic view where this actually gives all of us a lot more times in our day. Imagine if you take all the time you spend sitting in traffic right now and you're sitting in a driverless car, operating in a world where you can ask for any information you want. That creates tons of opportunities to start new businesses, to build new products and I think that's going to be really exciting.

Tom Godden:
So in closing here, what's your one piece of advice for someone about to start on this journey? You have a lot of experience in this. What's the one takeaway you give them?

Matt Fitzpatrick:
I do have one really specific piece of advice. So there's an interesting study I'd read recently that with the advent of Gen AI, you actually have seen decreased interest in computer science. You've seen an interesting spike where basically in 2006 onward computer science really rose. And then there's been kind of pessimism that there'll be less interest in that because Gen AI is going to solve this all. I think that is the furthest from the truth. I actually think an engineering mind will find hundreds of ways to take advantage of this. And so my advice to anyone-

Tom Godden:
My wife will be happy about that.

Matt Fitzpatrick:
Well, my advice for anyone thinking about this is, think about ways to study and learn more about this.

And so whether that means you've been working for 20 years and you're thinking about going back to some advanced learning at a more advanced age or you're in college, in all of those cases, I think an engineering skill set becomes a feature of everything. And your ability to really create things from that, even if you're not always the one writing hands-on keys, coding, is going to massively evolve over time. So that would be my main comment.

Tom Godden:
I love doing these because I always learned something new. I learned a lot from you here today. Matt, thank you so much for joining me. Appreciate it.

Matt Fitzpatrick:
Thank you Tom for having me.

Listen now

Listen now

Listen now