Right-Sizing Security Across HAQM Businesses

Clarke Rodgers:
Steve, thanks so much for joining me today.

Steve Schmidt:
Happy to be here. Thanks.

Clarke Rodgers:
So it's been a while since we last talked. In fact, I was doing the math last night. It's about four years ago.

Steve Schmidt:
Wow.

Clarke Rodgers:
So when we think about what was happening four years ago, the pandemic was going on. You and I were on a remote interview and you were in a different role, right? So you were the chief information security officer at AWS. Not too long after that interview, Andy stepped up to the CEO role of HAQM, and one of the first things he did was bring you on board as the chief security officer. Can you tell me a little bit about why he did that?

Steve Schmidt:
Sure. So one of the things that Andy really values is understanding how we protect our customer's information. And that grew out of necessity in AWS where it's foundational to the business itself. You can't have a business like AWS unless you do security right.

And when he got into his new position, he wanted to take that same mindset and apply it to all of our customers across the incredible diversity of businesses that HAQM currently operates. And he also wants to know that there's somebody whose day job every single day is keeping an eye out for that kind of thing. And so he asked me to step up and take that over.

Clarke Rodgers:
So as chief security officer, there's one word missing from there, information, right? So you're a CSO. We've had a couple customer interviews where they have also quote, unquote drop the I. Can you tell me a little bit about why it's important to be the chief security officer, not call out information specifically?

Steve Schmidt:
Sure. So when you look at the protection of information, which is the real currency of our realm, it is something that certainly has the logical component that people are used to with the CISO job. But increasingly our adversaries are using humans to go after information that is harder to get to from a logical perspective.

So there's been this sort of pendulum swing in the industry, and it is long ago, if you think about it, people had spies, physical spies who would go in and grab copies of documents or things like that. When we went online with computer systems, oh heck, there's a new opportunity. Let's go break into those things remotely, et cetera. As businesses get better about securing that information, whether it's on premises or elsewhere, it turns out that people go back to being spies again.

And so we have to integrate the protection of both our physical assets and our information assets, our logical assets in order to get a full picture on what we're doing with our customer's data and how we're protecting it, and also our business data. Because quite often now, our nation-state adversaries are focused not only in getting access to the data that our customers have, but also the information about where are we taking our business next?

How are we building the next set of very interesting products or services, whether it's a satellite or a robotic vehicle. Those have incredible value to countries around the world, but also to the businesses that they support. So as security professionals, we have to look at that complete picture of who are of our adversaries and how are we going to protect against them.

Clarke Rodgers:
For sure. So as you stepped into the CSO role and now you have physical security reporting up into you, I would imagine the language that the information security professionals speak and the physical security, and the language that the physical security people speak could be on two different planes. How do you go about integrating those two so they're speaking the same language and can work effectively together?

Steve Schmidt:
Sure. So it's less about the word choice that they have because they have their own lingoes. You're absolutely right. What's most important is understanding what their end goal is, how you're going to measure your progress towards that goal and to identify most importantly, the handoff points or the gaps between the physical world and the logical world.

Think about it this way, if I'm in the physical world, I prevent people from getting into buildings where I prevent our adversaries from being hired by the company. But if an adversary does get into a building, is that interesting to the information security side? If they're standing in a lobby, that's one thing. If they're in a closet that has a bunch of network switches in it, that's something entirely different. So we have to link those two together in order to get a proper picture of what's going on.

Clarke Rodgers:
Did you have to build any tooling to sort of cross these bridges?

Steve Schmidt:
Yeah, it's both a combination of tooling and processes that we had put together. So tooling is a constant evolution. We're always getting better in theory, at finding ways to collect, analyze and use data. Where it becomes more interesting and more difficult in fact is figuring out how do you hand that off between different parts of the business and how you make sure that they've got access to the things that are necessary for ... to do their job, but don't overwhelm them with data so that they can't find the important pieces amongst the huge piles of information that we collect.

And that's where the process component comes in. It's often about the bubbling up of things that could be sensitive, that are sensitive, et cetera, and finding ways to ensure that they're transitioned from one part of the organization to another. And that really takes highly calibrated people. It takes humans who know that, okay, this may be innocuous-seeming in its own right, but when combined with this other thing that we saw, it's really interesting. And unfortunately there is no system yet that can do that. Maybe with some AI training down in the future, we'll be able to get there, but it's just not there at the moment.

Clarke Rodgers:
That's very, very interesting. You started the security program at AWS, so you know AWS backwards and forwards, how to secure it, the threats that are against it, the risk appetite, et cetera. As you moved into the CSO role, you now to learn about haqm.com or what we call internally stores, Whole Foods, Prime Video, MGM, Twitch, all of these different organizations.

First, how did you get up to speed on the security profile for each of those businesses and sort of the risk appetite and then how did you sort of bring it all together? So the common term, the single pane of glass that you felt comfortable that the risk profile for Whole Foods is appropriate, the one for AWS is also appropriate for AWS, how did you sort of figure all that out?

Steve Schmidt:
Well, first of all, one of the things that I love about my job is the diversity of the businesses. People often say you've been in your position for 16 years. That's really unusual for somebody in the security industry. Why is that? It's because of the diversity of work that this company has. It's an opportunity to continue learning, which is, I love it.

I'm not that young. People say, how long are you going to keep working? Are you going to retire, whatever? And I'm like, well, I'm enjoying myself. No, I don't want to. This is really a lot of fun. And it's because you go from building the world's largest cloud provider to putting satellites in space and running grocery stores and the diversity there is just an incredible challenge from the business perspective, but it's also an interesting opportunity to leverage the scale of the company to make things less expensive to operate.

When you look at operating security organizations, they are not cheap. But when you can scale it across a business as large as HAQM's, it means the unit cost can be driven down.

Clarke Rodgers:
For sure.

Steve Schmidt:
So every business benefits from the scale of the other businesses. And so finding ways that a grocery store can take advantage of the same security that a satellite business does, which they can in many areas, for example, vulnerability management, whether you're patched on a computer system, is not really fundamentally different when you're building satellites versus operating a grocery store.

And so that allows us to do things at a level that a standalone business really could not afford to and raise the bar for everybody. And that's part of what my job here is making sure that we have a standardized bar across the company, whether it's for vulnerability management or incident response or any of the other components that go into a typical security organization.

Then figuring out what are the bespoke components that have to be put in place for particular businesses because of their unique situations. That way we don't try and apply the sort of one size fits all to everybody because that would just drive costs through the roof. If you look at a grocery business for example, the loss of a unit there has a relatively de-minimize value, whereas the loss of a satellite is the opposite.

Clarke Rodgers:
For sure.

Steve Schmidt:
And so we have to tailor the security situation for individual components.

Clarke Rodgers:
How do you hold ... I'll back up. You have chief information security officers running the security programs for each of these other businesses. How do you hold them accountable to run their security business?

Steve Schmidt:
So one of the things that HAQM focuses on a lot across the company is the idea of a single threaded owner. So somebody whose job it is just to focus on one component of something. And in security, that's why we have a CISO for each individual business is because of two things.

One is I want somebody who every day just focuses on that, whether it's Amy Herzog who's in the devices and Kuiper space, or whether it's Chris Betz who's looking after AWS. But at the same time I use common measurements across all of those things. For example, I look at a vulnerability management monthly business review, which encompasses the entire company and it uses the same numbers, the same methodologies, the same presentation methods, et cetera.

So we get a common view that's consistent across every one of those businesses. And it allows us to do two things. One is to make sure that we're meeting the bar that we require, and the second is to ensure that we are applying the visibility that we want in every corner of the business. Because quite often where people have problems is they think, oh, this is not an important thing, this is a small piece, et cetera.

And that's where the bad guy gets in and we all get bitten. So by giving that sort of 10000 foot oversight on everything, we make sure that we're doing the things that we need to in every part of the company.

Clarke Rodgers:
And then you have centralized systems under the HAQM security or AMSEC umbrella that everyone can take advantage of.

Steve Schmidt:
So there are a lot of things that are fundamentally the same across all of our businesses. The way that you collect certain kinds of data, the way that you analyze that data or report on it, and rather than making every individual business do the same thing over and over and over, we decide to move them into one spot. That allows us to save on things like developer time.

So if you think about running a large scale, we'll go back to vulnerability management, collection engine for that you've got to have on-call engineers. If you've ever run an on-call organization, which you have, you figured to have one person on call, you've got to have approximately seven people in order to do that effectively to accommodate leave and vacations and everything else.

Clarke Rodgers:
We want people to take vacations.

Steve Schmidt:
That's right. And so by spreading that across a bunch of different businesses from one place, it means that we do so more effectively because we get better tooling centrally and at a lower cost.

Clarke Rodgers:
What practices have you either developed or followed to report the status of security across all these disparate businesses to the HAQM board?

Steve Schmidt:
The HAQM board first of all is really interesting. There are very few boards that have the technical acumen across the population that HAQM's board does, but also HAQM chose several years ago to create a security subcommittee. So unlike a lot of places where security may report to the audit committee, for example, there is a dedicated group of people whose job it is just to look at security on the HAQM board.

That's great and it also means there's a lot of scrutiny on us in the process. So we've had to build a reporting mechanism which evolves over time because of two reasons. One is because we get better at doing our job on reporting. And two is the board becomes even better informed over time. They ask more pointy questions and want to know more specific details about niches in the business. We generally find that it's important to report up to them on specific components of the business every single time we talk. Are we meeting our security bar in certain places?

Additionally, they have things that they're really interested in, tell us about this particular part of the business or this thing we're getting into we think has a lot of risks, so give us a rundown on blah, blah, blah. And that allows us to have a consistent component, a variable component that's based on their interest.

And then we decide to put in something called current events at the end where we take all of the stuff that you've seen in the news, we distill it down to things that we think have particular lessons or points for us and present that to the board as an informative component at the end. Here is something that happened out in the industry, here is why we were not affected.

Here is the investment that led to us not being affected. And I think that has tremendous value for the board. Number one, they understand that we're in a good place, but number two, it helps inform investment decisions down the road. So when we can come forward and say we invested in multi-factor authentication eight years ago, 10 years ago, and that prevented this particular threat actor who got into this other large company from affecting us, they say, okay, great. What are the other investments we should be planning now that will help us in 10 years to continue to avoid problems?

Clarke Rodgers:
So many of our CISO customers, they focus on the board a lot. Some of them don't get as much FaceTime as others. And you already called out that our board is unique because of their security savvy. What recommendations would you give to your CISO peers or CSO peers who are reporting to their boards to really help sort of drive their points home, speak the language of the board, et cetera?

Steve Schmidt:
So the number one thing that I've heard from board members about why they like the way that we do things is we are very careful to avoid jargon. A lot of people in the CISO roles are technical and they want to report on things in ways that the-

Clarke Rodgers:
The bits and the bytes.

Steve Schmidt:
Exactly, are reflective of the way they think about it. But we've got to remember the board is the customer here. So when we are presenting to them, we have to speak their language and we have to find ways to explain things in context that makes sense to that particular board.

So number one is find a consistent reporting mechanism rather than varying it each time because that makes it harder for someone to grok it, basically. Number two is figure out what are the really two or three super important metrics that you want to get across in there every single time.

Don't drown people in metrics. For example, we always, always report on vulnerability management. It is the single most important fundamental security control that we operate and I think anybody operates and then figure out what are those things that are interesting add-ons at the end, which help you develop the should we invest in, so make that intentional separation between components of the reporting process.

Clarke Rodgers:
And is if we invest in here, we reduce risk over here?

Steve Schmidt:
Yeah, it is a combination of both current risk reduction and forward-looking reduction and the forward-looking is actually probably the hardest part of our job as security professionals because we don't have an existence proof to use. And a lot of people look at it and say, is that really necessary?

Do we have to do it right now? Do we have to do it that big? Can we go smaller? Those are the arguments we all have to have and we need to be able to build up that sort of knowledge base on the board over time and say, these are the examples of the real-world exploitation of things that look kind of like this and we think it's going to apply to us in this amount of time, therefore we have to go act now or in two years or in three years or whatever.

Clarke Rodgers:
Got you. So at HAQM we're known for our innovation, working backwards, listening to our customer, et cetera. As a security leader, you have lots of choices, and this may come down to your CISOs reporting into you, you have lots of choices about what tools you purchase versus what tools you need to build. Oftentimes that comes down to scale.

So the commercial off-the-shelf software may or may not scale to HAQM's scale and you need to build something yourself. Over the last year we've talked publicly about tools like Madpot and Mithra and Sonaris. As the CSO, how do you make the pitch to get the dollars to actually put the engineering resources behind tools like that, and I imagine many other tools that we haven't talked about, how do you make the pitch to say this is a worthy of your investment and here's the value that we're going to get out of these?

Steve Schmidt:
Sure. So let's first differentiate between a purchased tool versus something we build. I think that's an important starting point. We will purchase tools when they're a commodity item. So for example, endpoint detection response, we buy that as opposed to build it ourselves. Why? Because the Mac laptops, the Windows laptops, the Linux laptops that we use are the same that lots of other people use as well.

We may have a little bit of different software on them, but it doesn't really differentiate our business. Whereas we are the only ones who can build a very large scale system like Madpot as an example. Where we can build something that nobody else can is where we tend to invest.

The way we do that investment process is just like we do a lot of other things. You prototype it, you try it, you see what works. You guarantee that you're not going to get it right the first time. So you've got to go and re-engineer something, change it a little bit, et cetera. Now Madpot has been incredibly successful but didn't just pop up overnight. It is an investment of many, many years That started off with a single engineer who said, huh, I really like this idea. Let's go see if it can get anything interesting.

And then it's turned into this engine that allows us to acquire really, really timely threat intelligence data that we can take out of that we can process and we can feed into the security tooling that all of our customers have access to. And I think that's the part that's most important. So for example, a lot of our customers say, ooh, I want raw threat intelligence feed.

And like, well actually no, you don't. What you really want is the stuff that's relevant to you in the context in which you are operating right now. The rest of it's just noise and the volume of it now is so huge that it's pointless to try and digest the whole thing unless you have a business that's like ours.

And so a lot of our customers have found that they really, really like taking things like threat intelligence and consuming it as part of a managed service. And that allows them to not have to spend their time winnowing down very large piles of data or missing context because it's not been applied across multiple customers.

And that context thing is important. That's where the centralized visibility that only we have to go back to the original part about commoditization versus custom-builds, really brings advantage.

Clarke Rodgers:
And then I guess the internal pitch, for lack of a better word, it's, this helps us secure AWS slash HAQM and adds benefit to our customers. So I mean it's a win.

Steve Schmidt:
And in typical HAQM parlance will start off with the customers first and say, this helps all of our customers better secure themselves. And at the same time, it helps us with our business because most of our businesses are HAQM AWS customers anyway. So it is beneficial across the board.

Clarke Rodgers:
That's great. Let's switch gears just a little bit. This past year has been the year of generative AI. The majority of our customers that I speak with have been focusing on securing gen AI as a tool that the quote, unquote business is using. So maybe it's a third party tool, maybe they're using HAQM Bedrock, whatever the case may be. But let's secure the tool. What are you seeing or maybe what are we doing inside of HAQM actually using generative AI as a security tool or as part of your security tooling chain?

Steve Schmidt:
So the most effective use of generative AI that we've seen so far has been in the application security process itself. So as you're familiar in AWS, every single feature app, et cetera, that goes out the door, has a security review before it's launched. Other businesses haven't had that luxury in the past because it's incredibly expensive to do that, especially when you consider that it means literally thousands of security engineers who are focused on that.

Most companies out there do not have the number of security engineers in total that we have just doing AppSec work at HAQM. And so generative AI gives us tremendous leverage in that space. It's still in the science stage I'll say, but it's got an enormous amount of promise.

We think that over time we'll probably see a reduction in human workloads in the application security space in the 60 to 70%, which means that we can do things more quickly for lower costs with better consistency in places like AWS, which has always had an investment in evaluating everything and in areas where there's an opportunity to look at more that's never been evaluated before, it means we can get greater coverage. So it's sort of a double win.

Clarke Rodgers:
So you're going to take those human hours that you save with generative AI and put them on other cybersecurity challenges.

Steve Schmidt:
Oh, it's actually going to be in looking at in more depth at services that we've got and examining more applications across the board. The other piece is generative AI is not going to be the entire solution for a lot of this. What it'll do is it'll winnow out a lot of the sort of low-hanging fruit, if you will, and allow our engineers to focus on the really, really interesting problems that only a human can go into.

People think, oh, generative AI can solve all problems, not yet. And I think anybody who tells you it can in the security space is probably not really understanding what's going on. There always needs to be human oversight in that space, at least right now. And more importantly, there has to be human judgment on whether the generative AI has made the correct decision or not.

Clarke Rodgers:
Speaking or staying with humans. What makes a great security hire at HAQM or AWS?

Steve Schmidt:
I would say the number one thing that we look for in our security hires and most people are thinking is that this particular kind of technical acumen, no, curiosity.

Clarke Rodgers:
Tell me more.

Steve Schmidt:
What it means is someone who doesn't look at a problem and say, oh, okay, this is it. They say, well, why did that happen? Well, why did it get in that place in the first place? Why didn't we detect it sooner? Why was it something that a builder could even do wrong to begin with? That is the kind of person who keeps digging that we're really, really interested in.

And you're really familiar with the correction of error process that we use, COE, and it has at the bottom what? Five whys. And it's that same thing is really get down to the root of the problem as opposed to just addressing a symptom of the problem up at the top.

Clarke Rodgers:
So curiosity is king,

Steve Schmidt:
Curiosity is king. Yeah.

Clarke Rodgers:
If I give you a crystal ball-

Steve Schmidt:
Uh-oh.

Clarke Rodgers:
Sorry. It's the question. If you look into that crystal ball, what are CSOs and CISOs going to be focused on in the next year or so?

Steve Schmidt:
Well, I think that most people are going to have to focus on generative AI because their businesses are consuming those services at an incredible clip, and they're going to have to do two things. Number one is simply understand where they're using generative AI. And I think we're in a lucky place in HAQM and we've got centralized visibility that allows us to see where all of our builders are using generative AI.

Most companies aren't that way and they've got to go figure that out. The second part is you're going to have to figure out for generative AI, how you're doing RAG and whether you're enforcing authorization and authentication appropriately throughout that process. That is not trivial. It's something where the software world doesn't even have it completely nailed down yet.

And it's something that is incredibly important to being able to use generative AI in a way that presents only back the data as a result of the prompt that that person on the other end is authorized to have access to at that point in time from the place that they're currently sitting in on the piece of equipment that they're currently using. Most people have not even thought about that problem yet.

But in reality, if you look at our internal systems at HAQM right now, when you're on your laptop, we measure a lot of things about you when you're using our internal systems. Literally, is your laptop in the state we expect it in? Have you patched, et cetera?

We've all had the little notices that pop up that say you're going to be quarantined if you don't patch. And where are you in the world? What's your job? And things that most people don't even realize we're looking at like, what day of the week is it? And is this an hour that you normally access or are you coming from an unusual place? And those sorts of things.

Clarke Rodgers:
Is this normal for Clarke?

Steve Schmidt:
That's right. Is this normal for Clarke and is this normal for a person in Clarke's job? And those controls simply don't exist in most places. That's why implementation of guardrails is so important. I don't care what system you use to do it, you just have to have guardrails in place.

And those guardrails have to constantly evolve as the threat landscape evolves around you, as the science of generative AI changes, and as your business determines more and more what is acceptable use of the data that they're feeding into a generative AI system.

Clarke Rodgers:
That's fantastic and great advice. We'll see if your predictions come true. Steve, you are the chief security officer for all of HAQM. It's a high stress job to say the least. What do you do to keep not only yourself sane and sort of blow off steam, just stay current with the world, sort of tune in, tune out, and then how do you also make sure that your leaders are doing the same thing and taking care of themselves?

Steve Schmidt:
So the first thing that I focus on with all of our leaders, whether they're new or tenured at the company is how are you doing something to separate yourself from the job? Our jobs are incredibly high stress. There's something which are effectively never off.

So having a way to hand off responsibility between people formally and saying, all right, Clark, for the next six days you're on vacation. I don't want to hear from you. I don't want you online. And our people are incredibly dedicated. There have been places where literally I've had to threaten to turn off people's computers because like, you're supposed to be on vacation, stop emailing me.

I appreciate your dedication, but the problem is you're going to burn out. And these are marathons. These are not sprints. And so building that mechanism is super important. The second thing is having something that allows you to do something that's valued to you individually. Some people are musicians, some people climb mountains, some people go fishing. I personally am a volunteer firefighter and paramedic.

Clarke Rodgers:
Oh, wow.

Steve Schmidt:
It is something that for me is sort of the exact opposite of my day job. So we're human beings. We like interacting with people. I'm talking to you here face to face. Yay, not on video. That's awesome. But my day job is two things that are very different from that.

One is most of what I do in my day job only has effect three, five, 10 years down the road. So it's not an immediate reward. The second is it's all computers with some people thrown in and those people I can't even see or touch. So with the fun part of my life, the firefighter, paramedic side of things, if I do my individual job well, I can help a person who I can reach out and touch, have a better day. And that gives me that immediate feedback that I crave as a human being. And two, it's very physical as opposed to something that's purely logical.

Clarke Rodgers:
Different kind of stress, but probably a healthier one, maybe.

Steve Schmidt:
It is. And the irony is that there've been a bunch of studies there that people who have been in the fire service for a long time actually have their pulse rate go down when the sirens are on because it's their happy place and they're enjoying it. And let's face it, who doesn't like going down a road with sirens and lights on a truck? It's a lot of fun.

Clarke Rodgers:
That's awesome. And I would imagine and I'd love to hear more as CSO, you are faced with decisions and very critical decisions every day. Are you a higher ranked or a lower ranked paramedic or firefighter, meaning somebody else is making the tough decisions and you're just focusing on your day?

Steve Schmidt:
Ironically, I'm actually an assistant chief in the organization, but that's more about leadership skills than it is about technical acumen in the other spaces. But I've been doing it for 38 years.

Clarke Rodgers:
Oh, that's fantastic.

Steve Schmidt:
So I've built up a little experience.

Clarke Rodgers:
Great. Well, Steve, thanks so much for joining me today.

Steve Schmidt:
Thank you. It's been great to be here.

Listen now

Listen now

Listen now