Understanding the measurable value of the cloud

If you're here because you want to figure out what KPIs should you think of that feed into your business case and help you go ahead with a no go or a go decision for migrating incremental workers to cloud, you're in the right room.

Or if you're here where you've migrated on to cloud and you want to think of what KPI I should I put in place that helps you track the success of that migration, you're in the right room.

Everyone else you still have time to escape. I can give you five seconds. No one's escaping. Ok.

I'm Front of Bin. I lead Cloud Economics at AWS globally. I've been around for four years. My team does two things:

Step one, they help customers understand the financial benefits of moving on to cloud and help them make that go no go decision. Or if they are on to cloud already, they help them track their success.

Step two, those who have been spending on cloud for a while, they help them put in best practices that can govern their cloud spend and control their spend better.

And today we are going to talk about step one. I'm joined by Eric Satre from Cloud Economics and my old friend Fried from Adobe. We're going to talk more about this and some real world examples they have to share.

Alright, let's talk about this era. Anyone familiar with this? This was an era when you would think of IT resources, think of a project, raise a purchase order, wait for the hardware to be shipped. I can think of this many years ago.

I remember true story. I was leading an SAP implementation. I landed at this customer premise in Middle East and I had a team of 35 consultants. We were ready to start our implementation and we realized that the purchase order for the hardware had just been punched in.

So we got six weeks, we enjoyed the sun, we checked out new houses, ate great food. We loved our time there, but I felt bad for the customer like they had six weeks of 35 consultants to pay for and the consultants were just twiddling their thumbs.

I'll keep coming back to this story later. What happened was that we finally got the hardware. We went ahead with the installation. We were supposed to go live in December so that the customer could run their automated payroll from January. But that got delayed. As a result, the customer had to retain a bunch of their manual payroll analysts to continue processing that payroll manually.

So look at the cascading impact - IT hardware delays, business impact. Guess what? In today's world, all you have to do is click a button and you have an AWS instance up and running and you're ready to go for your projects.

Now, what happens is that in this entire calculation, when you start thinking about a business case, you start focusing on the economic impact by looking at only the cost of hardware. What you typically do is you come up with a cost and time curve. You think this is going to be my predicted demand and to match this demand, I have to buy IT hardware resources. These are step function investments.

Now the truth is that your demand rarely is linear. Your demand goes up and goes down. And for that, you then start measuring the opportunity cost - a situation where you've bought a lot of hardware where you don't have a lot of demand or you start measuring the performance decline where you did not buy a lot of hardware, you saw an increase in demand and then you start figuring, figuring out site impacts, figuring figure out employee productivity impacts.

Now with AWS, you can match the demand, right? The AWS instances can scale up, scale down, auto scaling works. But all this still focuses on the IT hardware cost impact.

So we wanted to understand from our customers, what do they measure? In 2018 we hired a third party firm IDC. It's a reputed analyst firm. They went to our customers and asked them that if you achieved value from cloud, how did you measure it?

And it was interesting. Our customers told us that the value that we measure from cloud was split across four categories:

1. The value that came from cost savings, which was all the IT hardware cost reduction that I talked about.

2. The value that came from staff productivity. So they had less number of people do the repeated repetitive tasks or they could use the same number of people to do more work.

3. Third was the value that came from operational resilience. Every minute of downtime saved led to higher productivity or better site impact.

4. And last was the value they derive from business agility. Business agility is the part which was interesting because our customers told us that because of cloud, they could move faster, they could launch new product faster, they could penetrate new geographies faster.

We ran the survey multiple times and asked our customers, what was the biggest or the smallest value? How many of you here thought that cost savings was the biggest value achieved? Nice. Well, good, good that we are aligned.

We our customers told us that if they distributed all this value across four categories, only 5% of the value was cost savings. Now, the interesting part is we spend a lot of time trying to measure that - all our focus is instrumenting systems, building processes and KPIs that focus only on the IT hardware cost reduction. But that really is just 5%.

We ran surveys to gather more data and we discovered that staff productivity and operational resilience constituted almost 40% of the value - 30% of staff productivity. Interesting that we don't put in a lot of KPIs in place to measure that or feed that into our business cases.

But the big elephant in the room was business agility - 55%. So if you achieved a value worth $100, $55 dollars out of it was just by business agility. And what we don't do is we don't put in a lot of KPIs to measure this.

And I'm reminding you of this because I challenge you as you start building your new business cases or you start thinking of KPIs to measure success, think of KPIs that you would put on business agility or staff productivity and report on them and track. Because if you just move over, do a simple lift and shift and you don't change your operating model, then you've not really benefited a lot out of that migration. There has to be a change in the operating model.

In 2022 our customers came with one more input and that was sustainability. They wanted to know that by moving on to cloud, how can AWS help them with carbon footprint reduction? So we've been working with our customers to help them quantify these benefits.

Now, the obvious next question was, alright. What does good look like? So we commissioned a benchmarking survey in 2019 where we hired another third party to go and interview 2000 customers and gather data on KPIs from them. These were blind surveys. The customers did not know that AWS was asking for this data and we did not know who were these customers.

We created this database of multiple KPIs from 2000 customers which showed pre and post migration - what was the benefit or change in those KPIs as customers migrated? And we started sharing them with our customers in one on one engagements.

For example of those 2000 customers, 100 of them are from the media and entertainment industry, actually sorry, 85 are from the media and entertainment industry and they told us on an average, they saw 25% increase in content views that helps with business agility. They also saw 30% increased in increase in processed customer responses. Those should be the benchmarks you should compare yourselves against.

And if you had to put a financial impact, I'm just giving you a very crude simplistic calculation - increase in content views, multiply that with customer conversion rate, revenue per customer per year into customers will give you a high level additional revenue impact that your migration could drive. Obviously putting in all the caveats here, there is a very simplistic calculation, but let's talk about a real world story.

PBS is a US based media and entertainment company. They stream their entertainment and education content from 330 stations viewed by 100 million viewers and 32 million of them are online. They moved on to AWS and they used Amazon Personalize. Amazon Personalize gave them recommendations, curated content and offered their viewers recommended personalized content. This led to lower customer churn for them. This led to increase in conversion of free subscribers to paid subscribers for them. They had 50% lower streaming errors and that led to a better viewer engagement. These are the numbers they track.

Let's talk about Netflix. Netflix is an online content provider. It streams content across 190 countries to 100 million subscribers. They had this challenge that they wanted to send transactional emails and promotional emails to their customers. Transactional emails think of user password resets, user names. Promotional emails are the ones which engage customers. But the challenge was these emails could get blocked by internet service providers as spam.

So they used Amazon Simple Email Service and they started splitting their email sending domains across multiple IPs. So one pool of IPs would send transaction emails, other pool of IPs would send out promotional emails and they avoided this blocking. They achieved a 99% customer inbox placement rate. They were able to process much more number of customer responses.

Similar example - going back to our benchmarking database of 2000 customers, we've got 100 customers from the ISV segment. Our benchmarks tell us that on an average they saw a 98% increase in that transaction volume. They were able to support 93% increase in the total data volume.

Again, we've got a bunch of these KPIs across different industries available in our database and we are happy to share them with them with you.

Very simplistic calculation - if you had to look at the transaction volume, increase in transaction volume into revenue per transaction and into transactions would give you a high level number.

Now again, I'm just giving you a mathematical equation to think through. But the idea here is to challenge yourself to think of KPIs beyond just the IT hardware cost reduction.

Let's talk about Salesforce.

So salesforce and amazon struck a partnership in 2016 where sales force decided to choose aws as their preferred cloud provider. You may have heard about the partnership expansion in Adam's keynote. This week sales force saw 25% growth in their digital marketing solutions. They are a crm company. The challenge for them was that as they wanted to penetrate new geographies, they had to wait six months for every new geography so that they could spin up a new data center and then start building their products. There were data residency requirements that prevented them from having data in any other geography by moving on to aws. They could just use the aws regions available in those geographies and they cut down this time from six months to four hours. Now they were up and running in four hours. They could launch new products in six weeks for each of these new geographies. So these are some of the examples.

I do want to call on Eric, a fellow cloud practitioner to talk about some real world examples and go through a recent journey. Thank you very much.

Password name's Eric Satre. And as mentioned, I work in aws cloud economics. I am a practitioner. And what I do every day is put together custom ROI analysis and business cases for migrations with my customers. And you can probably imagine as I do that work, I see a lot of business cases around it. And the trend that I have seen in the time that I've done this job is that there is so much focus on the cost of the hardware. If you know the equation P times Q, it's very simple. It's easy to understand this piece of equipment costs this much. I buy X amount. So what I wanna do is challenge the people in the room as you go through as you look at this framework and we're talking about economic value outside of cost to consider these things. At the end of the day, everybody sitting here is a business leader, you make business decisions based on whatever criteria that you'd like. But the challenge here is the next time you do that, please consider adding some of these equations in as well. To do that. I want to start in kind of the hardest area which is agility. We talked a little bit about it, but it's also the area that stumps the most people. How do I create these equations that I can put in a business case and hold myself accountable.

So I thought I'd talk about a process that's familiar to probably a lot of people in this room, the dev ops process you can see on the graphic that I have here. I've got it laid out plan, design build, test. Deploy in the blue bar represents when you have a marketable product that's earning you revenue. Ok. Through these benchmarking studies that were mentioned earlier, we have found that customers that have migrated to aws have seen about a 36% efficiency gain in their dev ops process across a wide range of industries. What does that mean? How is this achieved? Well, in the plan and design phase, you can simplify your design requirements using third party tools in cloud native applications. During test, you can do things like spin up instances when you need them and spin them down when you don't need them eliminating some long queues. And then during deployment, you can use smaller deployment, batch batches and automation tools to speed that up. All of it means the pink part happens in less time. And you extend that blue bar where you've got products and features out in the market, earning you money.

So how do we turn this into equations? I have a couple more case studies. The first one is Moderna, all of us remember a couple of years ago during the coronavirus pandemic, Moderna was one of the developers of mRNA vaccine. They had a huge high performance computing workload that needed to be done to sequence this vaccine. And they partnered with AWS. When they partnered with us, they were able to sequence the entire vaccine in just two days. This is something that would have been impossible without the type of scale that we brought to the table as a result within 25 days from the start of that process, they actually had the vaccine submitted for clinical trials that impacted not just Moderna not only extended their blue bar, but it also impacted the world.

Another great example on agility is getting to market earlier. Liberty Mutual Insurance had a great example where they had a product they were looking to develop that they anticipated would take 12 months in their current on prem environment. They leveraged a lot of serverless tools on AWS and were able to condense that down to three months. That's a nine month time saving nine months of revenue recognition, earlier launching nine months earlier, getting the market first.

Finally, last case study is Optima. It goes without saying if you can do something faster, then you can do that same thing more times in the same amount of time, if that makes sense. Well, Optima launches a lot of new versions of their software from time to time and they do dev ops both ways, both on prem and on AWS. Well, what they've done is implement a really good unit metric strategy that allows them to track the efficiency across both environments. And what they found is when they do dev ops on AWS, they're actually 56% faster. That's more than our benchmark average. They've also seen cost savings around 35% on the activities that fall into that process.

So what I want to really hit home here is that these are real examples, real companies can measure this. People do put these in their business cases and these are types of operational things that companies want to achieve and track over time. And so folks like me can help you structure these business cases if you need. But enough theory, what I want to do now is let you hear straight from the voice of the customer. I'm gonna invite to the stage, my friend Sridhar, who's gonna talk about a circumstance at Adobe last year.

Thank you, Eric. Good morning everyone. My name is Sri Dhar Ayala. I am the director of Fops at Adobe. And so helping quantify and measure business value and helping reduce spend and making sure we're making the right investments for our infrastructure is core and fundamental to my job. And so I wanted to talk today about a real life example where we took a lot of the theory, a lot of the benchmarks, the tools and the frameworks that Eric has mentioned and we walk through and put them into practical application on a real business decision that had measurable impact. And you've probably seen a lot of the impact in the last year alone.

So let me set the stage on the situation, right? Our teams have very good core competency around AI/ML technologies and we've been developing it for quite a bit of time and incorporating it into our products and services and into our research and the experiments that we run as the research teams over the years have continued to grow. We've seen a requirement and a need for GPUs and high performance compute. And this was well before the market took off and we saw the mad rush of enterprise companies trying to procure as much GPUs as possible for the AI/ML arms race, right?

And so our research team needed more capacity. And so they went internally themselves and created a business case to essentially own the stack end to end, have the management plan, have it sit in a colo us, procure the GPUs and run their experiments and workloads over there. And when they went through this process, they went the standard route that you see in traditional business cases, a lot of focus on bill of materials, right? Taking servers, taking GPU hardware prices, power the labor to rack and stack maintenance costs, license costs, et cetera. And looking at that in comparison to consuming it in the public cloud, in this case. AWS and so when they went and did that analysis, it wasn't even close, right? It's hard to compete with depreciated assets. I'll tell you from a finance perspective.

So what ended up happening was it looked horribly expensive, but there were many components that were missing from creating a complete picture or a view. Potential benefits of operating within AWS in terms of agility, speed and potential risks operating outside of it in terms of capacity and the ability to procure these instances, right? And so we wanted to create a better picture, a more complete picture for our leadership so that they would make informed decisions on their infrastructure investments to maximize value for our customers.

And so what we decided to do was to engage AWS's cloud economics team to run a value analysis for our organization to bring in some of these missing components and factors to bring in frameworks and methodologies and approaches and subsequently bring in benchmarks to fill in the gaps on data that we may not have to help us create a more informed picture and a business case.

And so I wanna talk about the conclusion and the outcomes and some of the lessons learned. But I think it's really important to understand the flexibility of these frameworks and the approach and how you can customize it for your use case and your organization. And I would love to talk and walk through what Eric and I put together and some of the factors that went into our decision making.

So actually, I'm gonna hand it off to Eric to walk through some of the analysis.

Thank you Sridhar. Excellent. Yes. So our analysis started a lot of the categories that were mentioned do in are included in the final. But this is how it started. It was really simple. It was basically a simple analysis that said, hey, what's the cost of a GPU? What's the cost of an AWS GPU instance? And guess what the result was? AWS looked way more expensive.

And so the first thing I did was I checked it, I looked at it, I looked at the pricing on the AWS side looked great air tight. The other thing I did is looked at the bill materials, did a little market research and the cost of the GPUs also looked above board. So I agreed with the analysis. It was more expensive, but I challenged Adobe. I said, let's think of a couple more categories here.

So what I challenged them to do is say, let's expand this to more of a cost of ownership analysis. Or you guys may be familiar with the term total cost of ownership or TCO, but it basically includes more of those associated costs directly linked to the hardware. And I said the first thing is you're missing a few things on the AWS side, right? Got to be completely honest, things like data transfer costs. There's gonna be a lot of that. When you look at high performance computing and machine learning in general, there's a lot of data flying around also enters enterprise support.

We were able to model economically and we also concluded when we were doing the architecture that you're probably gonna need some storage as you run these workloads. But then I said, let's be fair, let's do it on the COO side as well. And we came up with these categories, maintenance, GPU, warranties, provisioning for spares, rack, hardware, power, space, cooling, storage, maintenance, storage, hardware, networking, and bandwidth.

And we've got great established models for financially modeling these things, right? A lot of these are P times Q and there's a lot of great market data out there. But what we did is we went area by area with Adobe to make sure that we were matching their economics in each one of these areas as closely as we could.

It's one thing if a first party like AWS comes to you and slaps a business case on your desk, it's completely different when we work hand in hand to make sure we're matching the economics as close as we can.

So what was the result of this cost of ownership analysis? AWS was slightly less expensive. Great, not super inspiring, you know, no champagne was popped at this point, right? It's basically a coin flip, slightly less exciting, not something you're gonna put in your autobiography, right?

So I said there's an area we can go deeper and that's where it comes down to the theme of this presentation. We had done the simple analysis that AWS was more expensive. We expanded that to a cost of ownership analysis that showed that it was tighter, but AWS had the slight edge.

But I said, let's talk about why I'm in this room in the first place. What are the economic value, what is the economics of the value you want to achieve on this platform? And so we discussed a lot of things, some things worked, some things didn't. We landed on three in the final version:

1. The concept of price performance. This would fall under business agility. At the end of the day when we were looking at the architecture of what we were modeling out economically, we found out that the P5 instances on the AWS side were more powerful than what was being quoted for the colo. Well, if the machine is a little bit more powerful, then there's an economic benefit that you get from using that. It's not pure apples to apples. So we quantified that benefit.

2. The next thing that we looked at was pain points. What are pain points in the process that you're looking to tamp down through your operational metrics? And we came to delayed and canceled jobs if you guys have worked in ML before, you know that this is a big time suck. And we've got great metrics on the reduction in canceled jobs and delayed jobs, which is a resilience benefit.

3. And the final one that we looked at that was missing from both analyses before was staff productivity. At the end of the day, there are human beings that work with these workloads and the tasks are dramatically different on the AWS side, colo data center, whatever it may be, we stacked up those tasks on these workloads for the people that would be working on them, calculated the difference and applied an economic figure to that as well.

So now what do we have? We've got a pretty good picture here. We've got the simple analysis, the TCO or total cost of ownership. And then we've also got the economic value.

But as I look around the state or at the crowd that's here today, some of you may be crossing your arms and say that's great. However, I run a budget and I look at these three at the bottom and I say, I don't know how those are gonna impact my budget. What's the big deal? And to those people, I would say that's fine. You guys are business decision makers, you get to pick the criteria that's the most important to you. And so if one of these things isn't the most important, that's totally ok. But having a complete deliverable that has not just the infrastructure cost but has those other benefits as well. If you've got a business leader that's making the decision alongside you, that could be really important to them, somebody that's running these experiments that really cares about canceled and delayed jobs probably cares about that. And so that way you pick and choose what's important to you. Everybody's got a common plate that they're eating off of. And at the end of the day, everybody gets the information that they want.

Ok. So that's how our analysis wrapped up as far as the mathematics went. And I'm gonna hand it back to Sridhar to talk about what came next.

So this is an age old, you know, question that people ask right, build versus buy just with the modern context. And so I wonder if people can guess what we chose to do. I probably wouldn't be speaking here if we chose the colo. So we decided to go and put these GPU workloads in AWS and there were a litany of reasons why we did so, but I wanted to highlight a few primary ones, right this is something that I asked teams to not do, don't discount your core deep expertise. You want your engineers and your product teams to focus on creating and delivering new products and feature sets, not setting up rote infrastructure to be able to do so. And the value to enable your talent to be able to do that and be free to do so was extremely valuable for our organization.

Another aspect that was very key, that wasn't part of the initial equation but was a big factor in our decision making was time to market. We saw where the market was going before. A lot of other folks knew where the market was going and we realized we needed to get there quickly to set up a colo in best case standards, maybe nine months to a year before I get the physical infrastructure set up the management plan put together, have it integrated with my operations, guess what? We're already in AWS and we already have those things established and controls in place. And it just made sense from an operational perspective to do so.

The other factor that weighed quite heavily in our decision making process was also supply and demand, right? We saw where the market was going and we realized there would be a huge demand, not only for the physical infrastructure but for our products and services. And we needed a partner who could scale with our growth and have a reliable track record of delivery and have the connections from a procurement perspective to be able to make sure that we would hit our strategic objectives from a business standpoint.

And then finally, and I think Eric hit the nail on the head. Here is all about a call to action and bringing stakeholders together. You're not gonna get agreement across all of your stakeholders. But I want us to agree on the KPIs that we're measuring all of the methodology and the approach to defining those KPIs and the same information as we make decisions together. Our incentives may be different, right? But at least we should have the same information and levels when we're making decisions.

My finance teams are looking at the budget and how much this is gonna cost this year, right? Whereas the business teams are thinking 3 to 5 years strategically and where we can grow and bringing that together and bridging the gap is really what we do with these business cases so that we're all in lockstep when we do make a decision and go forward.

So we found it to be highly successful. You've probably seen a few of our products and services in the last year that have taken full advantage of this. And it's been great for our customers and it was the right decision.

But there are some other lessons that I wanna highlight that we took away from this for future workloads, right? And so here are a couple lessons learned in general:

1. Have flexible frameworks - that's very important. There's no one size fits all solution for companies across the board in different industries, even within companies between different product teams, they have different ways to measure value and different workloads have different attributes that we have to measure. So you need a framework that captures, you know, the pertinent details, but is flexible enough to adjust to the use cases and be fit for purpose with your customers.

2. The idea of enabling your teams to focus on their core competencies, don't undervalue that right. They're working on revenue generating products and services, they're working on things that delight our customers setting up road infrastructure doesn't do that, right. So I think it's very important to quantify and qualify the benefits for enabling your engineers and product teams to work on your core services.

We highlighted the economic factors and down below, there are a couple of them that we focused on time to market was absolutely essential and we quantified the benefit of getting to the market first and the market share that we potentially could gain.

Cost is an element but making sure that you're balancing both sides of the equation when measuring against each other, knowing that there are refresh costs, long term maintenance costs, labor costs, etc. associated with staying on prem

Demand and availability. Right? I don't think I have to tell anyone here, it's hard to get a GPU, right. And so knowing that you have a partner that understands your needs can get ahead of the curve based on the capacity and supply that you require was vital for our business moving forward and was a predominant factor in our decision making process.

And the overall overhead right to maintain and continue to operate those functions and the flexibility that you have to change those functions, right? And quantify what that value is to you.

The other thing I would also ask you to do when you're going through, this is quantify the risk. We oftentimes don't do a good job of putting dollar value to risk. And so understanding what missing a capacity delivery would entail understanding what workloads wouldn't be able to be developed and deployed. What products would face delays in deployment, understanding that and quantifying that from a numerical standpoint was very important to the business case as you go through this and get better at it.

And I think what Eric mentioned is absolutely essential when you're trying to tell your business what business value you're driving from your infrastructure. It's great to speak their language, right? Get to business KPIs that matter to the specific business we talked about, you know, revenue per transactions, we talked about customer acquisition numbers, retention numbers. There's so many business KPIs that are pertinent and the closer you can get to those KPIs and translate what your investments in infrastructure will result in terms of benefits to the business. The easier it is to demonstrate the value to your stakeholders when you're going through this process.

And finally, I think this is actually the most important takeaway is building muscle memory, this isn't a one time activity, right? We did it for this specific use case. But the lessons we learned the processes that we created the frameworks and the methodology and the assets that we developed we now have and we apply to other strategic business priorities and high dollar value workloads to make sure that we're all speaking the same language or we're all on the same page and aligned when it comes to making a decision.

So these are the key takeaways from going through this process. I know a business case is as old as time, but you're gonna constantly do this as technology evolves and changes and having a standardized approach, having a lexicon that different stakeholders can understand and definitions that are clear and established. And KPIs that you're measuring to is absolutely vital to make sure that everyone is behind the decisions that you make from a business standpoint.

So that's the takeaway from the Adobe experience. It was very successful working with the Cloud Economics team to be able to help accelerate that business case and improve our own ability to continue to create these cases in the future.

And so with that, I think I'm gonna hand it off to Bruno to recap what we learned today.

Thank you, Erica. Thank you for your partnership and thanks for choosing AWS.

We're nearing the end of our presentation. So before we open up for Q&A I, I would like to remind everyone of three things:

1. Remember, cost is only 5% of the equation. So spend your time at looking at the other pillars.

2. Challenge yourself to think of all KPIs beyond cost, track them, share them with your business report them internally. Figure out how do they impact your overall success or what can you change to show success?

3. There are benchmarks available with AWS. If you need any help with them, reach out to your account teams, your account teams can engage the Cloud Economics team. Our team members are spread across the globe. Our team will be happy to engage with you. Share those KPIs and benchmarks. Remember these are gathered through third party research. So these are more or less independent benchmarks from third parties. We've written white papers which we publish on our website, access those white papers, you get access to some of this benchmarking information.

Thank you.