Accelerate generative AI application development with Amazon Bedrock

All right, welcome everybody. How's it feel to be on day three of reinventing the afternoon? Anyone got any energy left? Oh, wow. This crowd rocks.

Ok. Uh I'm super excited to be here talking about the hottest topic uh all year. And uh why is it so hot? It's really, in my opinion, game changing. So we're seeing builders everywhere, invent brand new experiences using generative AI and also tackle problems that previously were really either impossible or too expensive, too complicated, too time consuming to build and with foundation models, they're able to get that done.

So excited to talk to you about Bedrock today about how Bedrock will be able to accelerate your journey, help you take advantage of generative AI much more quickly securely and with choice and we'll get into that quite a bit more.

My name is Mark Roy. I'm a principal ML architect at AWS and uh the lead archi solution architect uh for Amazon Bedrock. So it's been pretty exciting. I've been with AWS for six years now. Uh worked with hundreds of companies on putting ML into practice uh at scale. Uh this year has been 24 7 Amazon Bedrock. Uh I don't know about you, but I've had 40 years of building experience, anyone else out there with 40 years uh or it feels like 40 anyway. I don't know. Um but to me this, I've worked on so many cool projects over those years, but this is really the most compelling thing I've ever worked on. So, really excited to talk about it.

I'm here with Uh Gisha Runi, an ML specialist who will be uh talking right after me and Sanjay Nair from United Airlines to go into how they're using Bedrock.

Uh so before I go into the actual agenda, let me kind of frame the discussion a little bit and I were thinking a few weeks back as we prepared for this session, what do people need to know? They probably heard about Bedrock endlessly at this point, maybe not. Uh and they heard a great keynote from Adam on Tuesday, another great keynote today from Swami, a boatload of announcements about Bedrock. And so what we decided the piece that was missing that you couldn't get from all the other sessions is tying it all together.

So a bunch of you here might be figuring out, how do I get started? How do I plan to fast track what I'm doing here? Taking advantage of all of these things in Bedrock. So that's what we're going to take you through for the most part here in this session.

Uh here's the agenda, the next 15 minutes. We'll be having Garish up here to talk through use cases. Uh and then after that, if there's co-operation from the demo gods, we'll have to all say a quick prayer. Uh we'll see live demos of Bedrock in action. Uh Sanjay will come up and talk about how Bedrock is helping United Airlines in their journey and then we'll wrap up.

So before we get into the use cases, let me frame it here. You have all been hired by OC Tank Financials. This is a fictitious but very large financial services firm with multiple business units. They do investment management, they do retirement services, they do insurance and more. They do have dozens of teams. I've worked with them directly and they do have at least 500 use cases teed up.

So what we're going to do is take a look at three of the most common generative AI tasks and Garish will walk you through a use case in each of these and he will make it so clear how the different pieces of Bedrock help you deliver this faster and better.

Uh so with that, I'll hand it over to Gris green button. Thank you, sir.

Good afternoon everyone. All right. So you're all with Tan Financial. And I am here as a strategy consultant for your generative AI journey on Amazon Bedrock.

The first use case that we are going to see today is question answering. We all have questions, we all want answers, the answers have to be correct to the domain in which we are referencing the question to at Ang Financial. You have hundreds of thousands of documents could be policy manuals, it could be prospectuses, it could be private placement records, it could be anything your customers want to interact with your data with your documents and they want to be able to get an answer. that is correct. They do not want a generative AI foundation model to give them an answer that the model thinks is correct. And you at Ang Financial want to be able to give them the right answer.

How would you do it? You would build what is called a knowledge base. A knowledge base is built from essentially ingesting documents from your document archive. It could be really any data. In this case, we are considering a document archive. You take all of the documents you you call and ingest API into knowledge bases for Amazon Bedrock.

What is a knowledge base? A knowledge base is a combination of two things. One, it has an embeddings model and the second piece of it is it has a vector store to store the embeddings that you just created from all of the documents that you have ingested into the knowledge base.

Why should you use knowledge space knowledge base has intelligence built in that will help with automatically chunking the data that you are putting through into the knowledge base. What does a chunk mean? You take a pdf document, a pdf document. Uh let's say a pdf document is 100 pages long knowledge base has the intelligence to identify what is what should be the appropriate chunk size to create an highly efficient vector. It then sends that chunk into the embeddings model. In this case, I'm recommending you to use uh Titan text embeddings. You use the title text embeddings, you create a vector store it into a vector store of your choice. Thus, you now have all of your documents as vectors ready to go.

We're gonna look at two types of interactions with this knowledge store. Number one, your customer comes in a chatbot that uh that is built into your website. They ask a question, let's say they wanna cancel a trade that they had put in for a specific uh security that the trade hasn't been executed yet. But the question is I wanna go ahead and cancel this trade and also I wanna go to cancel the state. Do I still qualify to cancel the state?

So you want the query to go into the knowledge base? Look up who the customer is, sorry, who look up what the policy of cancellation is? Get that information back to the customer. Uh but before it gets that information back to the customer, it sends it into a language model to generate text tied to all of the search results that came back and sends that information or completion back to the customer. That's your chatbot experience.

At the same time, you've got a support associate sitting on a support portal and she receives information that the customer wants to go ahead and cancel the trade and you know, make sure the customer doesn't get charged for it. She goes in, she wants to make sure that the uh policy allows them, allows her to customer uh cancel the trade. She uses what is called as agents for amazon bedrock. In this case, we are calling it. The order support agent, she then sends a query into the knowledge base. Asks for the uh the policy. The policy comes back and says, yeah, it hasn't been executed yet. Go ahead and cancel it. No problem. She then simply gives it an instruction that says, cancel the order and process any refund.

At this time, the agent sends an uh query into an API call into the orders database. The orders database returns is still pending. Order gets it back into the agent. It takes the next action, canceling the order and sends that information back over to the support personnel.

There are two features. Um yeah, support person. There are two features on the slide that I want to highlight. One is knowledge base and the second piece is agents.

Let's take a look at what knowledge bases are knowledge base helps you connect the foundation models securely to your company's data. You don't ever want your company's data accessible to the foundation model directly. You want to keep your data secure. That's number one. It helps with increasing the accuracy of the responses because it is not giving you a response or giving your customer a response from what it has learned from the public domain. But it only looks at the data that you have provided as a part of that call to give you a response back, right? It helps you with an easy way to implement RAG um with respect to the ingestion, automated ingestion of the data, any prompt augmentation that you need to do. Let's say if the prompt is really just I want to cancel my order, but you want the model or the back end systems to understand who is this? I you need to pass in the additional metadata on me as the customer that's prompt augmentation that you're doing there. And of course, anything, any, any retrieval that you need to do from your knowledge store knowledge base makes it really, really easy for you. And most importantly, a lot of our customers are saying great. All of these answers are good, but I really need to know or, or my users want to know where are these answers coming from? Citation is available automatically through knowledge based integration where applicable

Agents is the next feature that I want to cover as a part of this use case. In simple terms, agents will help you plan and execute tasks on your behalf. It takes in a a query, a question from the customer, it breaks it down and essentially creates a pipeline of tasks on behalf of you. It then helps you connect or rather it then helps the model connect to the various data sets um securely through API calls, it makes API calls on behalf of you all securely um through um agents for amazon bedrock.

And lastly, ok, great

The agents took a bunch of actions on behalf of you. How do you know that you don't want that to be a black box? What do you do?

Agents give you a trace function which you could use to uh troubleshoot if you have any questions on. Okay. What are the various six steps that this agent took to complete this action? You could use the trace function in Agents for Amazon Bedrock to troubleshoot uh the interaction. So that's Agents for Amazon.

Great. Alright. Now we've completed the Q&A part of it. The question answering uh use case.

The second use case that I'm covering here would be document summarization. This is again, you're all with OC Tank Financial. You've got research department at OC Tank. You've got hundreds of research analysts that are tracking thousands of stocks on a daily basis. There's a lot of documentation that comes in your earnings called reports, your ESG scores, your, you name it. They've got a lot of different documentation that they need to process mentally through, read through it, provide that research to their portfolio managers to make portfolio decisions by cell hold and so on.

You've got a ton of documents that you need to summarize and store it into what we are just calling a document summary DB. That's the use case. How do we do it? In this case, the builder creates a curated test suite and has access to pre validated uh summaries that she can use to create prompts. She uses prompt engineering to test how the model works. In this case. She um she has heard a lot of good things about Cloud.

By the way, Cloud 2.1 was announced yesterday, it's available starting today. Um and it's much cheaper. Uh it's about 20% cheaper than GP T four turbo. I just wanted to get that out here. Um you've got um the builder comes in, she does prompt engineering. She's heard good things about Cloud. She sends the data and the prompt into Cloud. The prompt could just be summarized this document for me. She sends in the document. She looks at the response she's like, ok, great. Um but she wants to see if she could do this um quicker and cheaper le lower latency and lower cost.

So she's heard good things about Cloud Instant as well. She sends the same prompts and the same data set into Cloud Instant all through the same API it's all one invoke API on Bedrock. She gets the output both of the outputs over into um the model evaluation feature that we launched yesterday does a quick comparison on what she finds out is, yes, the instant is much faster and cheaper. However, she wants the higher precision because they're making financial decisions. You want the summary to accurately reflect what is the document? No shortcuts for that. So she chooses to go with uh the larger Cloud model.

The first step after POC going into production is to do a backfill. She's got about 500,000 documents that she's got in the document repo she needs to be able to take all of the documents run it through. Very simply put Batch for Amazon Bedrock, which I don't think we uh it was a public announcement, it's available starting today um uh which will help you run an asynchronous job to pull all of the documents from the document repository and get that through the model pipeline. In this case. Remember I said Clod, she uses Clod the premium model, write the output into an S3 bucket from the S3 bucket. You stick the data into a document summary DB. Great. She's all set now. She's done her work, she's done her homework, she's completed her POC, she's loaded all of the 500,000 documents from the repository.

We all know these documents arrive by the minute, by the hour. What do you do? You set up an event driven pipeline is an S3 trigger. Once the document lands in S3, it calls a Step Function Step Function native integration through Amazon Bedrock was announced on Monday. The stuff function calls the invoke model API to Amazon Bedrock. It sends the document into Cloud Cloud does a quick summary of the document, writes it into the document summary DB that you have built and now you have a very happy research analyst that doesn't have to read through volumes of documents. She has quick summaries into um all of the documents. If the summary is something that appeals to her, she is then able to, it's a he a summary appeals to him. Um you are, she's then he is then able to look at the document and take additional action on it. Document summarization. This is something we see pretty much with every customer that Mark and I uh speak with document summarization. Yeah, exercises.

Alright. In this uh use case, I wanna highlight Batch. Like I said earlier, Batch helps you with running inferences on large volumes of data. You're not going to be throttled with the uh all the other traffic that's on the service. It essentially takes your workload, it runs it asynchronously gets the job done faster. There is really no throttling there but it is asynchronous. So it's, you're not going to be able to get the results back in a second or two. You could set it up and schedule it at a time that works for you. Typically, you would use this for backfilling jobs or overnight processes. If your use case can afford to do that as it can support doing that.

It's a fully managed model invocation job. There is no code that you need to re write to retry any failures. And lastly, uh most importantly, be it a base model that's available directly from us or a custom model that you choose to fine tune on Bedrock. It's the same API Batch Batch for Amazon Bedrock works on both types of models or rather I should say uh both inference types.

Alright. Down to the third use case entity extraction. Let's talk about this tan financial millions of customers across the globe. Um they get emails about order cancellation. Um new account opening. Uh where is my last month's statement? It could really be anything. And lately what oct financial has been seeing is the response times back to the customer has been trending bad. So they want to introduce 10 to B I. What do they do? They wanna be able to read through the emails to pull entities. Entities could be a customer's account number. It could be the stock symbol that the customer is asking about. It could be um it could be the city, it could be your address. It really could be any entity that you're pulling in. Uh that you want the machine learning model to pull from the email. You want to pull the entities out, extract the entities, store it in a database so you can take further action on it. I'll walk you through what that action. One of those actions could be.

Again, you've got the same builder. She's got access to a curated uh test suite and pre validated emails that she knows what the entities are essentially labeled data. She then sends it into, she does prompt engineering with it. She, she gets the prompt ready. She's heard in this case, she wants to try Llama two, which we recently made available. The 13 billion was available about two weeks ago, 70 billion was announced earlier today.

Prompt engineering does a little bit of prompt engineering sends it into Llama two. What she finds out is the accuracy is not quite there yet. It's about 80% and she, and the reason for this is Llama two is trained on public data sets here. She's dealing with data sets that are very specific to her domain. Perhaps very specific to f uh octane financial.

She now chooses to fine tune the model model customization. She fine tunes a model. She creates a fine tuned version of Llama two sends it into model evaluation and moral evaluation shows her that it's about um 90 the high the accuracy is higher, let's say 90%. She now takes it. She's now ready to take it into production.

The pipeline goes this way, email comes in from the server goes into an agent. The agent wakes up, takes the payload, sends it into the fine tune Llama two if the response it doesn't lama two, it doesn't enter the extraction. It creates an automated response. If the automated response, uh if the agent sees the automated response to be over 85% confidence rate, it sends an automated response back to the customer. If not, that email is now sent over to a support teams. Q human is get if it is pulled in the human, creates an assisted response and sends that message back over to sends an email back over to the customer. Thus reducing the the volume of emails that go to the customer.

In this case, I want to address fine tuning. Fine tuning is helps you maximize the efficiency of sorry. Sure. Um fine tuning helps with maximizing accuracy of your foundation model by providing your own company's data. Uh we announced uh fine tuning availability on Bedrock for uh Llama two command in addition to Titan foundation models, and you're able to access fine tuned models just like Batch. You're able to use fine tuned models. The same way you would use um the uh base models.

The common thread that you've probably noticed in all of the three use cases is the choice of models.

We currently have six model providers that includes us, Amazon and over a dozen models. You so you certainly expect to see the number of model providers and the number of models grow that are available on Amazon Bedrock.

And as the IT leader at or a business leader at ang financial. What you have here is you're able to support multiple use cases all on a single platform. And that's Amazon Bedrock for generative AI.

To that heading over to Mark for a quick demo. Thank you.

Ok. Now we have to log back into my demo machine here and then press the big red demo button. It's actually green Mark. Um let's see. Yeah, it's labeled well demo. Oh my gosh, it worked awesome.

So I'm gonna show you three or four demos here. We'll see how much time we've got. I wanna check my, check my score here to make sure we don't overrun. Uh so one of the things that we emphasized over and over when we talk about Bedrock is the concept of uh model choice.

So here, uh I wanted to just quickly highlight that I'm just gonna throw in a prompt, who was the first president of the United States. Anyone know that? Uh so we'll hit return here. And what I've done is I first, I used Bedrock to generate a streamlet app because I'm not very good at writing user interfaces. So, uh and then in that app, when you hit return, it spins out multiple threads against a list of the model I ds one for each of the available text models and sends it in parallel to each of them.

So there's Coherent command. It's a pretty long answer. I could do some prompt tuning there. Uh and then we've got Cloud Cloud instant, Cloud v 2.1 already here. Uh Titan text light and express uh Jurassic two ultra and mid and Lama 13 b and Lama 70 b. Ok. Yeah, I'm impressed as well. Uh let's take a quick look at the code here.

Um so here's my list of model identifiers saying you could get these from the console or there's an API to, to list the foundation models that are available from Medro as well. And here's a simple call to invoke be bedrock model uh which is called from uh each of the threads that we spin out uh when we hit return there. Uh and it's just using the model identifier passing in a prompt. Uh how many tokens you wanna get back the temperature and things of that nature.

So one API multiple models and those who will evolve and get better and faster and there will be more providers and more models over time. The idea is your entire company can take advantage of a full suite of models and they don't have to learn a separate piece of infrastructure for each and because it's fully managed, you don't have to worry about all the in intricacies of how to best host each of these models.

And if any of you have tried to host large language models, you've got a feel for why they're called large language models, right? So there's a little bit of work involved there. Ok. So there's a, a quick demo of model choice. Let's move on to knowledge bases. You heard Garish talk a lot about knowledge bases. Let's uh see if they work.

Uh I'm going to select a model here to be used. I'm going to put in a little query here and then I'll explain what I'm doing. Here. So as you can tell from the name of the uh database maybe of the knowledge base. It's a tax knowledge base for question answering.

Uh so ii i downloaded uh 50 or so uh irs publications. They were each maybe 5, 1025 pages uh worth of um documentation here and its thinking this is what we get for doing a live demo. Uh and what it's doing is it's using the retrieval API. There we go. Uh pretty busy here at re invent.

Uh so it called out to the knowledge base using an API that says take in the query, it uses the embeddings model associated with that knowledge base. And then it does a similarity search in the vector store against all the documents that you chunked and that you vectorized and you saved and all without writing a single line of code here, i created the knowledge base through the wizard. You could do it through the command line or through the API fully managed, pick my foundation model, pick my vector store point to my documents and it's available here.

The API lets you plug in a language model of your choice and ask questions and you're good to go. Let me show one other little feature here. Um you don't have to have it use the language model. You can just ask another question here. Am I allowed to deduct? Uh let's say oil changes from my commute. So I'm trying to get as many.

Do you see how fast they came back? Uh ok. So you can then look at the results. This is using the uh just direct retrieval. So you've got flexibility here. You can just get back the chunks that are matching your question or you can have it also invoke the LLM for you and give you back a friendly answer here.

You can see how fast it is when you just get the direct uh retrieval and it gives you access to the individual results and you can open up the, the source document as well. So that was knowledge basis. Uh let us move on to an agent.

One of the coolest new capabilities that went uh generally available on Tuesday in Adam's uh keynote here. I've got a customer relationship management bot. So maybe you've got sales force or some other um uh system managing your customer information, tracking your activities and your opportunities and, and other other things related to customers.

So here we're trying to help out account managers or sales teams. They've got a bot that wraps some of those API s that are available and I'm going to ask it to suggest a meeting with this customer. I'm meeting uh you know, soon, maybe next week. Uh I've worked with multiple customers. I can't keep track of everything that's going on in each one of them.

So here I'm using an agent to figure it out. The agent knows a few API s so ok. It didn't like that for some reason, I'm gonna try that again was recent interactions with it. Didn't think it had enough information there. Let's try uh try another one and that one's not working either. I'm going to try once more. Ok? A little bit better.

So on the right, I'm showing you the trace that Garish talked about where it uh called various actions that were available. It comes up with this plan, it takes the request, it knows what actions are available, what knowledge is available, it dynamically figures out a plan, what is the right actions to call, what is the right order to do them even some iteration.

Uh and here's an interesting piece, it called a list recent interactions API that was available and that looked up a recent set of meetings, for example, we met on this date, here were the notes, it looks like they're interested in uh my product smart ship, but they're also talking about budget cuts and you know, in a meeting even further back, they weren't happy about our reliability.

So what did the agent do? It took that information and it said, ok, let's have a meeting about uh smart ship. We'll demo the new product, we'll review pricing for them, we'll talk about our timeline and we'll look at reliability. So it generated this agenda for me, it even suggested when we should meet because it knew this customer likes to meet on Wednesday afternoons.

So there you go. All right. Ok. Let's see what we have in store here. Um let me show you a little bit of the agent console as well. So that was using the agent API the same agent is available through the console. Uh and uh you can test it through the console as well. You can version it, you can give aliases to it. So it's all, all very flexible.

Uh it has the latest working draft that you can edit. Uh and one of the things that you provide to an agent is a set of instructions here, I'm saying, hey, you're a customer relationship, management agent. You help people with uh you know, understanding what's going on with that customer and so forth and so on.

Uh and also I've configured a set of actions available and you can see uh that part of making an action available is providing uh a schema. You say here's the name of the interaction, the name of the API a description of that, what are the input parameters? What are the output parameters? And that's what helps the agent figure out what it should do and what sequence.

So it uses generative AI to dynamically come up with a plan and then execute that. Um the other thing I'll quickly show you is that actions are implemented using lambda. Anyone here use lambda a few hands? Yes. Um so the lambda functions are very straightforward. You're basically just writing a, a simple dispatcher that says what kind of a call is coming in. Uh and then you go execute that.

Here's, it's just a, a dummy function i've got right here where i'm returning a set of meetings, the real implementation would have called the sales force API to look up that data and return it and then using the power of the LLM not only to execute the plan or come up with the plan, but also to process the results and do something interesting with it.

Uh so with that, I think it's time now to hear from Sanjay at United. You are. Thank you, Mark and uh good afternoon everybody again. I'm uh Sanjay and na I uh I'm from United Airlines. I head up our data AI and analytics engineering. I'm the part that uh gets you out from the generative AI created tech bank scenario to the real world customer perspective from one of the largest airlines in the world and to build on the story, the product and the technology that Mark and Girish walked you through with Amazon Bedrock and I'll start with a quick introduction of the team.

Talk about the challenge and the opportunity that presented itself this year. Talk about how we strategized, designed and implemented our solution in partnership with strategic partners like AWS and then talk about the path forward, how we think we can unlock the vast potential of generative AI using Bedrock.

Our mission at United Airlines is to be the best airline in aviation. And what that means is more than just being the best airline in terms of size or our route network in the number of destinations we fly to or even the number of customers we carry. But it's to be the best in driving the best customer experience for our customers in driving the highest customer loyalty and engendering an engaged employee workforce.

And what's really exciting about our story is the growth strategy which entails adding over 700 aircraft to our fleet over the next few years.

Sanjay That's almost doubling our fleet. But doing it in a way that we've called our motto. Good leads the way and it's about doing it in a way that does good for the communities that we are privileged to serve and doing right by the customers that we carry on our aircraft.

Now, we are a complex business and we believe at United that AI and ML and in particular generative AI is going to have significant incremental value to our business and the many different domains of our business, whether it's network operations or in flight or flight operations, revenue management, the commercial side of the business or tech and maintenance.

And we have my team which is privileged to support the entire enterprise, whether it's an operations, commercial, digital and beyond to implement the platforms on generative AI that we believe will add the significant value.

I think it would be a safe bet to say that when 2023 done that most of our companies here did not have a plan for generative AI it was more that gender de way hit us than we had a plan for it. But we had to react quickly, all of us. And the reason we at United had to react quickly was a lot of our vendors. A lot of our partners were reaching out to business users across the enterprise with solutions that are going to solve all their problems.

But why do we recognize the vast opportunity that existed with generative AI? We also recognized the risks and the pitfalls of this technology. Number one, of course, it's new, but of course, we also knew that there are risks with. But what we set up was a set of guiding principles that helped us strategize for this program.

The first of course is we're going to listen to all of our business users and all the ideas that we have that we call innovation, propelled by inclusion. So we want to make sure we considered all the use cases and we had the right approach to select which ones made sense for generative AI.

And of course, we had to get them to market quickly because that's where the all these different use cases and partners were coming in with saying they can implement these things quickly. So that was a prime consideration for us.

And of course, thirdly, we had to have an easy and quick integration because that was also being promised by other partners and vendors.

So what did we do? We got a few of our strategic partners together and AWS of course, is a strategic partner that we've using for our modernization of our digital technology platforms to the cloud and to help us separate hype from reality. And that's where we got introduced to Bedrock is one of the first companies to preview it.

And the few of the features that really convinced us about Bedrock was the fact first, it's a fully managed AWS service, which is great because it's built on the existing AWS platform that we have. And second, it works with like Gish mentioned an API invocation framework that makes it really easy for our business users and our application development partners to use it.

And third and most importantly, it provided the choice of models that we really need it because we all knew and know this is still a nascent emerging world. There are a lot of things we don't know about this program if you will. So we wanted to have choice that we could experiment with, play around with and then decide what was the most optimum optimal model that worked for a use case.

On top of that. Of course, we didn't want to have these point solutions all over the enterprise that would make security a nightmare, it would make responsible AI a nightmare. And so we wanted to make sure that we have centralized operations to enable us to drive this whole body of work forward.

I talked about security already security matters for a large enterprise. And so this was critical for us to make sure that Bedrock has a lot of features and capabilities that comes with the implementation of the platform that was important for us to make this selection.

And I talk about experimentation and quick prototyping. That's where you decide whether the use case is good or if not, then we have the platform for failing quickly and moving on to the next use case which brings us to where if this use case makes sense, how do we scale it and get ready for production? That really is the strength of AWS, which provides the platform for us to scale these use cases to production quickly.

So let's talk about a little bit about the technical architecture that we built this from now AI and ML is not new to United. We've had AI and ML for almost 15 years, but what's really exciting was the last 15 months that we built our ML Ops platform on top of our United Data Hub or our data platform that we modernize on the cloud.

And that really was the platform that enabled us to accelerate and implement generative AI for United. And that's the platform that we integrated Bedrock with, in partnership with AWS with all of the solution architects and, and, and partners that we have, we were able to create the platform with Bedrock bolted on top of our ML Ops platform.

And that gave us those capabilities. We talked about previously the API invocation, the multiple choice uh of our LLM models. And then from there, we could extend Bedrock and our mls platform into other things like what Girish talked about in terms of the knowledge base and the vector database that would help us with embedding documents and document junking that would preserve the context and the memory for the prompts and the responses and for a continuing conversation for these LLM models.

And of course, we would extend this further as we implement the API framework into things like service logging to log the prompts and the responses to make sure that we are able to store it in our S3 in our United Data Hub. And then we can use it to further improve and fine tune our models going forward.

We also added some accelerators and adapters to this framework which is CS templates here. And these templates are things like for summarization, which again we talked about as well as chatbot templates. And these really helped our use cases and our business users and this will continue to pay forward as more and more use cases come these templates can be used to accelerate the implementation and more importantly, reduce the custom work that our business users and an application development partners would need to use.

And most importantly, no discussion on a technical architecture for generative AI can be complete without a responsible AI conversation as well, especially for companies of our size. And a lot of these functions come in built with Bedrock and a lot of these things we are extending as well with things like response filters and context filters to make sure that we are monitoring things like tonality and toxic responses, potentially that we can then work and, and, and tune out as we put a human in the loop as we work these use cases forward.

But really the key takeaway from this slide really is that AWS and Bedrock provides the foundational infrastructure and they'll continue to innovate and, and, and support that and what we believe we'll focus on is the data to create the data knowledge base and then focus on the prompts and the responses to continue to generate value, significant value for the business.

So let's talk about some of the learnings over the past few months based on what we talked about in terms of the technology and the framework. First, really that was helpful to us was we didn't have to pivot our data and ML engineering team, 100 and 80 degrees we were able to build and grow from the base that we had built on our AWS ML Ops platform. And that really was a validation as well as a great accelerator for our engineers and my team, especially to be able to deliver these solutions to market quickly.

Secondly, was really to build the organization and the communication framework because uh to support an enterprise of our scale and magnitude, we needed to make sure that we are getting the message about not just the strength of generative AI in Bedrock, but also about the potential risks. And how do you manage the responsible AI components? And how do we manage the risk associated with it? So that really set us up to be able to grow and these use cases and proof of concepts with the right framework to support it.

So if I was to summarize the learnings in a sense, there are three chapters to this book. First is to create the program structure that provides the framework to intake these use cases and proof of concept and then evaluate their readiness. And the second chapter really is around the readiness itself to make sure that you have a framework by which what we used was a matrix with value in one of the axes complexity on the other. And really started focusing on the ones that have high value and low complexity to start with. And that provided the experience for us then to go towards the more complex use cases. that would yield high value as well.

So what were some of these use cases? So we focused really on our customer and we talked about the employee base and a lot of our employees who are now joining the company to support our growth are from a demography that loves technology. That's the grown up digitally native, loves to use generative AI really and so that we feel was a perfect fit for us to enable them to be able to support the customer better optimize communications to the customer. And more importantly, with generative AI, understand the customer sentiment, understand customer satisfaction and drive actionable insights from that.

So we put the customer at the center of these use cases as we develop them and building out what Girish had said, I would summarize by saying we had four or five categories of use cases models. The first was around summarization red writing. We have a lot of complex policies, business rules and generative AI can be particularly useful in summarizing and bringing the relevant portions of that information policy and business rules for the employee and the team member to be able to present to the customer that we feel is going to bring significant value.

Second is around the conversational, the chatbot type use cases. That's more interactive one on one with the customer. Again, we feel there's a lot of value for generative AI to summarize and bring the information back to enable this one on one conversation, whether it's by an agent or with our digital tools, including our award winning United mobile app that really can use this technology to provide a lot of value to the customer. And the employee third is around sentiment analysis.

Like I mentioned, we have vast rows of customer feedback that come in and use this knowledge base, use our platforms to be able to glean from their real actionable insights that will drive forward more projects and more initiatives that can improve customer satisfaction and customer experience.

And the fourth is around really the more emerging area of multimodal LLMs if you will. And that's where things like text to speech and speech to text and, and really a summary of all the other use cases that I mentioned can provide a more immersive experience, whether it's the customers or our employees to be able to improve customer experience. And that's an exciting frontier if you will as we see going forward um in this space.

Now, now let's talk a little bit about how we believe Bedrock can enable these use cases to be brought to life and gain fruition with. And the first theme really and the recurrent theme in a sense is for us around responsible AI and security, right? And, and a lot of this is what Gish had mentioned as well. The trust and the belief that our data will not be used to train foundation models. Our trade data is ours and that's really a key for us to be able to have the peace of mind that it is not going to be used for other purposes.

Second, of course, is to enable the democratization of generative AI. And we believe at Bedrock with the choice of the models that it provides, it will really help to get AI spread across the enterprise, which is key to delivering the business value.

Third is around business efficiency. We believe that there are as we modernize our, our, our way to the cloud, a lot of our legacy platforms, software can be transformed using generative AI and we believe big area of value as well going forward and last. But I think most important what really Bedrock provides is this model choice and customization.

And that's where we don't yet know what we don't know. And to have this choice at our hands to be able to experiment, play with it and continuously improve the output coming back from these LLMs is going to be key in the intermediate and the long term to generate value for United.

So now let's bring this back to where we had started with in terms of where this is going, what's a path forward, how do we think this innovation will work? And the key theme again is we're using Bedrock and partnering with AWS because we believe that they will continue to innovate with Bedrock and then we can focus on how do we generate value with the data that we have and the use cases that we will create an idea upon.

But there are three categories that I want to talk to here. First of course, is again the recurrent theme around responsible AI and security. And we are putting the right guard rails in place to be, to be safe and keep our customers and employees safe as well. And we are working on things like prompt engineering. Now, Girish talked about retrieval augmented generation. That's going to be a key focus for us as well to how we add incremental content and knowledge basis to the data from foundation models to make sure it's providing responses and we can fine tune these models for our specific purpose and value.

The second is around the platform and this is really where the optimization needs to happen through this presentation. You haven't heard a lot about cost, but that's going to be a big important consideration for any company including ours. And so cost optimization is going to be a key component as we look forward and how that happens as these tens and hundreds of use cases come to market. We will have to manage this cost and we believe that that having that top of mind is going to be important as we go forward.

So things like hardware optimizations having the right chips for the right use case as well as the database optimization is going to be important for us going forward in terms of also model monitoring, as well as carrying out these tests to make sure and evaluations, to make sure that this is generating the value that we think we had envisaged and lastly and, and and equally important as any is the models themselves, right?

And then we talked about the choice that Bedrock provides in terms of customer models and more that will come in the time going forward. So we are particularly interested and excited about this mu multimodal opportunity as well. And then also working towards fine tuning some of our the model here for United trained in specific use cases that will also help us optimize the costs going forward that we don't have to use larger trained models that are that that are out there as well.

So that's those are the areas that we'd be looking to going forward um in in the near future. Again, bringing this to close, our focus is going to be on our use cases and our data. While we partner with AWS on enhancing and working towards a more innovative Bedrock platform, so that our data engineers and our ML engineers can focus on partnering with our businesses to make sure that we create business efficiency and more importantly, create the customer experiences that enable our customers to keep coming black pack and flying on United airlines. That's the story thank you. And with that, I'll hand it back to Mark.

Mark: All right. Uh this has been great. Uh thank you Sanjay. You know, you can hear Garish and I talk all day about Bedrock. We'd love to, but it really hammers it home when you get a real customer explaining how they're going to deliver on generative AI and how Bedrock's uh going to help. Uh so hopefully we've uh hit the mark here. We set out to not just explain pieces of Bedrock, but to do it in context to be able to tie all these pieces together. A lot of the core capabilities, a lot of the new features that were announced this week and put it in the context of generative AI tasks that are pretty common and use cases that you, I'm sure you're seeing and it was great to see United uh kind of reinforcing all of that with their platform.

So, a quick call to action here. Uh if you have not yet tried out Bedrock, uh please do so check it out. You can go to the AWS console uh and use that really uh easily there's uh workshops available uh to do some self-service training. You could even visit uh Party Rock and, and try out creating your own generative AI apps using Bedrock.

So, uh if you're already using Bedrock, we'd love to hear from you uh about what you're running into what you like, what you don't like what you want to see in the road map, uh, as you know, at AWS, we drive our development based on all of that feedback. So very much looking forward to that.

So with that, we've, uh, we're kind of out of time for questions. So, uh, we're gonna close here and, and we're happy to, uh, talk to you, uh, after, uh, outside. So, thank you very much.