How to deliver business value in financial services with generative AI

John Kane: Hi, I'm John Kane. I lead the market development efforts for AWS and financial services and I wanna thank you all for joining us this evening. Uh it is my absolute pleasure to introduce uh our session this evening, how to deliver business value in financial services using generative a i.

One of the most fantastic things about re invent is for the ability of customers to kind of meet each other and kind of share those journeys and the experiences that it's taken to take advantage of new technologies. And if you're like almost all the customers that I've been talking to over the last six months, you too are on that journey to figure out how to derive value from generative a i and i am very happy to have three panelists with me who are sharing their own experiences in how they approached, generate a i from a business proposition perspective, how they looked at it within the firm to make it successful, some of the challenges they faced and the lessons they've learned to actually how to drive those applications into production and how that is ended up benefiting them.

And i can't think of three better speakers to kind of join me on stage tonight. So thank them very much.

Now, it's not surprising that generative a i has become such an important topic. Um when we think about how much artificial intelligence itself has worked its way into the value chain of financial services, everything from on boarding customers, matching images with machine learning technology, using text uh uh using artificial intelligence to extract text from documents, doing credit checks, automating lending algorithms. It's all been incredibly beneficial to the industry as a whole.

But now with the introduction of generative ia i the ability for single sets of tools to do just those things and even more has captured the imagination of all our customers.

Now, some of us have just been introduced through the chat interface being really impressed with kind of what gp t can do from a functionality perspective. But it's really bringing those technologies into enterprise processes, unlocking the real value and benefit of using large language models and foundational models to actually drive efficiency within the industry. And i think it's why it's been such a top of mind conversation, not just from technologists, but from boards of directors and companies because they see the ability to transform the industry and they don't want to be left behind.

And we're very fortunate to be able to kind of be still in the early stages of adoption in generative a i i think we've all seen the power of generative a i evolve just from a model perspective, you know, where gp two was then gp three and then four, even what we've seen from claw and now the release of claud 2.1 just the power of those models and how much more efficient they've become. But we're still at the early stages of what's going to be continuing evolution in this space.

We've seen the investments in some of the announcements that aws has made from a silicon perspective, bringing even more compute power to ever larger models and other greater capabilities to the fore cost effectively, providing that choice that allows our customers to fit the right use of gen a a i to write business models and how to actually drive value there and this will continue.

So today, we're going to see how we're already seeing the benefits of gen a i. But really, this is the beginning of a broader transformation across the industry.

Now, we've been seeing customers experiment over the last six months in a number of different areas and broadly, if we were sort of excluding sort of what we're seeing from a code automation perspective, they fall into some broad categories using generative a i, particularly with sort of rag patterns to really take advantage of unstructured information across the organization, to find insights into greater more complex documents to really drive effectively effectiveness and efficiency among staffs and how they help within the organization to drive operational efficiency.

Some of that works its way into the product process. As general a i gets embedded in the value chain. I think you saw an announcement about q this morning. Um as amazon q gives you an indication that more and more of these interfaces will be embedded in other applications and you'll get that natural benefit of gen a i within the application set without even knowing about it.

And then lastly, those capabilities really allow firms to take that ability and then actually make a more beneficial experience for their own customer. Now, in a regulated industry, most of us aren't able to expose that kind of generative a i technology directly to our customers. But we do have the ability to build human in the loop systems to make all the people that work with our customers ever more effective.

Now, it's my absolute pleasure to introduce my panel. Uh janet walden, assistant vice president n had global strategy and enablement at sunlife. Jen is gonna talk to us about sun life's own journey, how they've approached generative a i, how they actually thought about it from an organizational perspective and how they took their large number of po cs and got to a smaller set of business applications that will drive value then jeremy, but who's head of anti money laundering product development at verin and nasdaq company will talk to us about how they're using generative a i technologies to allow them to more effectively fight financial crime.

And then lastly, and certainly not least zach anderson from natwest uh chief data and analytics officer. He's gonna talk to us about how natwest is using personalization with gen a i to improve the customer experience. And that's an area that our customers have really been focused on significantly over the last two years.

And without further ado, i'd like to welcome janet walden to the stage. Thank you.

Janet Walden: I'm excited to be here as a global strategy leader. I often get asked, how do we incorporate innovative technologies like generative a i in such a highly regulated industry? And my answer is typically, i don't think we have a choice. Generative a i is here to stay. So to get started a little bit about sunlife.

So sunlife is a financial services company. We offer wealth management insurance, group benefits, group retirement, and even virtual health care. We've got about 30,000 employees around the world united by our purpose of helping our clients achieve lifetime financial security and live healthier lives as a digital company. When it comes to new and exciting technology like gen a i, we know we need to invest to achieve our purpose.

So i'm gonna tell you a little bit about some of the main stops we've had on our journey. So far, we've brought people process and technology together to help us understand where the potential value lies with generative a i in our organization. I'm also gonna share with you some of what we've learned in hopes that you can apply that within your own financial institutions as well.

So what we did first that set us on the right path and really accelerated our efforts was bring business leaders a i professionals, innovators and control partners together to create that healthy tension between innovation and managing risk in such an uncertain space. We brought legal risk, compliance, privacy, security, everyone together. And we created a set of guiding principles that gave us comfort of a space where we could experiment and know that we were managing the risk of the unknown.

This community started with about 10 folks and i'm very excited to say that it has now grown to over 900 people across, across the world as a result of that risk and control framework. we now have 10 simple questions that we ask ourselves at the start of every experiment. it helps us make sure that what we're embarking on is safe and that we've got a risk and control checklist that we can use to make sure we understand the d os and don't of of how we've proceed once we had our community in place and we had our framework for experimentation.

We then needed to understand what problem we were trying to solve. We very intentionally decided to test and learn with our own employees. We focused our early experiments on employee productivity and how we could improve productivity by how people do their jobs.

So it could be cogeneration for a developer. It could be call summarization for a contact center agent. It could be sentiment analysis for a contact center manager who is using that information for coaching for an agent.

So with our framework in mind, our community in place, one of our guiding principles was show, don't tell, we knew we needed to be able to show our business leaders the possible value that we could generate through rapid experimentation. What that meant was we really needed to work differently.

Sunlife historically hasn't been great at rapid prototyping. So with the help of our aws account team, we adopted an innovation delivery technique called an experience based accelerator.

So what an eb a does, it allows us to get a small group of cross functional people together in a room for a couple of days to focus and solve a problem for this to be successful. The teams need to be comfortable with ambiguity. They need to be able to fail fast and they need to be able to work through the inevitable roadblocks that come up with any new technology.

There's three things that i really love about an experience based accelerator. One is they're super focused. You've got all the right people together in a room with, with one problem to solve the other is they're global and cross functional. One of the eb a si was on recently, we had representatives from america, ireland, canada and asia all on the call together for one day.

And the other thing i love the most is they're so fun. I've seen props ranging from cow bells to mad scientists, lab coats to virtual backgrounds. We've done it all. Um, really great way to spend a friday night for those of you in that time zone.

So, well, the experimentation is great and we've had a lot of success with the eb a framework so successful. In fact, we've been able to do 14 of them in six months, which i understand is unheard of. But what we need to do now is figure out how we take the value that we've been able to learn about through those experiments and now bring them to production.

And part of that is the decision of what not to do is just as important as the decision of what to do. So we created this flywheel that says ideas are gonna come into our system. We're gonna identify those high value opportunities, we're gonna rapidly experiment and we're gonna throw out the stuff that doesn't work and the stuff that doesn't work typically we found has kind of set in two buckets.

One is not all problems are solved with generative a i a lot of them are solved by traditional a iml or other application services. The others are things that quite frankly were just too risky. We didn't know yet enough about the enterprise versions of this technology to feel comfortable moving forward.

What we have to do now is figure out the best way to get these experiments that we've proven valuable to scale in the hands of all of our employees. So that's the people in the process.

I'm going to talk a little bit about the technology. So our early investments in cloud infrastructure have proven to be the foundation for adoption for generative a i. We've been so fortunate to be early adopters of new technologies like bedrock and foundation model hub as they've come available.

So my favorite example of how we've matured alongside the technology is our pattern for chat bots. So back about, i wanna say 56 months ago, we created a, a pattern of foundation model hub a flan t five ll ma simple user interface and 90 documents indexed in kendra. And we did something unheard of for sunlife. We gave this beta to all of our employees.

So for anybody who's ever used chat gp t, i'm sure you could imagine how disappointing that was. So over the course of the next couple of months, the newer versions of enterprise ready generative a i technology started to become available.

So we took that same pattern. We added more documents to it. We um went up to close to 1000 documents indexed in kendra included some of our hr policies. We um kept the u i largely the same. We moved to bedrock and we switched to a bigger llm quote, we felt like this was a great improvement.

Again, put the beta in the hands of all of our employees still not quite what they were looking for. So we're very excited about the third iteration. And keep in mind, this is three iterations in six months for sun life. This is a big deal.

So we've got the third iteration that's actually uh coming out as a beta release this week and we've actually eliminated the rag pattern. We now have a simple u i send over bedrock and clo that allows our employees to have that chat gp t like experience that they're familiar with.

So we know we can manage the risk by using bedrock in our vpc. And we're giving our employees a tool that they can really experiment with and get to know how generative a i can help them in their day to day jobs.

So through our experimentation, we are starting to see three buckets of value emerging. The first is around cost avoidance. So by using gen a i to solve business problems, we can potentially avoid some very costly projects. By improving employee productivity, we can redeploy people to much higher value work.

And we know there's value in improved experiences, employee retention, client retention satisfaction scores, the more we can improve experience, the more um valuable gen a i is to our business.

So what do we learn in order to highlight what we've learned, i love using the example of sunny and not just because sunlife has given him such a creative name. So our business partners um in hong kong asked us if we could make sunny conversational. So sunny is a q and a chat bot that sits on our customer facing uh uh website in hong kong.

So we were successful in making sunny conversational in in our test environment

But we learned a few things.

One is in this case, Sonny needed to be able to interpret both English and traditional Chinese characters. So what we learned is not all LLM’s are created equal, so not every LLM could respond in traditional Chinese characters. So through the power of Bedrock, we were able to actually experiment with a few different LLM’s that offered that and look at the accuracy of the responses in both languages.

What we also learned is the RAG pattern doesn't prevent all hallucinations. So well, we had the RAG pattern in place. Sonny continued to loosely hallucinate to the point that we weren't comfortable with um putting that in front of a client.

So what we also learned is for these use cases where accuracy really matters. As an organization, we have a new muscle to build, we need to get good at prompt engineering model tuning and things like that, that help us make sure that the output from these models is the level of accuracy that that we expect.

So where do we go from here? We've been leveraging this concept of the three horizons of innovation and we're in horizon one, we're testing, we're learning and we hope over time is our willingness to accept risk and the technology continues to evolve and the regulations and the industry continue to evolve that we'll work our way through these horizons.

I was asked actually this morning, how do you know when you've graduated from horizon one to horizon two or horizon two to horizon three? And I said, I will let you know when we find out, don't know yet.

So what I might ask of all of you today is even though we are in highly regulated industries, even though we're, we're working in financial services around the world. I encourage you to play with this technology to experiment, to recognize that the enterprise versions of this technology can be very powerful within a business.

How do you start? You can build a community around learning about generative AI, you can contact your AWS account team and try an EBA. You can even go to Party Rock and build your own gen AI application which I did in three minutes. It was pretty cool.

So as a takeaway, gen AI is here to stay and my two colleagues are gonna tell you a little bit more about how they've done it, Janet.

Thank you very much. And if I could only echo one thing from your presentation. There's so much to take away is the importance of experiment iterating with a very thoughtful mindset. Um so many of the conversations I've had with customers who have been successful in this area have been a willingness to experiment to find out the known unknowns, right? To understand what they do and don't know about this technology and really appreciate you kind of walking us down that.

For a next presentation, Jeremy will come on stage and I, I will say I'm a little fanboy of this one because I used to have this job at JP Morgan helping on the financial crime side, slightly different. But I understand sort of the challenges of having to deal with large data sets and leverage them to actually report out from a regulatory perspective and to actually sort of see what you're doing here is super interesting because it's kind of personal and I know my team would have been much happier at JP if I had this then for sure.

Thank you very much.

Hey, everybody. My name is Jeremy Butt and I'm an AML product lead within product development at Verin, a Nasdaq company. And today, I'm here to talk to you about how we're leveraging generative artificial intelligence to fight financial crime.

So the agenda for today, we're gonna talk a little bit about who Verin is and what we do. We'll get into the industry problem that we're seeing today and how we think that generative AI can play a key role in solving that problem. I'll take you through a little bit of our approach and the benefits of the approach that we've taken and then we'll get into where we are today and talk a little bit about some of the successes that we've seen.

So first, who is Verin and what do we do? We're an industry leading anti financial crime management solution provider and our purpose. Mystified crime.

Financial crime is a multi trillion dollar industry. It's often misunderstood as schemes played out in, in ledgers and spreadsheets, transactional clinical distant. But this could be not anything further from the truth as it underpins many of society's most insidious crimes, human trafficking, terrorist financing, elder financial abuse and fraud and scams that impact the most vulnerable within our society.

It's a deeply human problem to solve it. We need to embrace innovation, leverage the power of large data and bring aboard all key stakeholders to ensure that we can better protect the global financial system and the people and communities that it serves today.

Verin’s consortium data set is populated with data from over 3,800 financial institutions. This is a massive data set that consists of both transactions as well as profiles for over 575 million entities across North America. Our team of over 600 technical experts have developed a deep understanding in artificial intelligence as well as the financial data and crime that can hide within it. This allows us to develop an understanding and expertise in banking, as well as artificial intelligence and other new emerging innovative technologies.

Verin has been fighting financial crime for over 20 years now, delivering effective AI solutions. Our core technology and analytics have been based on artificial intelligence. Since Veron's early beginnings, we offer a suite of products across our platform to help financial institutions monitor, detect, investigate, report, and more importantly stop financial crime. This includes fraud detection and management ESAML compliance and management, high-risk customer management and sanction screening management.

As we innovate within each of these product segments, we're constantly engaging both our customers as well as key stakeholders to understand how they work, but also what they're seeing in the industry today.

One of the main things that we learned early on was a lot of time is spent on manual processes and tasks. These tasks can include reviewing customers, transactions, counter parties, leaving the platform for intelligence gathering of unstructured data, trying to understand the relationship between a counterparty and a customer finally decision the activity. And then after all of that, having to summarize all of the data that was used all the data that was collected, what the decision was made and why that decision was made.

This adds up to a whopping 60% of time of an investigator. This is time that's taken away from actually analyzing and investigating activity and looking for potential crime.

The problem that we're seeing is that financial institutions want to shift, will increase investigation efficiency and shift human resources focus back to what they're trained to do and focus on fighting financial crime. Instead of following these manual processes.

We've been working with a lot of our customers to develop solutions for these efficiency gains and improvements have definitely been made. But reliance on these manual processes and procedures are heavily embedded within the financial system.

So this includes intelligence gathering which requires interpreting storing and reasoning over unstructured data entity disambiguation, which requires understanding and reasoning over similarities as well as differences between multiple entities and summary writing, which requires a holistic understanding of what's being summarized.

Now, the majority of today's manual tasks have been exceptionally hard to solve with traditional machine learning and artificial intelligence solutions. This is because they require human input and human reasoning that has been previously exceptionally hard and sometimes impossible to fully replicate.

This includes reading, unstructured data writing or content generation and more importantly, having a contextual understanding and this is where gen AI can play a major role in generating huge cost reductions and allowing investigators to spend more time doing effectively doing what they're trained to do, which is investigating financial crime.

Gen AI and large language models play a key role in solving a lot of these inefficiencies, unstructured data can be difficult or inefficient to access and interpret LLMs can reduce manual workloads by summarizing low value information for the sole purpose of documentation. It can expedite decision making with typology descriptions interpreted from evidence models and it can utilize unstructured data that may be critical in making a determination of suspicion.

So here's a diagram that is an illustration of a rudimentary version of the architecture that we've built to introduce gen AI into our product. Today, I will note that it is quite easy to stand up a gen AI product with the help of Bedrock and complementary AWS services.

So I'll just walk you through the execution flow. Uh so we'll start here with System One and Service A specifically on a schedule um that service requests work to be done and places requests on an SQS queue this cue um then each one of those requests on that cue invokes a lambda. This lambda has three main roles. It does some data collection for retrieval, augmented generation. It does LLM inferences and then finally, it stores those inferences into a data store for later retrieval.

So it starts off with reaching out to a data access API to pull in some data that the, the model then can uh reason over once that data is returned, we pass it off to a SageMaker endpoint here, we're using a, a few very targeted domain specific LLM’s that are much smaller than a foundational model uh to do some preprocessing and pre filtering of the data that we're going to provide the larger foundational model and we'll get to the benefits of that in a second.

Once those results are prefiltered, we pass that off to the Bedrock. Uh VPC endpoint, this provides access to the Bedrock service account. The request is then passed over to the Bedrock endpoint service and then on to the, the request was passed along to the foundational model that's hosted within Bedrock. This response is then passed back along to the lambda and stored in a data store specifically for us, it's RDS.

At some time later, we have System Two Service B. This could be another system doing some preprocessing or some analysis or something like that. Or it could be access to a UI or a platform to provide data to a user. It reaches out and invokes a lambda to request access to the pre computed LLM inferences that lambda then queries the data store and returns those results.

So as we discussed, uh we, while building this, we actually found two main techniques that really, really helped us minimize our tokens and optimize our LLM usage, which then obviously correlated directly to optimizing reducing our product cost.

The first one batch preprocessing, it allowed us to preselect organize batch and schedule the work to optimize that the LLM usage and Bedrock usage. We also leveraged the smaller LLM’s that were more domain specific to handle some data preprocessing which minimized and optimized. the tokens going to the large foundational model within Bedrock.

These two optimizations accounted for an 83% reduction in input tokens going to to our foundational model. And this directly correlates to product cost is huge. Now, that's a huge win. And we did have some good, good wins along the way, but we did encounter some problems.

I think the biggest problem that we encountered and i think it sounds like a few other people are encountering this problem as well. Hallucinations. So this is where the model makes up things that doesn't exist. It doesn't make sense or just has no context to the prompt to overcome this. We created truth data sets for validation and testing. And then we leveraged those during our prompt engineering technique or iterations to validate and test those approaches and see their performance and their effectiveness.

I want to stress here that the key to rapid iteration is starting off with these validation and testing truth data sets. Some of the more effective prompt engineering techniques that we used and are actively using today includes multi shot. So this is when you provide the model, multiple examples of both the input as well as expected output. It helps the model follow an output constraint and reduces output formatting based hallucinations. For us, we specifically needed the output to be in a json format for another system to interact with it. We provided the model multiple examples of a clear json structure to follow within the prompt. And it did another technique we used was thinking step by step. So this is when you give model, the model hints to work in a step by step manner towards an output objective. This provides the model more clear context around the direction it should be taking to achieve that output goal were able to help the model and guide it through the steps that an investigator would take today if they were doing the task, manually, assigning roles or roll prompting was another technique we used. This is when you tell the model its role or who it is in the situation. This gives the model more context to the steps required to achieve that overall goal. It also gives the model a little bit more context around the industry specific meanings of words that could be within the prompt or within the data. This helps reduce hallucinations that are based around the semantic complexity of natural language. For us. specifically, we told the model that it was an investigator within a financial institution. This gave the model more context, not only just around the prompt and how to understand that prompt, but also the data and how to um how to reason over that data and what it was actually looking at and finally retrieval augmented generation or RAG. This helps basically you give the model specific documents to reason over it limits the context of the model and it reduces the possibility that the model will actually get confused and start making things up. We found that giving the model the most accurate and recent data is key for getting truthful and trustworthy results.

All of these prompt engineering techniques as well as a good few others really helped us significantly reduce the hallucinations that we were seeing. We found that a key, the key here to a great generative AI product is having an iterative prompt engineering approach.

Today, we're seeing really good gains in investigator efficiency in some of our best case instances, time spent on these manual tasks is reduced from 60% like we saw before all the way down to 5 to 10% as well as keeping the investigators focused within the platform. This gives time back to the investigators to spend more time on investigating the activity or maybe even reviewing more activity for financial crime.

Verin’s experience in applying AI to financial crime. gives us the right combination of skills to effectively deliver a gen AI solution that works

We were able to successfully shift human resources focus to fighting crime instead of following manual processes with the help of Bedrock and AWS. Thank you, thanks Jeremy.

It's amazing how many of those processes all financial institutions have to go on the back end, particularly, one is from regulatory filings, developing loan documents. That ability to bring insights to disperse data sets is just sort of the beginning of what's a broader trend.

As I mentioned earlier, so much though, of what's been going on, at least from a retail and customer perspective in financial services is about personalizing and tailoring kind of user experience, meaning that customers in a more meaningful way that's more relevant to them at the time, they're engaging with financial services.

And Zach, I'm thrilled for you to kind of take us home in your experience and kind of creating more personalized experience at NatWest. Thank you very much.

Thank you. I'm going to talk about something different. I think so many of us and actually many of the things we're doing at NatWest are worked on are working on augmenting our colleagues and making things more efficient in the background. But I just have this kind of sense that the real changes that Jai are going to contribute to financial services and most other industries actually aren't going to be there, that's going to help us take out costs become more efficient, do our jobs better. But the differentiators feel like things where the stochastic parrot can come up with an idea we haven't ever seen before. And that's what's really interesting to me.

So I'm going to talk a little bit about how we're using it on our personalization journey. So let me first tell you about NatWest. So we're a, if you don't live in the UK, you probably won't have heard of NatWest, but we're one of the four major banks in the UK, we've got about 19 million customers. And we've been on a huge journey like many other banks in the world to become digital. And for the most part, we are now, especially in our retail bank, primarily a digital bank. 96% of our sales are done digitally. We are focused on our app as actually the best, the biggest and best branch that we have and we do a ton of our business interactions, customer service all through our app.

So a ton of our capabilities now are digital. The interesting thing when you become digital and you realize that your interactions aren't expensive anymore, they're really cheap. What you want to do is have more of them, not less for years. Financial services tried to figure out how to get people to not go into the branch, not call the call center. And instead now what we would like is actually more people to interact with our app, for us to be able to provide help and and other ways to manage your financial financial, to get to better financial outcomes for each of our customers.

But the problem is is that we're a financial services company. We're not a content creation company. So we have this giant bottleneck of, we want to create more and more personalized content, more and more helpful ways to work with our customers. But content creation becomes one of the biggest bottlenecks.

Luckily, for us, we've been working on this problem and how to use AI in it before gen AI really for the last three years. So we've been building really strong foundational capabilities. We started using SageMaker three years ago, we moved a ton of our data and, and key elements of our customer data in particular to the cloud. And then we've been hiring and building a machine learning and data engineering group within the organization that's embedded in each of the businesses.

So as the, as NLP started to change over to LLMs, we've really been in a good place to be able to make that shift in the last year. We've really accelerated, built all of our pipelines up on SageMaker so that we can take models to production very quickly.

So four years ago, for example, when we would build a new model in say the fraud or fin crime space, it would usually take us, you know, 6 to 12 months to get from idea all the way into production past all the regulations, past all the model governance and actually have it being used. But that's just much too slow in a space like fraud, but it's also much too slow when you're trying to react and build content for customers in your app.

And so as we built all of those pipelines and model risk controls into our SageMaker and around our SageMaker environment, we now can do get new models to production in a couple of weeks and kind of on average, usually in a couple of months, which means that we can react and build many more models with the same number of data scientists and actually get to market much faster as LLMs became a thing probably for many of you that the public engaged in.

We had a massive poll of demand from our organization for use cases. So we now have over 100 priority use cases identified across the bank which we are working on. We've done 10 for initial focused and done PC this year. And we think we'll get about a 10x return on that entire portfolio of work over the next couple of years as we implement it, we actually just this week put our first customer augmentation or I mean sorry colleague augmentation gen AI model into production and it's being used by some of our colleagues in coots.

But the really interesting one to me is this is the creative space that this enables. And really I think that that's that's the key outcome. So right now, probably 90% of our use cases are on the colleague enablement side, new ways of working, helping people actually produce code better produce, do their work better actually very similar use cases to to some of what was already presented.

But the interesting one to me is this customer's side, how can we help customers, how can we improve what they're doing with their data and what we can do with our advice and our support of them. And really, I kind of look at this personalization move over a couple of phases.

So, you know, pre 2020 I think everybody kind of person went to their advertising agency and said we want copy that speaks to, you know, 18 to 34 year olds with this much money that do these kind of things about this product. So big segmentation approach, create a couple of segments, go to market with that.

I'd say in the last three years, we've seen a pretty big evolution of AI assistance variation creation for us to do testing. So an initial concept and then variation on that using various tools to try to see what, you know, switch a word in or out or to improve the salience of it. And then also personalization around specific things, of course, using somebody's name in an email, but also maybe something like if we're referencing a transaction or a type of transaction, tell people about what that transaction was, the name of the merchant that they bought it on or something like that.

And that kind of personalization is where we've been going. And frankly, for NatWest, that's been a really big gain we've gone from, for example, in our app, about 250,000 clicks and interactions with our prompts in our app. to over 15 million so far this year, over that same three years. So a huge benefit in terms of the engagement that we're getting from that kind of normal personalization.

But now we're starting to be able to use the LLM's to reduce the segment size to reduce the content and to create more and more personalized messages so that we're not swapping out a word or uh or an amount or a name, but we're actually shifting the whole message for each customer before it goes out.

And I'm going to show you a couple of things as we go forward. So we have a new tool which we've been using. And this is we've been experimenting with using the LMs to create content itself, which this will show you but not in very much detail. But what I'm going to show you is actually the way that we're using the LMs as a guardrail in our content creation process.

So for both human created content or AI created content to check it against our brand guidelines, the rules that our regulators put on us about words we can use and how we can talk about our products and 10 and then to actually suggest changes to them as a process. So this is a simple little tool the team built, but it is actually already being used to review our content in the bank.

And so you'll see, we don't show you much of how to do the personalized content. Right now, I won't go into that, but we take a set of content and then we do a review of that content. So copy and check it against all of our brand guidelines and our rules inside of the company, including our regulatory guidelines.

We then suggest here are the places where it would violate those or there could be changes and then we would rewrite the message with advice on that. So we're essentially using, you know, balancing LLM inputs in order to create and then review our content against a set of guidelines.

And then of course, we have right now, of course, a human in the loop to check any messages that are created. But the benefits of this have been really interesting. So one, it's given us some confidence that we can review a much larger set of messages and then give them to our, our control officers that make sure that our stuff is correct.

It's also actually found interestingly, we built this tool to check the M's generation of content. When we started using it on the human generated content. We had a lot of violations there too actually, which was quite interesting. Maybe not as many but not, not law violations, but I'd say brand voice and guideline violations more of.

Interestingly now we're actually using it to move into, into production and to be able to review everything that we're doing. And adding more and more controls into that process. So the kind of process that we're using is generate the content and we're doing a bunch of prompt engineering in the background to generate our content and and create the content specifically for our customer sets, then check it against brand adherents, tone of voice compliance checks, do an approval of that.

And right now that includes a person, but we'd actually like it to get to move away from a person or to be able to spot, check it over time and then cue that message for contact in the channels. And the nice thing is here is you have lots of as you have lots of different channels, you have kind of different context windows that you have to put channels or that you have to put contact into.

So, you know, in a, in an SMS text, you only have a few lines in a big banner on your website, you have a whole paragraph that you can use. And so you might want to adjust actually the number of words in it, the text in it. And this allows us to create all of those type of controls also into it.

So we say we want something that's no more than two sentences long that's targeted at this customer about this product or in this situation. And that kind of starts the whole process, which is a pretty powerful tool set to generate a whole bunch of content that's of a similar theme, but actually has different requirements based on where you need to put it.

I mean, as you think about the number of channels that you want to go to customers, that you want to go to tests that you want to run, you very quickly see that you have a lot of variation in your content generation that you have to manage.

What's been pretty amazing is is that our initial work on personalization has been able to help us drive a big increase in the number of savers across our bank. And we've been doing a lot of work in that space. The second is, is that we've been using it on savings accounts applications. So new customers coming in and how we drive some of this out into the web.

And then finally, the first messages that we've put live for our no, our credit score product, we've actually seen a four x increase in the control groups between human created messages and the best of the LLM created messages, which is really interesting.

I think that the M and the hyper personalized messages are performing that much better. I personally, I'm kind of surprised by that level of gain. But it's pretty interesting. It's the first, we're still early days on it. So as we build up a better bigger knowledge, corpus knowledge on it, we'll get to finer tuning to more finely tuned messages both from the LLM, but also better comparisons across the data set. It's quite an interesting space.

And I think for me it's exciting because I think we're all spending our days trying to figure out how to control these things, how to stop hallucinations, how to make them act the way we want. What's interesting to me is is that actually they're quite creative things. Hallucinations are a good thing when you're thinking about advertising copy, you know, you don't, you don't go to your agency and expect them to look like a person in a suit, right? You want them to be a little bit crazy.

And when you're writing an advertising copy, you kind of want your LLM to be a little bit crazy too, but then of course, have a set of guidelines that controls it. And so I think it's quite exciting to find these use cases where the creativity that's coming out is more valuable than the kind of boring thing that we all want it to create in 90% of the use cases.

Thank you, please join me in thanking my panelists one last time, Je Jeremy Butt. Thank you so much for sharing your insights and experience. These sessions are just so kind of critical to building the community and financial services and I really wanna thank you for your time.

The panel has also been nice enough to take Q&A after the session off to the side so if you have any questions from today's presentation, please come join us to the side of the podium afterwards.

And before we let you go for the evening, if I could ask you just to fill out your surveys. We always love and appreciate your feedback on what we can do kind of better and what you appreciate from Re:Invent. So please take a few minutes to kind of fill out this survey.

But again, thank you for your time and thank you for joining us this evening. Have a good evening.