Improving patient outcomes using generative AI in healthcare

hi, good afternoon. uh, i'm rod trigo. i lead clinical informatics for academic medicine at aws. and today the three of us are going to talk about some really cool stuff that we think are gonna, is gonna be really impactful to patient care.

now, i came to aws a couple of years ago and before that, i was a pediatric critical care physician for 20 years. in that time, i got to see some amazing things. i got to see kids come back from the brink and, and make full recoveries. but i also saw instances where our system wasn't necessarily functioning as, as well as it could.

so i'm gonna start you with a story and i want you to imagine a little four year old boy, this could be somebody in your family, somebody, you know, but this little four year old boy is otherwise healthy, but he has seizures. now. fortunately, his seizures are pretty well controlled. he takes the medicine for them and does ok.

and so this little boy and his mom, they go to their neurologist to have his regular checkup and they look at him, everything looks good. they give mom a new prescription for his seizure medicine. give him a nice little piece of paper with the hospital logo at the top with a list of the medications and then send him home and everything's going great. he's had no seizures.

about two weeks later, one morning, his mom goes to wake him up, goes into his bed and shakes him to wake him up and he doesn't wake up. she shakes him a bit more and he still doesn't wake up. and after a few minutes of shaking him, she realizes something is wrong and she calls 911, the ambulance arrives, they pick him up, they take him to the local emergency department.

and when he arrives, they do all kinds of stuff. they draw blood, they do a scan of his head to make sure there's nothing going on in there. they even do a spinal tap to make sure that he doesn't have meningitis, obviously pretty traumatic for not just him but his mom. and about an hour later they get the labs back and it turns out that the level in his blood of one of the seizure medicines is toxic. essentially, he overdosed on his seizure medication and everybody sort of wondering why because his mom shows him the bottle and said this is what he's taking.

so they look back and she pulls up, she goes, reaches into her purse and pulls out the list of medications that they gave her at the last visit and she says, look, here it is. so i started looking at the medication list and it turns out that when the neurologist refilled the medication, he had changed it from a generic to a brand name drug, but it hadn't removed the old one. so this mom was giving him exactly what was that one that was on this list? she was giving him twice as much as he should have. and as a result he overdosed and all of this stuff that happened was because of that single error on that medication list.

so this is an example of a preventable medication error, preventable error that we see in health care. and unfortunately, this is all too common. and when i went to medical school, one of the first things we do when we go to medical school because we take the hippocratic oath and part of that hippocratic oath is primum non nocere, which means first do no harm.

now, while we've made a lot of progress since the times of hippocrates, we still see way too much harm in health care. patient safety really sort of came to the forefront in 1999 with the publication of to air is human uh by the institute of medicine. and for those who haven't read it, it was a very comprehensive report looking at patient safety in the united states.

and in that report, they estimated that up to 98,000 people in the united states die every year for medica medical errors. it could be medications, it could be all these other things, almost 100,000 people a year. that was almost 25 years ago. and we've certainly made some progress. but as recently as this last january david bates and some colleagues looked at inpatient hospitalizations in the us. and they found that one of every four patients uh undergoes some type of harm from patient safety errors in this country, one out of every 4 25%.

so you go into the hospital, there's a one in four chance that something bad is gonna happen to you due to an error now that not only affects outcomes, but it affects costs every time. there's those errors that prolongs hospitalizations that causes other procedures to happen. so that's a problem.

and when you look at how we're doing in this country compared to other countries, it's not very good. this is the commonwealth fund. they come out with their mirror mirror report every year. and ideally, you want to be up high that tells you that you're having really good outcomes for lower costs. notice where we are, we're in the lower right. we have a very, very expensive healthcare system with not very good outcomes. so it's a problem and we need to look at new approaches to doing uh to addressing this.

fortunately, unless you've not been seeing, not been looking at the news at all, we do have some new opportunities. artificial intelligence, including generative a i offers us some tremendous opportunities to look at these things in a different way. but we have to do it carefully.

now, when we talk with health care organizations all over the country, there's lots of different options for use cases in health care. we can use generative a i to help researchers query data more effectively, more efficiently and easier. maybe they don't have to go to the data scientists anymore. we can help clinicians take all of this administrative work that they're unfortunately used to doing now and automate that we can help summarize information so that if i'm a referring physician and i'm getting 100 page pdf, i don't have to go through it manually. i can do it electronically, lots of different opportunities, but we still have to look at that end goal in mind.

this is the quadruple aim. uh that really is the basis for what we do in health care. we can use artificial intelligence and work backwards with the goal of improving our outcomes, reducing our cost and in the process making the lives of the patients and the clinicians better all while having this foundation of fairness and equity.

so with that in mind, uh it's my pleasure to intro introduce chad vandenberg. he is the chief quality and patient safety officer at uc san diego health. we've been working together on some pretty innovative stuff in using artificial intelligence to improve patient safety. he's gonna walk through their journey at u cs d with aws, how they have used artificial intelligence to this point and then how we're looking to do it in the future.

all right. well, thank you rod and thank you all for being here today. um i uh walked in this afternoon, uh not knowing exactly what i was walking into. uh and it's a world filled with innovation. it's a audiences like this through many, many casinos uh trying to figure out how do we improve uh the world we live in. so thank you for being here. thank you for the opportunity to share some of our story with you this afternoon.

so, um i'm gonna share a few things uh some examples on how we're using the technology, how we plan to use the technology. um the underlying uh sort of approach that we are taking is uh everything must be operational, everything must be seamlessly integrated into clinical workflows. uh we can't have our providers spending any more of their precious time having to go outside of the systems they're ordinarily working in.

um and it has to have a fundamental impact on that quad quadrupling that you heard about earlier. so our chief medical officer and chief digital officer who is uh actually on his way to testify in congress around uh health care a i um has a has been quoted as saying that a i uh could be in as impactful as the introduction of penicillin. and i think that when we look back on this time, i think we're all going to agree because we are uh in my opinion on the precipice of fundamental shifts on the way we approach health care and in my world, in particular, the way we approach quality and patient safety.

so uh the university of california has a uh a long established relationship with aws. um we're partnered in a whole variety of ways. uh but uc san diego health is looking to push that envelope even further. we're trying to figure out ways again to leverage the technology to improve the outcomes for our patients. um we, we believe that uh with the technology, we're gonna be able to do some things very differently uh in the future. and i'm gonna show you a few of those examples here today.

um there's, there's a lot of quotes out there around a i and as we were sourcing some of those to put on the slide here today, uh i chose this one because there are some fears associated with a i, we have to be cognizant of what are we getting ourselves into. and so there's been a lot of talk about replacing jobs with a i. and i truly believe that the future is not about replacing jobs, but by uh putting the technology in the hands of those who can use it more efficiently to drive better outcomes for our patients.

so i'm gonna tell you a few stories. um i think you're all uh familiar with this little thing we went through called the pandemic. um we had uh the second plane uh carrying evacuees from wuhan uh land at the air station just north of san diego at miramar in uh february of 2020. um this for us ultimately led to us caring for one of those very first patients um in, in the country. uh and what that did for us is, and i'm sure for all of those in health care is put us on a pace of change that we had never seen before. we were forced to innovate then to integrate that change at a pace and a scale that we had, i don't think we had seen. certainly not in my career, my 27 year career.

um and, and what happened with this patient who arrived um like many other healthcare systems who ultimately cared for patients with covid. we spun up a command center. we had um leaders, including our clinicians, our pharmacists, our nurses, our physicians in the command center. thinking every day about how do we care for that next patient in a, in a new innovative, better way than we knew how to the day before.

so we had to deeply, deeply integrate uh informatics into this equation. so every day, uh truly, every day our dashboards were changing. um the technology supporting those dashboards were changing, we were bringing to bear data to help inform that next day's care.

um and one example from this uh is the way we leveraged. um and then brought to scale uh a algorithm, an a i algorithm that one of our uh uh physician scientist at uc san diego health had developed, he had developed an algorithm uh focused on detecting uh pneumonia uh through the use of a i and image recognition. the challenge that we had uh very early uh in our journey uh was that we had that algorithm on a desktop. it wasn't deployed widely

Uh we didn't uh we weren't using it widely. Uh but through some conversation through those discussions at our command center, uh it quickly became apparent that this technology that this in this uh provider had developed um would be useful uh very useful for us, certainly in those early days in detecting covid.

So we uh quickly uh spun up uh some support partnered with aws began taking those images uh sending them out to the cloud, uh applying the a i algorithm and bringing it back to our providers uh in less than a minute again. Uh there will be a theme through my conversation with you this afternoon, which is integration into workflows. We can't have uh our precious time spent going elsewhere. So we had to bring those images right back into the place where our providers will ordinarily look for them and then uh see what this uh technology could bring to bear.

And it was that innovation, that innovation at scale that allowed us very early on. And I apologize, i'm gonna take a quick sip very early on, uh detect uh what we suspected to be covid. And if you recall in february and probably even in march and parts of april, um testing wasn't really ubiquitous. We didn't have a lot of testing capability. We had a lot of treatment, things that we were trying to do. Um but we didn't have easy access and robust access to testing.

Now that changed pretty quickly. That was part of the innovation and the pace of change early during the pandemic. Uh but this algorithm allowed us uh through incidental finding and sort of direct uh uh inquiry uh to detect those patients who uh were likely to have covid. And then of course, we could get them into care, begin the treatments and therapies that were necessary to sustain their lives.

All right. So staying with that, um i wanna talk a little bit about uh impact um because again, uh and i probably will be ashamed for saying this here, you know, technology for technology's sake is maybe another toy uh for the shelf, maybe another opportunity to uh bring some innovation. But without the impact to provide our workflows, without the impact to the care we're delivering. And, and certainly without the impact to the patients uh, for whom we're delivering that care. Um this is kind of, uh, a fun exercise, but not one that's gonna fundamentally change what we heard at the start of the presentation this afternoon, which is 90 that number has actually grown with some of the uh newer studies, but it will say at least 100,000 lives lost to health care every year. And that's what drives us at uc san diego health.

So impact impact, impact. So this was uh some work by dr deef dr carlyle who wanted to study now that we were bringing this technology to bear. What kind of impact was it having? We knew we were getting folks earlier into treatment, but was it affecting our clinicians decisions? And in fact, it was uh about one in five of our providers uh attested to changing their clinical decision or changing the clinical direction and care for our patients uh based on the algorithm.

So this, this is actually uh an image uh of what comes back on those uh uh radiology exams uh where the lungs light up uh based on the algorithm. Uh i remember very early um in february march, uh walking through uh our er uh talking with our providers, talking with our nurses um and having conversation about that uncertainty that we were living in with the next patient who was arriving and them showing on their desktops. Hey, look, we have one here, right? Pulling up the image visually being able to detect this early, again, changed lives, certainly for our patients and got them uh earlier into care.

But it's about outcomes, right? It's about that care. Um and impacting the quadruple aim. Um so there are and were and probably continue to be lots and lots of articles in health care. Uh talking about the use of a i during the pandemic uh specifically with covid-19 thousands. Uh this was in march, i think when we pulled this, uh this stat statistic here with over 9000 articles mentioned, but only four of them spoke directly to the clinical outcomes that they impacted. And this is what motivates us at uc san diego health. We, we are on the active pursuit of that next thing that will allow us to directly impact the outcomes for our patients.

All right. So we are very fortunate um that we will be able to amplify and scale that impact uh certainly partnering with aws and, and others um at the great uh generosity of uh joan and irwin jacobs. Uh and under the direction of our chief digital officer, doctor longhurst, uh we are able to stand up uh what we are calling the center for health innovation. And its entire pursuit is to find these technologies and these innovations that will have an impact on outcomes, have an impact on the patients and the lives uh for those who are privileged to care for at scale this is um not about nipping at the edges which i if i were to be frank, i feel like we've been doing for 20 some odd years. This is about having a true seismic shift in the way we're delivering care to have an impact on uh the quadruple a really impacting our provider experience, our patient experience, uh the cost in which we're uh able to deliver that great care uh and certainly the outcomes for our patients.

So i'm gonna give you a few examples of uh how we've uh deployed some of this technology. And again, um with the underpinning of how do we do it in uh within the workflows in which folks are already operating it, how do we do it so that we can demonstrate impact um how do we do it so that we're um making fundamental shifts in, in the way we're uh delivering care for our patients.

So the first example i want to give you is our work in sepsis. And i say our, let me give the credit where credit is due. Uh doctor uh wordy with a physician. Uh sorry, excuse me, a data scientist. Doctor shama. Uh shame nama uh worked together on uh a iteration of an a i apsis algorithm. Um if uh any of you are um in the direct health care space and you use epic. Um and it's probably true with cerner and many other uh e hr s uh there are embedded algorithms already for some of these high prevalence diseases, sepsis being one. Uh but what we found um with this uh embedded uh algorithm is that it was alerting uh way too often. It wasn't as sensitive or specific as we would like. Uh it wasn't in our case leading to change in the way we were delivering the care, which again is the underpinning of the way we're going about all of our quality and safety work. It must do that.

Um so they uh developed this algorithm iterated on the algorithm. Um and uh one of the sort of key fundamental aspects of the development of the algorithm is to teach it to say, i don't know to actually allow it to not have to deploy a fire or an alert every single time. So when in a space of uncertainty or when it is not uh sort of uh at the degree of confidence that we would like, it doesn't fire, which is great because i'm sure you have all heard about alert fatigue.

And as we think about that quadruple aim and we keep bringing us back to that um it, you know, the provider experience, our nurses and our physicians don't want to see a bunch of flashing alerts that may or may not influence the way that they're gonna deliver care. And we see a lot of those, although we monitor them and then begin to turn them off at uc san diego. But we see a lot of them where folks are just like bypass, bypass, skip, skip, skip.

So it's important that when we develop the technology that we develop the algorithms like in this case, that it's that it's actually something folks are going to react to. And so this ability to teach the algorithm to say, i just don't know, i don't think this is a time to fire was a critical, critical piece of the success here.

The other really important part of this project was this pairing of a data scientist and dr nemati with a practicing physician in our emergency department, although he's board certified emergency department and our emergency care and critical care um and an incredible uh physicians uh doctor ward. Um but that partnership was immensely invaluable.

And then when you layer on a nurse partner from our quality and safety team who was there to work by their sides to say now that we think we've got the algorithm, right? What does it look like in application? How, what do we do with it? Where do we fire it? When do we fire it? Who sees it? What's it come? How does it come through?

Um and uh the way we deployed this sepsis algorithm is uh we're starting in the emergency department. And the outcomes you're seeing are from that deployment um where it fires in the emergency department to our nurses, not to our physicians. So our nurses are often the ones who are doing the bulk of the documentation of those vital signs and other things that are going to be uh key to the alert for sepsis. They get that initial alert right in a way that they are all enabled to take action out of it.

So that action is actually the ability to securely message our physician so they can go right from an alert to talking to the physician around the patient. Hey, i've got this alert. Uh here's the clinical picture of the patient. How should we proceed? And of course, the the care proceeds in a way that is uh evidence based and um has led to some pretty significant um results.

Uh probably the most important for our patients is we've seen a significant decrease in sepsis mortality. So early intervention because of the alert firing because our nurses and physicians are taking action on the uh the alert. Um we're seeing an actual decrease in mortality associated with this disease.

The second um uh impact associated with this algorithm is the ability to evidence that we are following evidence based care. Um and that is through the uh completion of the se one bundle and making sure all elements of the bundle, the getting the lactaid get lactase is getting the um uh antibiotics on board, et cetera.

So again, the physician partnered with the data scientist thinking about how do we seamlessly integrate um into our workflows. Uh to demonstrate impact another example um of our use of technology and a i in this case, generative a i uh is with our approach to uh the inbox fatigue.

Um so, uh our providers, um like others at healthcare organizations get hundreds and thousands of these inbox messages from our patients, critical, critical opportunity to sustain that physician patient relationship. Uh certainly helps uh our patients um of course, correct on their care, maybe avoid having come into the emergency department or uh other visits. Uh that mode of interaction we think of as very precious.

Uh but it's precious at a cost and the cost is our provider experience and provider experiences across the country. Um we uh know that most of the time spent responding to patient messages is what we call pajama time. So this is the time where our providers should be going home, perhaps enjoying some time with their families or friends and then uh getting a good night's rest to repeat, uh the immense uh care that they provide to our patients.

Um but what happens when we ask them to uh maintain that relationship with their uh patients in this way? Uh they're often doing that at, you know, either when they get home right after they put the kids to bed or maybe deep into the evening again, uh taking their precious time uh to sustain that relationship.

So we said, is there a different way? Um and you may have seen some of the headlines. I don't know if we have them here. We don't. Uh but some organizations decided that the way to curb some of this uh was to uh begin charging patients uh for these messages. So looking at it as though it were an encounter and there's been some success at curbing that.

Um for us, we felt like that might jeopardize that relationship with the patient. So we wanted a different way to approach it uh ways that would benefit the pro the provider and the patient.

So uh the, the one for the patients may be perhaps the most obvious, we don't charge them for these, these interactions. Uh the one for the provider is we're trying to make it easier and easier to do this. So we started in our primary care uh locations uh begun, have begun to deploy it in other parts of our uh ambulatory environments.

Um but to be able to leverage technology generative a i to look at the message, contextualize it against the medical record of that patient and then respond with a um a, a uh appropriate sort of draft for the physician to edit. So instead of them having to create it de novo, they're getting something to respond to.

Um we're optimistic that ultimately, and we haven't realized that quite yet, but ultimately, this will save them time. Uh this will be better uh for our patients. Um and uh enable again that continuation of uh the relationship.

So one of the interesting things from this, i think if it's on, yes. Uh the next slide is there were a lot of headlines about this and there's some studies around, you know, the capabilities of generative a i to generate these uh responses to our patients.

Um one of the more uh sensationalized headlines said something like uh that these responses were more empathetic uh and of higher quality than our physicians. ordinary responses, we interpreted that study a little bit differently.

Um, we think that given the time our physicians and the data suggest this, given the time our physicians would absolutely generate the same kind of high quality empathetic response. But as I started with, they're doing this deep into the evening after a long shift, perhaps with whatever pressures or stressors they have at home deep into their pajama time.

So what we seek to do is to cut down on that pajama time, enable them to do what they already would love to do, which is to generate a, a empathetic, complete high quality response to our patient questions, but to do that in a way that takes them absolutely less time.

So we have much to do still here. And I think that's another thing as we get into uh these kind of conversations is to appreciate that this is not going to be a, we're gonna put it in place and grand slam or whatever your other sports analogy will be um we have to iterate and we're gonna have to learn and we're gonna have to iterate and then we're gonna have to learn and then we're gonna get better and better at this. The technology will help us do that.

Um, we, we see uh that our patients equated a longer response to their queries as a higher quality response. Ok. So how do we, how do we use the technology to try to again, maybe shift that a little bit, maybe we can get a little bit shorter response. Uh but still have it be seen as, as high quality again, leveraging the technology.

How do we begin again, chip away at the time that the physicians are editing these responses, make the response that is generated for them of higher and higher quality. Um and then again, try to mitigate that, that editing. So they are still editing, right? We don't ever just send the response the physician has to take an active step. Um because we know we're early in this pilot. Uh we, we knew that that was a necessity um for the the quality of the care we were providing.

Um but again, what our goal is is to cut down on the amount of editing they're having to do, right? So that they can see the response the drafted or crafted response and have I'll say less interaction with it. Uh but still make sure it's, it's, it's what they would want to do in terms of a response.

Um, let me, let me shift gears just a little bit. Um, and, and talk about large language models because for me, um, when i, um, thought about this presentation, i saw the slides that doctor tara uh presented earlier and then thought about my own 27 year career.

Um, i, i don't, i don't, we haven't made the kind of dent that we, we all want to do in terms of providing safer and safer care. Have we gotten better? Of course, have we mitigated errors along the way? Of course, but have we eliminated them? No, we have not. Will we ever eliminate them? I don't know, i'd love to keep shooting for that.

Um and I think uh about the capabilities of uh large, large language models as a way a pathway for us to pursue that. So let me tell you a little bit about uh my team, um my team, like most quality and safety teams is responsible for um um responding to incident reports, um uh doing all of the quality public profile reporting to c ms joint commission and otherwise, uh supports peer review, uh examines uh incidents that happened in our organization.

Again, looking for root causes to mitigate them. Uh works with our physicians and uh nurse and pharmacy partners to look at ways to implement um care that mitigates variation and improves outcomes for our patients that work today. Is very manual.

I, i, i say without sort of precision, but uh probably good uh broad stroke that half of my team spends at least half of their time looking in the electronic health record for an answer. Whether that's an interpretation of the care we provided, whether that's a piece of evidence that we gave the drug at a certain time is, or is it, um, what was the sequence of care?

Uh we have to make sure we can uh publish our uh our performance for folks like c ms and joint commission. We spend a lot of time looking in that record to again, find an answer or evidence the care we've provided.

Um that's not even accounting for how much we spend on other partners outside of our organization abstracting records for us because we also do that. So I think that of a large language models as really the ability for our teams to um synthesize what is still in the electronic health record at over 90% of the record today is unstructured text.

It's notes, it's um phar you know, pharmacy, nurse, physician notes, it's uh nutrition notes, it's all sorts of notes, unstructured text and the promise in uh my belief of the large language model capability is that we're gonna be able to uh significantly cut down on the time uh that our team members are, are, are looking for those answers.

Um and then what we'll do with that time is that we'll redeploy all of that um talent back to the bedside. We're gonna equip them with the technology of uh understanding what's happening based on s uh s uh scrubbing the medical record, um deploy technologies that will allow them to round with our providers uh with the understanding of what happened yesterday uh that might influence to tomorrow or today.

Um as opposed to what happens today in quality and safety worlds. As we often wait for a report, then we have to analyze that report and we're often very much looking in the rearview mirror about what happened and and the capabilities of being able to read the record, read this text, this a vast amount of text and then present back that information in a summary format with critical pieces of data. Is, is in my opinion, why i think we're on the precipice of um really fundamentally shifting the way we approach quality and safety and health care because we're gonna be able to uh respond to information guided uh by uh clinical decision.

Uh the graphic here is that the, you know, it's, it's the person with technology that's gonna be greater than the person. It's not the technology alone, it's the person using the technology.

Um and um and again, i think that we are on the precipice of fundamentally uh shifting the way we're approaching this, this space just doing a quick time check and making sure i'm i'm on pace.

Alright, let me uh share one more example.

Um and then uh and then turn it over to my colleague. So uh 11 other example of how we're thinking about uh leveraging technology and in partnership with aws uh is I mentioned those incident reports um that our team is responsible for.

So this graphic is uh a graphic of bi week, how many incident reports are submitted, uh some general categories related to those incident reports.

Um and then the real question is then what do we do? We have 293 in this last week on the slide. Uh what do we do with that data? Well, our, our vendor partner for our incident reporting system um helps us a little bit uh by giving data and some graphs and some charts over time.

Um but what it doesn't do is read those reports. And as I said earlier where it's a very manual world we live in today. So we have team members who are reading all 293 of those reports. And by the way, we're encouraging more and more reporting.

It is through this reporting that we're understanding what are the root causes, what are the processes uh policies or practices that are failing our providers or failing our patients? So we're encouraging this more and more but somebody is reading each and every one of these and then somebody is interpreting that data, somebody is bringing back graphs and charts and looking at.

Ok, what are the areas that we need to go pursue again with very precious performance improvement resources. So the promise of the large language model and the partnership we have underway with uh aws is to test the ability to read each and every one of these to bring summary information back to the same people who are reading every word today.

Uh but allowing them to do quicker interpretation, quicker sort of uh assessment of relative significance. Um and then of uh most importantly, reacting to it. Uh so what, what we're uh testing is the uh the pre read and the categorization of these incident reports.

Uh again, thinking that ultimately, we'll be able to save that precious piece of time uh to redeploy resources.

Um, the, the the goal here is um not to respond to incident reports. So uh we think that uh there is technology today that uses triggers. You can, you can embed them in your electronic health record. If an event, if a sequence of steps has, has happened, uh it will alert you, you can embed these, this ih i is published on uh these triggers for some time. You can embed triggers in your e hr but those are all still events that have already happened.

Somebody has documented that a sequence of things have happened and then the alert goes off. That's often what you see in these reports is somebody said, oh, something has happened or something is about to happen and i wanna prevent that.

Um, where i think and why i'm so bullish in this space. Uh, we are going is that and if you think about the, um, and we and dr tarango shared a, uh, tragic story to start our conversation, uh, this afternoon, um, those kind of incidents are seen as in the medication safety world.

There's a categorization scale of a where no, no event, no deviation if you will through uh ultimately uh death. And it's a scale uh al alphabetic or alphabetical scale where a and b are events that um uh either didn't reach the patient or uh reach the, the patient um with mi minimal harm and where i want us to go and where i think we can go is to leverage the technology to understand the patterns of care in such a way that we can actually detect those a and b incidents of uh incidents of harm.

And um that is, i think aaa ways off. Uh but it's, it's what we are um uh passionate about uh striving for is it's about the prevention of harm instead of reacting to harm. And um the more and more we leverage the technology, the more we understand the processes that are underpinning these um uh events and the, the failures in our care processes.

Uh the better off we'll be able to prevent them. Um, and certainly detect them uh much earlier. So with that, i wanna thank a few folks. I wanna thank uh my partners here uh this afternoon who will be presenting to you all? I wanna thank you all for being with us.

I wanna thank uh the jacobs who uh have graciously funded our center for health innovation. Um and for my team back uh at the office uh doing all the hard work while I get to share our experience here today. So with that, I'll turn it over to john.

Ok. Ok, great. Thank you, chad. Appreciate it. So let's go ahead and talk about what we're building uh at u cs d health. And uh my name is jap. I'm the senior solutions architect actually supporting u cs d health. So let's talk about that architecture.

So we start off, this is the general architecture here. On the left hand side, we have two different data sources that we're working with. We're working with the epic uh which is the electronic health record system that u cs d health uses. And then we're also working with rldx, the patient safety system.

And so with the epic data, we have several 100,000 patient records that are actually stored in that system. And in the rldx patient safety system, we have uh different incident records that are are uh that chad just spoke to a little bit earlier that are recorded in the system.

And so both of these data sources actually have structured and unstructured uh data inside of these that we want to capture. So the first thing we're doing here is we're doing a, a bulk extract uh fire extract from the epic uh uh electronic health record system into s3.

And then in parallel to that, we're doing a bulk export of data from the rldx data database into s3. And we're staging both of those sets of data uh so that we can consume them later on in the process here.

The next thing we're doing is we're leveraging aws health lake import jobs uh to ingest the data. Uh the epic health data from s3 into a health lake data store. And so our health la data store is our hipaa eligible service that helps you to do clinical data ingestion store, the data and actually do analysis all using the uh hl seven specification uh fire specification.

And so from here, um what we're, what a aws health la is doing is it's using two different services. It's using aws lake formation, which is our fully managed service that helps you to create maintain and secure uh data lakes. And then it's using the aws glue service, which is our serverless data integration service, which helps you to catalog, modify, move and integrate data into our health lake.

And so in the, in the portion of having to integrate the data uh into our health lake, we have to take our rldx data uh data and integrate that into our health lake so that we can pair it up with our electronic health records that we've imported from epic

And so what we're doing is we're using Lake Formation and we're using Glue to create a database table to store that RL data data in our health lake. And essentially what we have here now is we have uh a joining of both the electronic patient records and they are all data, patient safety records.

And so now we can query those both together using a unique identifier like a matching patient identifier. And so we have the data staged here now. And there's a few things that we need to do in order to progress the that data forward and be able to be used by our generative AI application.

And so the first thing we need to do is we need to convert the unstructured data. All the notes that Chad was speaking about earlier to embeddings and embeddings are mathematical or numerical representations of text in the form of vectors.

Once we have those vectors of those embeddings, we have to store them somewhere. And with the amount of data that we have here, we have to be able to do this in a very efficient way.

So what we do here is we have the data stage. Like I said, we're gonna use a SageMaker processing job and that SageMaker processing job is gonna take that data and run it against our Amazon BEDROCK service and our Amazon BEDROCK service is hosting uh the Amazon TITAN text embeddings uh model which we use to create those embeddings.

And so uh once we have those embeddings, we'll retrieve those. But let me pause there for a second. What that SageMaker job is doing is it's actually running on a cluster of nodes, right? We're dealing with large amounts of data and we want to make sure that we can do this efficiently.

And so we have the data distributed uh from the S3 bucket across multiple nodes so that each node can access subsets of that data. And then on top of that within the actual nodes themselves, we're actually running scripts that are working through creating these embeddings. And those scripts are using Python parallelization in order to add another layer.

So we have parallelization at the cluster level and also at the node level. And so again, we've run the uh data uh through our unstructured data through our BEDROCK service. We are now uh retrieving embeddings back from that.

And then our StageMaker processing job is re uh receiving that those embeddings back um and then passing them on to store them along the way before it actually stores them. It acts the structured data to our embeddings as metadata so that we can store that in OpenSearch and OpenSearch is our distributed search and analytics service.

And in this case, we're using the uh uh uh nearest neighbor plug-in enabled in OpenSearch, in order for us to be able to do similarity searches in our OpenSearch uh database.

And so one thing to note here is um with AWS, you have your choice of many different services that you can use. So in the case of a vector store, you have the ability of choosing things other than OpenSearch. You could do uh a raw PostgresSQL with uh the pg vector extension. You could use our serverless vector data store through OpenSearch as well.

Um and there's probably a few extra announcements too coming out if they haven't, if you haven't seen them already, but essentially, you're able to uh switch out any type of service that you need to use for the vector data store.

And so it's important when you're doing this kind of work and you're trying to find the right tool for the job uh that you keep your embeddings and your uh and your data close to each other so that you can experiment because a you want to save on time. Uh you wanna make sure that the uh ingestion of this data of the creation of the embeddings is performant and is scalable, especially with hundreds of thousands of records potentially that you're going through.

So let's go ahead and proceed here. So i said we have all the embeddings already stored in the uh vector data store. And so now we have uh our application on the right hand side and the application is uh a chatbot interface that we're using uh with.

So the UCSD uh health staff can interact with the data that we've stored. And so essentially what's going on, it's just uh three different components of this. It's a web application, there's the API Gateway that's backing that and then it also the API Gateway will invoke a Lambda for us.

So what the user does is they input their question into this chatbot interface and then it gets passed to our API Gateway and then from there, it gets passed to our Lambda. And so what the Lambda is running, it's actually running a fra a framework called uh LangChain.

And it uh provides us with very flexible abstractions to different types of resources that we can use. Um and also different types of tool sets that we can use. And essentially this is a space where we did a lot of work because we have to figure out what types of interactions we want to have with our l uh large language models later on in the process, you know, uh which data stores we wanna connect to.

In this case, we're using the open source vector store, right? So there's a lot of different uh pieces and flexibility within LangChain you can use. Do we want to use agents? Are there tools we want those agents to be able to use in order to be able to interact with the large language models in a specific way.

So there's lots of variation. We did a lot of experimentation there. This is also a place where we're passing the hyper parameters potentially to different models and making sure we're fine tuning those hyper parameters in a way that actually make this process efficient and then return some data back.

So now that the Lambda has the user question, uh we're gonna go ahead and pass that on back to our Amazon BEDROCK service. And we're again using the Amazon TITAN text embeddings model in order to be able to retrieve embeddings from that user question for us to do the similarity search a little bit later on.

Again, this is the same model that we used with the original data source so that we could uh make sure to be consistent with our response back uh from our em uh for uh for embeddings from the service.

So, 011 other thing i forgot to mention is uh when you're dealing with a chat bot, you have a conversation history too that you have to maintain. So that is actually being stored in DynamoDB, which is not in this diagram.

So now that we've passed the input, uh we're gonna go ahead and have those embeddings retrieved back by our Lambda. And then we're gonna take uh those embeddings uh that we've received back from our Amazon BEDROCK service.

And we're gonna go ahead and pass that on to OpenSearch. And we're gonna take and start initiate a similarity search using those embeddings to see what other embeddings are close distance using the uh nearest neighbor algorithm to the embeddings that we're looking for.

So we're looking for what's, what's the most similar uh uh embeddings that we can find in our OpenSearch vector store. And so from there, OpenSearch uh sends back to our lab. But again, this is using LangChain to enable that connectivity back and forth.

OpenSearch returns back all the similar uh data uh that corresponds to the questions that we're look the that the end user actually put into our chat interface. And then we take that data, we generate a prompt uh from uh using LangChain from that prompt and then send that prompt back to BEDROCK.

And then from that point on what we do is we uh are leveraging uh Anthropic Cloud B2 as our large language model. Um and then passing it that prompt, which is again the user question and all the similar data that we retrieve back from OpenSearch from that point.

Uh we uh BEDROCK returns back the uh generated response back to our Lambda which is then sent back up to our end user. And so you can see that this allows our end users to be able to interact with very large amounts of data in a chat interface to surface up some of the things that we talked about uh that could improve or have an impact on patient safety.

And so from there, um uh we can, we can start to uh fine tune and figure out different ways of changing different components. So, like I said before, another thing, we're experimenting for our vector database is actually PostgreSQL as well.

So i encourage you to take a look. There's many blog posts on that we could share with you after afterwards that actually have different components of this. Um and i wanna thank you very much for your time. Let's go ahead and hand it off back to Rod. Thank you.