Learn how Black Knight is using AI to accelerate mortgage workflows

So we are going to be talking to you today about how Black Knight, one of the leading technology solutions providers in the mortgage and the home equity lending space, how they use AI solutions to power those capabilities for their customers.

My name is Adi Krishnan and I'm the General Manager for Computer Vision Services at AWS, that includes Amazon Recognition and Amazon Textract. And what I'm going to be helping here today with is provide you with the overview and introduction, do document processing and focus a little bit on what makes it really challenging and gnarly problem to solve.

I'll share with you what are some of the pain points that we have observed customers experience when they try to build these capabilities by themselves today with the kinds of offerings that already exist. I will then transition to share with you how Amazon Textract, which is our AI driven capability for extraction of data from documents of all varieties, how some of the capabilities over there help customers build for these intelligent document processing solutions.

And then will come the really exciting part where Frank Poise the Business Strategy Director for Mortgage Origination Technologies at Black Knight is going to share with you a deep practitioner's view of how these solutions are built in the real world. And hopefully that will allow you as developers, as technology decision makers, a good sense for how you may want to tackle these problems.

As you go through your journey turns out that despite the digitization and the digitalization that's happening in the world, there is a heck of a lot of paper that still powers all sorts of businesses. And this is happening virtually across all industries and barring health care. Probably one of the segments where the intensity the complexity of document processing is right up. There is the mortgage and the home equity lending space.

We see that that this industry has some of the most document intensive workflows, consider a loan packet, a loan packet that could have hundreds if not thousands of pages across dozens of varieties of different documents that contain within it that represents all sorts of data. It can represent income statements. The borrower, the co borrower history of debt, the credit history. It can include, it does include identity documents of all kinds. It includes documents that try to make sense of what the asset is that is being bought. And this kind of complexity is something that happens at massive scale every single day.

So how have customers tried to tackle some of those challenges when it comes to this incredible volume variety and diversity of document types that they have to process today? And typically, it's fallen into one of three buckets.

The most common one is when customers leverage optical character recognition or OCR tech legacy OCR tech has been around for a long time now and it's commonly used across a variety of these document processing workflows invariably. What we have learned is that these technologies, OCR technologies tend to work better on more simpler documents. And the extraction process invariably results in a bag of words that tends to lose a lot of the inherent context that the document contains. It can strip away everything from the structure of paragraphs, the tables, the lines, the words which means that there is a heavy lifting on the implementer, the developer to now start to make sense of what exactly was extracted.

The second big approach is manual processing, your human review and to be clear, these are not either or but manual processing via humans in the loop is very pervasive in the industry. But as you can imagine this, this is humans are tend to get tired, we tend to make errors and when it comes to managing that kind of staffing at at scale, it doesn't always follow the the demand cycle that a customer may see when it comes to their end users. And that's fundamentally challenging.

The third approach is around using rules and templates to deal with the bag of words that have been extracted. But what we have discovered then is that when customers build these rules and templates, they tend to be brittle, they tend to be brittle because ultimately a human has to figure out exactly what template and what rule to write and which may break when a new kind of document shows up, or it may have to be rewritten. If the underlying business workflow is changing.

When customers embark on this journey, there are two big sets of issues that that are below the surface with these legacy approaches, one big bucket is really around lost revenue and the lost revenue is really stems from the fact that there is with legacy systems, the way they are composed together, it is invariably hard for them to grow and shrink in elastic ways. It is invariably harder for technology technologies to build them and compose them to leverage the best breed of technologies. And that fundamentally is a throttle on your ability to grow effectively.

The second big drawback is really around slowness of the processing of data, which means that the ability to get to a high quality business decision takes that much longer and that percolates through the entire business. And this ultimately leads to lower end customer satisfaction and which can drive churn on the flip side.

We notice customers who still deploy a lot of this legacy tech are also facing higher costs. This can range from the the staffing related costs of managing and maintaining and keeping the lights on. It can extend all the way to inaccurate extraction leading to sub optimal business decisions which can cost the business. And then when you think about a lot of these legacy technologies being composed together there invariably is a lot of scaffolding code, a lot of tech debt that teams tend to accumulate over a period of time.

So when we build from the grounds up AI and machine learning services, we do benefit from the fact that we have learned from a lot of customers in the real world. And that allows us to then build the kinds of capabilities that take advantage of advances in computer vision, national language processing and other machine learning innovations to go beyond what traditional OCR techniques have allowed us to do so.

The key benefits here are that we are no longer talking about, you know, the traditional grow fashion OCR. But we where we can now preserve document context of complex forms of complex tables where we can up level the process of data extraction to think about things as documents that could be specialized like identity, documents or invoices and receipts and much more. And the goal here is is that by doing so, we are able to get accurate and faster throughput on that kind of data extraction that preserves the context, which means developers and technologies can then integrate those insights as part of their business systems that much more efficiently.

This has invariably the downstream effect of reducing the total cost of processing, especially when you think about the scale at which these document processing work flows run. So let's look at some of the features within Amazon Textract that help customers get to more efficient ways of document processing. Some of you may already be familiar with our traditional text extraction and the OCR extraction capabilities. But in addition to that, there are lots of capabilities that include handwrit text as well as the ability to extract more complex structures such as tables or nested tables, forms of different kinds, being able to yank out key value pairs that exist within the documents while still preserving the context. And this can happen across a wide variety of documents that we have pre trained the models on financial statements, loan applications, identity documents and many more.

The other thing customers told us is that ok, we like your, you know, basic OCR and it's accurate and cheap and all that great. But could you specialize this for specific kinds of documents that we see commonly in our business workflows? And here's where we can specialize things like identity documents. So for now it's US passports, US driver's licenses, so on so forth or invoices and receipts that have very peculiar weird ways of representing data and customers then use them to accelerate their use cases.

So for example, if you're using our analyze expense capability for invoices and receipts. You may have accounts payable invoicing, an internal payroll, internal financial processing workloads that could go a little bit faster.

A place where I'm going to spend a little bit more time on today is with Textract Queries. But before I get in there, I want to linger just a little bit to really dial in on the data extraction challenges that we know customers face. And maybe some of this will deeply resonate with you too. And if you're early in the journey, then it serves hopefully as a um as foreknowledge about the kinds of challenges you will likely encounter.

The first thing is that data is just exists in so many variations across documents or a similar stack of documents that it's not a given that the system will automatically figure out that date of birth, DOB, birth date, actually all mean the same thing. And these are the little things that matter much when it comes to data extraction accuracy.

The other thing is that how the data is laid out the structure with which it's represented can also be incredibly complicated and diverse data showing up in tables, nested tables forms that have very, very different structures in terms of how they accept what kind of data, whether it has fields that are labeled or fields that are implied. And all these variations exist within a set of even related documents that customers have to process.

And then you've got a bunch of variations that are around how the data is laid out itself the orientation of the text. How does a table get interspersed between paragraphs of text? How do section headers putters? How do they make sense to lend extra context to the document itself? All of these are real world challenges that customers face and invariably what do they have to do? They end up writing a bunch of post processing logic. It's a bunch of code, ultimately, that has to be written, it has to be maintained sometimes good, old fashioned deterministic code doesn't do the job. So customers have to build customized machine learning models. And now when you start to think about the complexity that you have to manage, it gets expensive and time consuming.

So for those kinds of unique issues that exist pervasively, these are not corner cases, they exist all the time, we just don't know in which document is going to show up in what form for that reason. And for particularly gnarly difficult documents that are inherently difficult to process through the set of OCR OCR plus plus techniques, we invented Textract Queries.

So customers see the benefit of accuracy and speed using the existing capabilities that we have on forms and tables and OCR to detect text. But when, when they hit against these roadblocks, they have to figure out, you know which key value pair, which table value corresponds to what information they're actually looking, looking to extract from a given document.

And so for such customers, we built the queries capability which enables them to specify using natural language queries and extract pieces of information from these documents. So given a document, it enables the developer to express what exactly it is that they're looking for what piece of information in a natural language form.

So what is the borrower's tax id? What is the social security number of the co borrower? By doing that Textract Queries uses a combination of vision language and spatial cues to extract with much higher accuracy. Exactly that piece of information that the developer has requested without having them had to build any customized model or presenting new data to train such a model at all.

And because it's a simple Q&A based process, it makes it feel a lot more natural like you may ask your colleague. But hey, can you tell me what this doc says when it comes to the specific field? And because this is part of a well formed API response, this means that a developer can yank the output and integrate it into whatever next is in the business workflow. Maybe it's an insert into a database or maybe it kicks off a different scheduled workflow depending on what the response is.

Let's look at some, some examples here. This is the common Fannie Mae form 1003. If you notice the this this form has has a borrower and a co borrower that's gonna split halfway through the page and traditional OCR will extract the information. But now as a developer of a business application, you're trying to figure out whose birth date is this exactly? Is it the borrower or the co borrower? And it turns out that systems fail at that seemingly common, simple task for us humans when you're looking at it with our eyes.

And so in this sort of a situation where there are these nested sections, you could ask Textract query, what is the borrower's date of birth or what is the co borrower date of birth? And you could issue many such queries for any given document and you will get precisely the answer to the query that you can then build into your application.

Here's another example of a bank statement. You know, there are hundreds of thousands of banks plus credit unions that exist at least in the United States. And this is an example of an implied field. So highlighted in red is the customer's name and address. But notice that there's no field that says name, colon or address, colon and then on adjacent to it within the box is the bank's details and there is a mailing address in there. Now, when you try to extract data, you will get an address but you don't know whose address it is. And this is where issuing a query to say what is the customer's name. What is the customer's address is going to give you an accurate response back rather than having to decipher that once regular OCR does its job.

Here's another more interesting example in a way of, of tabular data. Now, if you notice the table here, this is, you know, a pay stub. If you notice that the table here, that's highlighted for gross space a little different because it's sort of regular rows and columns, gross pay is offset as a cell value under rate under the rate column header. And then you've got two values for, you know, for this period and the year to date gross pay. And turns out this again, seemingly simple problem when it comes to actually using OCR tends to really complicate matters.

And so in this example, a developer could issue a query that says what is the gross pay for this period? And what Textract queries does is it takes the image, it takes the hint from the query that the developer has issued to say i must look for that kind of information and that kind of abstraction. Those cues are what help us extract that data with higher accuracy even though the Textract system may not have seen that kind of document before.

As I wrap up here, I do want to share that we recently launched the Analyze Lending API targeted to help customers build more efficient mortgage processing documents, mortgage document processing workflows. And what we observe that customers repeatedly follow certain set of workflows. Given a large set of documents, they have to classify them quickly. They need to redirect those documents to the right kind of NLP models that are specialized to extract that data from them. Then you do a high quality job of data extraction, of course, and then run some validation. And since we saw these patterns happen time and time again, as we heard from our customers, we were able to build out this capability as a single well formed set of APIs to help customers just advance their journey a little bit more.

With this with the Analyzed Lending API I would now love to transition to Frank's portion of the presentation where he's going to talk to us about how Black Knight extensively uses AI and ML technologies to accelerate mortgage crossing workflows.

We have to get it right. This is a very complex industry. So when you think about all the paper, if you've bought a house and you've gone through the process of getting the loan closed and been concerned that there's a stack of paper about this high at the end of the process, it's largely because all of these entities have something to do in the process of financing a home.

Of course, consumers will they get paper, they get disclosures, they get information about their property, they have documents to sign legal documents that evidence their debt. The lender has to get that documentation and gather information in order to underwrite the loan in order to know that it's a good loan. Banks finance these loans and put them on their balance sheets. They need to manage the risk associated with these transactions.

The regulators, the FHFA, the FFIEC and others all regulate the institutions that make these loans so that consumers are treated fairly, and so that the financial system is not as fragile as it was prior to 2008 crisis going on. There's title services and settlement service agents that support the process of closing loans. Servicers again manage the loan throughout its lifetime.

There are insurers in the process who help to provide secure funding, they create credit enhancements and they monitor the property's insurance. And finally, there are investors. This is one of the most interesting aspects of the mortgage lending process that most consumers don't know about lenders, don't keep your loan on their books. They sell the loan. Many of you have heard from your lender. Oh, we've sold your loan to another lender. That's because there is an active securities market that is driven by the debt in residential real estate in this country. It's how the industry works. All these stakeholders have an interest in the paperwork.

Why is it still so crazy? There are hundreds of documents that support this process at Black Knight, we support more than 800 different document types and we extract over 600 data points from those documents. And by the way that the demand for additional data is growing every day. As Adi mentioned, many of these documents are highly variable. We're going to talk about bank statements in particular and why that is such a to use to use. The great term to use is such a gnarly problem. Many of these documents are not only highly variable but very few of them are actually standardized. And even the ones that are standardized can be produced in different ways that 1003 that was used as an example earlier is the standard application provided by and and mandated by some of the regulatory bodies that manage the industry. Well, they can be handwritten, they can be produced by computer systems. The data can vary widely from one system to another.

Most of the documents that support the process are originally in analog in paper form. Their analog. The industry can drive improvement in this and has been for well over 20 years, there have been legal abilities to e sign documents and e deliver documents. However, even today, 24% of consumers do not want to receive their documents digitally. 24% of the time we are printing them out and putting them in the mail when they come back from the consumer, then we have to scan them and process them. And despite that 23 year old law, only 4% of loans are closed digitally, meaning the signing of the mortgage and the all of the transactions that handle the the moving of the funds from one party to another, only 4% are closed digitally, there's a lot of reasons for that. There's a whole session we could do for do about that process.

So bottom line it's very expensive just under $11,000 per loan is a statistic that the industry is, is fighting with today. Think about that $11,000 per loan and millions of loans originated. So what is our mission Ava document services has a mission of classifying those 800 plus document types. Classifying means not only knowing, ok, this document is a pay stub, but also is the second page. Also a pay stub. If there's a loan app, there's an appraisal of a property that that appraisal can be five pages long or 10 pages long. Understanding not only where the document, what the page is, but also what the document is.

What was the solution that we arrived at? Black Knight has been working with Amazon for just about four years. Our CTO is right there and we can uh we can ask him exactly how long. Um but we've been working with Amazon for a long period of time trying to slay this particular dragon with a combination of AWS infrastructure. We were hosted on AWS Amazon Textract, which is the subject of much of this conversation and Black Knight's own proprietary AI and ML models, software and data.

We've created an environment where we are finally making exceptionally good progress at nailing down those 800 documents, classifying them correctly the first time and extracting a large body of data that can save our customers money. We have a humans in the loop process because you have to have humans in the loop to read the ones you can't read with the computers. So if computer vision fails, we have to have a human read the document. If we don't get a return from Textract that makes that, that gives us the information that we need. We have humans to do the reviews. So we have exception processes and we have processes to detect whether our models are starting to drift.

One of the things we have to do with, with our AI/ML just like everybody else does. Our models can drift. We put new models into the wild, they can regress. So we watch them like a hawk. So we have humans doing essentially three things on documents that can't be read, they're reading them on documents that may have problems, they're checking them and on documents that are coming through. Just to be sure we do sampling to make sure that we're getting it right.

Our key strategies to solve the problem include a classifier, just open an open source OCR product. Um Black Knight trained models. It's a transformer based natural language processing model that we've been using for several years and it gets trained and retrained. And by the way, we also for some documents, we use Textract, we also do extraction using largely the Textract knowledge graph, the information that comes out of Textract. We use all the tools that Audie mentioned and we pick the best of them, but we're still handling 800 plus document types. And we have, we have a lot of complexity there that we have to solve for our clients. Um that just has not been able to be accomplished across the industry.

So we focus hard for our clients on getting extraction, right? So we've got proprietary models. We've got the Textract knowledge graph and we have a business rules system to use the data. And this is key. It's one thing to extract data and it's one thing to get back documents. You're gonna save time by not by avoiding keyboarding. But what you're going to do the most by having this data available is be able to apply business logic to accelerate processing. Don't just tell somebody somebody that they don't have to keyboard but bypass steps. I have all the W-2s for borrower one check, no human, like no human had to look at it. That's because we have a document called a W-2. We extracted information from it that tells us that, that's for borrower one, we can then pull all the document data, all the data about their income and populate the system so that a human doesn't have to touch the analysis necessary to do income validation. The business rules are the key element that drives value.

So let's talk about bank statements for a minute again. Big gnarly problem. There are 9600 plus financial institutions in this country that are federally insured. So banks credit unions thrift, each one of those has its own bank statement. Even though there are a number of providers that are the same across banks. Every bank configures them their own way and every consumer's bank statement is different. Some consumers may have one checking account, another consumer may have two checking accounts and a savings account. Another consumer may have a checking account, an investment account, an IRA account and a business account and they may all be on the same statement. So we have very, very different documents.

There's also a complexity that may seem surprising but it's not when you think about it. Transaction lists are one of the important things that people that we have to look at in the process. So we have to look through a list of transactions to decide. Is there a transaction of interest to an underwriter? As I review the document, I might be looking for large transactions that are unusual. So I have to be able to read every row in a table of transactions that may span multiple pages and be able to identify the separate tables within that document and extract the data reliably. This is where Textract tables comes in. However, it's all of that complexity and the fact that the tables are oriented differently. We have, we had to do more to get it to be reliable. To the point where we could drive those business rules, driving that business value meant we had to be able to do things like take the beginning balance and the ending balance. All the transactions add up to those two numbers I had be we have to be able to reconcile that data in order to know we got it right.

So what we did is partner with Amazon, the Amazon Textract team got together with our team, we studied the problem and the Amazon team delivered a model with and a set of capabilities, a set of rules and model adjustments that the Black Knight team took delivery of early in the year. We then started tuning and improving that basic framework. And you can see it was a great framework because we went from very, very low scores. This is uh for those of you who are statistics oriented, this is, this is F1 a measure of accuracy that we use. We had very, very low scores. At the beginning of the year, it took off very quickly but didn't get where we needed it to for several months. But during the time from July until the end of September, we were able to achieve a 0.9 F1 in handling this very complex problem. This is a great success because what that means is that 90% of the time we can do what I told you earlier, we can see everything in the bank statement. We can identify transactions of interest and we can apply business rules to help solve our customers business problem.

So what is Ava's value proposition for our customers? What's the ROI to our customers? Well, we are able to process now about 14 million pages a month. We are extracting over 7 million data points every month and our clients are able to generate significant savings and the savings vary from customer to customer and depending on what kinds of loans they make. But we're seeing seconds per page. Remember 1514 million pages. That's a lot of time, seconds to minutes for each data point. And remember the minutes can turn into hours if we can take a number of data points and use them to drive real business process improvement. And that's the phase that we're entering now at Black Knight to serve our customers in partnership with Amazon adoption is accelerating. We have a line up of customers wanting to adopt this technology and we're seeing real value among our customers and other industries as well. So it's a very exciting place to be right now and back to Artie. Thank you.

Thanks Frank. So we understand though that getting through this journey can be complicated, especially given all the systems customers have to build. Um and so for that, there are numerous AWS partners that we work with closely to help customers bring their toughest solve for their toughest business problems. Here. There's also a, a nifty little matrix to help some of you who are going to embark on this journey decide how do you want to make some of those trade offs in terms of where you want to invest in house versus where you may want to work with a partner to help you build those solutions.

I do want to wrap up here with a, with a few few links and then open up sufficient time for us to, to chat and, and answer any, any questions that you may have. Uh but there are a number of different uh sources of documents, blog posts, um and source code through numerous samples that we've put up on up on github for developers to get their hands dirty with, to play with the product, play with the capabilities as well as several prototype uh solutions for you to see how that might make sense for you to do. And i um i would encourage you uh uh the builders here to give that a shot. Should you, should you choose to do so with that?

We'd like to wrap up kind of the formal presentation here. Uh I want of course, deeply thank Frank uh and the Black Knight team uh for the partnership. Uh we know that we learn deeply from them and that pushes us to build better capabilities and together we're able to hopefully solve problems uh for as many different kinds of customers as we can as possible. Um and of course, you know, with Black Knight leading the charge in and how they're able to deliver these cutting it solutions for customers in this space. Uh we just delighted that we can be part of that journey uh with them.

So with that, uh we have uh a lot of time i think for us to, to talk about questions if we haven't.