Vector database and zero-ETL capabilities for Amazon OpenSearch Service

Hello. Hello, good morning everyone. Wow, it's Friday. We all made it. It's wonderful to see such bright faces. Thank you so much for coming, for coming to our session.

A quick poll over here. How many of you have started experimenting with generative AI applications? Wow. Ok. Yeah. And how many of you are looking to optimize costs? Absolutely. Yeah.

So these have been the consistent themes this year for us, both of us from the open source service, which is how do we build interactive generative AI search experiences and how do we optimize costs?

This re:Invent, we addressed both with number of features that we are going to be talking about today. My colleague Joshua and I are really excited to be talking about how you could build some amazing search experiences and also optimize costs with zero ETL capabilities.

Thanks. A quick look at the agenda. We'll start off with the recently announced Vector Engine for OpenSearch Service. Then we'll talk about the new R6g instance family that gives you up to 30% price performance benefits. And then Joshua will talk about one of the most awaited features which is zero ETL integration with Amazon S3.

This year has been incredible when we're talking about AI capabilities. And there's a lot of excitement when we talk about generative AI. Large language models have really enabled a new wave of generative AI applications. Large language models, also LLMs, have been trained on massive data which enabled them to have now humanlike conversations.

Not just that, the other trend we are seeing is lowering the barrier to building these AI based applications. APIs and no code tools are really democratizing AI. What would have taken months to build now you could build within hours, namely chatbots and AI based assistants.

In short, it's really exciting to see AI based and generative AI based applications going mainstream. Since many of you are here to see how you could build those interactive experiences for your digital users, let's quickly talk about the evolution of searching.

Search started with basically the keyword matches, right? So what you would do is you would put in a query, you would query for that and you're looking for that exact or precise match in your documents.

Then there was the next wave of innovation with faceted search. Faceted search helps you filter on multiple attributes. So take an example, a shopping site with faceted search, you can basically filter on consumer ratings, brands, price. So it makes it more flexible.

But generative AI has helped us build, you know, even more amazing experiences. It's opened up multiple possibilities for search. Search has become more interactive. You can now have conversations and the search can make recommendations and steer you in the right direction based on your intent and context.

So let's take a look at this demo. This is a very simple demo. I mean, if you think about, you know, you're starting to build your configurations, what you would do is you would go to the documentation you would type in “How do I find VPC?” or “How do I configure VPC?” Or if you have to make a choice between managed clusters and serverless, you would try to see you want to know immediately what is the difference between the two with this?

What we've done is this is a RAG based framework. We basically put everything in LLM, right? We basically vectorized this data and now you can see you can just have start having conversations, “Help me configure” and look at this. It's not only is it returning the information, it's also personalizing it. And you can see it says, let me know if you have any other questions, so you can continuously have this conversation until you get the information that you're looking for.

So let's quickly look at how do you build these sort of applications? There are two ways you could build search applications, one, you could fine tune the model and in fine tuning what you're doing is you're pre training the model with your data. Fine tuning works great if your data is relatively static, that is it's not changing. For example, analyzing your historical sales data.

But when you're talking about information that's constantly changing and you want to keep it up to date, RAG or retrieval augmentation generation is the way to go. In RAG what you're doing is you're improving the LLM, that is large language models, responses with an external knowledge base.

So let's try to understand how does that work, right? So you have your private data, your enterprise data. What you do is you extract that data chunk, the data and convert into vector embeddings. The many models that enable this vector tight and embedding model would help you achieve that.

Once you've generated this vector embedding, you then add it or load it to a database, a vector database, namely OpenSearch Vector database, right? Once you've loaded the data, the vector database is now ready for your queries.

The next process over here is the retrieval and the generation process. In the retrieval process, what we're doing is we are taking the user's query. Now remember this query is in natural language format. “Show me a picture of a red car” or “Find me the best deals”.

Once you actually input the query, the query then is preprocess and converted into vector representation, right? Then what we're doing is taking this query and running it on the vector database. The vector database is already holding your data. Remember, you've actually loaded all your data into vector database. That's the vector representation of your data.

Vector databases basically run a similarity search algorithm. They compare the documents that match that are the closest match to your query and return these documents. These documents are, you know the rank search results.

Then what you're doing is you're taking these documents as well as the user input. There's a prompt generation process and you're feeding it into the generation LLM models. And that w that's what is taking all this and creating that personalized response in your, you know, for your chatboard or personal assistance.

So your, your response can only get as better as your similarity search algorithms and what you're using. So what is really the vector similarity search over here? What you're doing is you're vectorizing unstructured data. Now, this data could be audio, it could be video, it could be text, but it's unstructured. You really can't fit it into a particular data schema.

So what you're doing is you're converting them into vector embeddings. Why vectors? Well, humans understand context machines understand numbers, right? So what happens is when you generate these vectors, the closer these vectors are in distance, the more similar they are.

So for example, if you're looking for say a red car, you might be searching for a red car. But you may also be considering you know, different shades of red. If you ran that with the exact match, you wouldn't be able to find crimson red or even say blood red. But when you're running it with similarity search, it understands because it's basically bringing all those vectors with similar meaning and context together.

Once you've done that, you know, you basically get the results and these are, you know, run, you get these results by running the similarity search algorithms. We'll talk about that in the next slides.

So in in a sense or in summary, the proximate the vectors, the more contextually relevant your responses are you know, based on the query that you're inputting.

Vector databases are one of the major components in the modern AI stack. These are designed for storing searching and managing vector embeddings. You want a vector database scale that can scale from you know, few vectors to billions of vectors.

We all start with prototyping applications, isn't it? And then we move to production. Now this is an interesting phase where we all are trying out different models, different chunking strategies and also different dimensions to see what is the best dimension application model for your particular applications, right?

So you want to be able to add delete update vectors in real time. Vector databases enable you to do that.

Your results. I think we already emphasized that and we'll be talking about this a lot is the response generated by the LLM you know, the generation model can go and only get as accurate as the results that are being provided by the vector databases. So you want to make sure you are choosing a vector database that is giving you a results with a really high recall, 95 to 99% recall, right?

Introducing Vector Engine on OpenSearch Service list. Vector Engine on OpenSearch Service list is a simple scalable and highly performant vector database capability that enables you to build ML augmented search experiences and also generative AI applications.

But before we talk a little more about the features and functionalities of vector database, let's quickly talk about OpenSearch. OpenSearch is an open source distributed search engine that supports a wide range of use cases. The very first one being search, of course, it's a search engine. So you can basically build ecommerce search or you can have applications, you know, full text search for your applications.

But because it's a it's a distributed system, it can analyze large amounts of data. So very quickly, we've seen. The second use is that many of you have been using for which is log analytics. You can use OpenSearch for application monitoring and as well as security analytics.

Since its inception, we've seen great response from the community. Thank you all. We've had over 300 million downloads and we are not the only ones that are contributing to this project. We have over 1500 folks that have already been actively, you know, contributing and they're all not from AWS.

Amazon OpenSearch Service is a managed service for OpenSearch. With Amazon OpenSearch Service, you can now deploy provision, deploy and secure clusters within minutes.

We have two deployment options for Amazon search OpenSearch service. The first one is managed clusters. Tens and thousands of customers have been running OpenSearch clusters, managed clusters and they have been storing petabytes of data in these clusters.

With clusters, you have full control on the configuration that is you can select which EC2 you want to use. You can select the storage that you want to use for your clusters so that you basically can have the price performance balance.

Earlier this year, we released OpenSearch serverless. We had many customers come to us and say we enjoy the powerful query that OpenSearch provides, but we really don't want to be managing the shards. We don't want to be managing the index life cycle. And that's why we introduced OpenSearch serverless.

With OpenSearch serverless you really don't have to worry about provisioning and tuning your clusters. OpenSearch serverless automatically detects your workload and provisions resources and also scales resources based on demand right.

To get started, it's really easy because OpenSearch serverless supports the same set of commands as the OpenSearch project and we have a wide range of ecosystem to build your applications.

Why Vector Engine on OpenSearch serverless? Well, very simple. Many customers want to focus on generative AI experiences. When we're talking about building generative AI, they have to try out different models because there's never going to be one model that fits it all. You want to be trying with various dimensions, various attributes, how you want to qualify your data.

And they've come back and said that, hey, we want a simple database where we can just send the embeddings. And we really don't have to be managing the underlying infrastructure. And that's where we have Vector Engine for OpenSearch serverless.

That when g a this re:Invent during the preview phase, we've had some great response from our customers, right?

When we are talking about Vector Engine, it can store and search billions of vector embeddings with thousands of dimensions in milliseconds, you can add delete and also update vectors in real time without really affecting your indexing or search performance.

Vector Engine generates highly relevant and contextual search results based on the vector embeddings that are trained on your business data. Not just that, you know, we all are talking about vectors, but there is a need for you to be able to co locate your data with your te you know, with your, with your normal text based data.

And one of the reasons is that when you're, when you're running your search results, yes, you can run vector embeddings. But if you want even more precise and accurate results. You should be looking at hybrid search where you can filter on a particular metadata and then run the vector search vector similarity search that way you're getting even better results.

So let's take a look at the Vector Engine architecture. It is based on OpenSearch serverless. So the very first thing that you'll notice over here is there is the storage and compute decoupling with this. What we've been able to do is have separate pipeline for ingestion that is separate computer nodes for indexing and separate computer nodes for search.

This really helps us scale all these layers that is compute and storage separately and independently, right, based on the workload demand.

Now think about it many times what you're doing is when you're indexing a data, especially for, you know, vector use cases, you wouldn't probably, probably being indexing it every minute or every few minutes. You would be doing that, you know, in a, in bulk and once a day.

Although that's an area of innovation that we, you know, we think that in few years or, or even a few months, perhaps you would start updating the vectors in a couple of minutes.

But while you're doing that, you want your search performance to be unaffected and this architecture helps us achieve that.

The other thing that you'll notice over here is we have an active and a standby node. This is built for production environments so as you're actively ingesting the data, we have the standby node that basically protects against any AZ failures or any or against any node failures.

And on the third side, we have two active replicas. Now, all these replicas are running in separate ACs, right? And that's how we are able to ensure availability for your workloads.

What's happening during indexing is as you're ingesting the data, the compute nodes are building graphs and the graphs is what is able to give you the similarity search results. Once the graphs are built, they are stored in S3. And the benefit of that is when you're querying, say suddenly there is a spike in search query. We are able to pull those graphs directly from S3 so that we can meet your concurrency requirements.

There's seamless auto scaling. So you really don't have to worry about reindexing for scaling purposes. And of course, all the data is stored in S3. So that's giving you the 11 9 durability.

Now let's take a look at the vector engine features, right? Vector engine supports hnsw. This is the state of the art algorithm that's used for approximate search for nearest neighbors. We've implemented two popular libraries F and NMS lib. But what we've done is we've taken the distributed architecture and these libraries that helps us scale out and also is able to, you know, handle updates and deletes in real time.

Distance algorithms. I think this is you know what we are running or, or the vector engine is running to find the closest match to your query. We support Euclidean cosine similarity. And in our product, again, the choice of this algorithm really depends on your data. Cosine similarity works great for text based semantic searches.

Now, while you have this one thing we have to realize is, you know, it's really important to have the highest recall, but it's also important to be able to tune these hyper parameters so that you're able to balance between latency and recall. Latency and recall are inversely proportional, right? So the higher the recall higher the latency. Yeah, high the recall. Yeah. Oh, they're directly proportional.

Next vector engine also supports efficient filtering, right. What is efficient filtering is based on your data sizes? We figure out whether we want to run a NN or we want to run KNN, which is the brute force KNN algorithm, right?

We bring in the query power from OpenSearch. So now you can run compound scoring, you know, compound queries with various scoring and boosting options as well. Not only that we support up to 1000 data types in um in in a vector engine for opener serve list. How does that help? Again? As we spoke, as you're building the queries, you want to be able to try out different types of queries to see how you can basically give the most precise answers to your questions, right?

The benefit of using vector engine is that you don't really need a separate vector store to for your lexical search. And uh you know the vector embeddings, you can combine them all vector embeddings is a data type in vector engine.

So how do you get started with vector engine? You simply create vector collection. Now, vector collection is a group of indices that is basically built together to support your workload. And then we also have you can easily create a vector index on your AWS console. The index is the one that's basically ingesting all your embeddings, right? And then you simply start querying for similarity search results. You can get started in less than 10 minutes with vector engine.

So let's talk about what is the compute unit for vector engine vector engine. We basically use OCU. OCU one OCU comprises of one vCPU six gig of RAM and 120 gig of EBS volume. So the number of OCUs will increase or decrease based on your workload. We scale these OCUs horizontally and vertically to meet your workload demand. You can set max OCUs, you know to control your costs and OCUs are automatically provisioned.

Once you create your vector collection, these OCUs can also be shared amongst collections. So if you have two teams that are using vector engine in an account, you can create separate collections, there is tenant isolation because of the security policies that we apply or you configure but they could share the same resources provided, they're using the same KMS key. It really becomes economical, especially when you're trying to experiment with different, you know, models and dimensions for your workload.

Security all the data in vector engine is encrypted at rest. And in transit, you can bring in your own KMS key or you could use the AWS or the service provided key. Since you're dealing over here with your private data, many customers have come back and said they want VPC and point support because you want to be, you know, because of the sensitivity around the data that your that your embeddings carry, you can now store the data behind a public endpoint or within a privately.

You also one of the things that's been consistent this week, we've been, we've been getting a lot of questions around how is the policies, how is the data access policies supported in vector engine? You can set account level policies, you can also set individual collection level policies or you could go down to the index policies. So you have that flexibility, it's hierarchical and you have that flexibility in vector engine.

Next, let's talk about performance. One OCU can handle about 2 million vectors with 128 dimensions and 99% recall. We've tested for billion scale during the preview, we had limited support. We were supporting about 2030 million and now we've extended that to billion scale and of course, we support millisecond latency.

Did did any one of you go or attend? Um um the bedrock knowledge based session if you haven't, i recommend highly recommend. This is something you know, we recently released. Even the bedrock knowledge base vector engine is one of the recommended vector database for bedrock knowledge base. With the better of knowledge base. You really don't even have to create vector indexes. It's the RAG agent is automatically doing that for you. All you have to do is point to the S3 bucket. It's going to pull the data from there chunk, it create embeddings, create vector collection on your behalf. So the vector collection still is in your account and it basically ingests the data and adds the embeddings to a vector index automatically.

We also have integrations with SageMaker and Lanqing. Let's take a look at pricing. We talked about how we have the decoupled architecture where the indexing OCs and the search, the search OCUs can scale independently. So we charge by indexing OCs and search OC. And of course, the data that is stored in S3 for durability.

We i think we already mentioned one of the consistent theme is how do we optimize for cost? We got a lot of feedback from our customers who said that we're still in the experimentation stage. So we really don't want or don't see a need to have redundant replica. So now you have an option for your depth test workloads with this, you can run vector engine collections with just two OCUs, which is one for indexing and one for search that reduces your cost by half.

And not just that if you have fairly small workloads, you can go down to 0.5 OCU. So essentially we are talking about one OCU for your experimentation um you know, for your experiments.

So let's like to summarize uh you know, vector engine for you. It comes with robust dat you know, data privacy security and encryption at rest. It's really easy to use. As we said, you can get started in about 10 minutes. It's feature-rich, it also has deep integrations with Amazon bedrock LQ. The number of examples RAG examples on how you could create Q and A applications using vector engine. Highly recommend you to try those out.

We've had customers who got this running on their private data in less than an hour. They basically had a Q and A application up and running. I think the other thing that's really important is scalability when we are starting build. When we, when we start building these applications, we start off small. You want a database that can help meet your scalability requirements without you really having to reindex your data.

The next we talked about performance, you can vector engine supports billions of vector embedding. And last but not the least reliability, you can reliably build scalable applications using vector engine.

Next, we move on to the or one instance family. This is the new data node uh family type that we are supporting for our managed clusters or one is is unique. You know, we've taken the learnings from sist and incorporated that in our new or one instance family. This can give you up to 30% price performance improvement over your existing instances.

So if you look at the typical deployment architecture, you can see that instead of using your current data nodes, you could replace them with or one. So what, what have we done that's different in or one? Just like servius, we've talked about where we have decoupled architecture in or one. We have decoupled the indexing and replication operations.

If you think about a cluster and how it works today, you would have a primary node and then you have replicas and these replicas are basically, you can select the number of replicas based on your uh concurrency requirements and you could also have rep replicas for availability reasons.

So when you're indexing the data, you, you know the primary, writes the data to the replicas and you have to wait until the data is written across uh all the replicas that you have configured, it is also cpu consuming.

So with or one, what we've done is we've introduced a new feature that's segment replication where you take the data and you directly write it to S3 and the replicas are reading from S3, this really fastens the induction process. And now you're using Amazon S3 as your remote stores, you get the same durability benefits, which is the 11 9 durability.

It also has built in automatic recovery from red indexes. So how does this help us? You mean, you know, especially for your data that you're ok with considerable latency, you could start off and you may not even have replicas, you could just go with the primary node because or one can automatically recover should there be an availability or a data um or a data node uh hardware failure.

So summarizing this or one, you know, we highly recommend you to try it out. It's great, great for log analytics use cases. You can achieve up to 30% price per per per price performance improvements, high index and throughput data is indexed to S3. So you know, you have durability, you really don't have to worry about writing the data back to sc four. You know, our first system of uh ba basically, you can use this as a system of record now and it can automatically recover from red indices.

So we just didn't stop here, right? We see that you have massive amounts of data that you're storing for log analytics. So now i call upon josh who's going to talk about yet another co cost cost optimization feature for uh log analytics use cases which is zero etl with amazon s3.

Thank you. Let's go wow, i was, you know, you heard me clapping and hooting and hollering over there. So i, i mean, i was just listening to the or one presentation and i think it's, it's very uncommon to get better price performance and better uh better durability as well. So it's really noteworthy.

Um today, what i'm gonna be talking about is amazon um opensearch service zero etl integration with amazon s3. What we've talked about so far is primary data, primary data. In this case is data that's been fully ingested into amazon opensearch service. It is queried often. It is used for content that is required that's urgent as well as important.

But there is this grouping of data which we'll call secondary data that exists in amazon s3 and various data lakes that is not queried very often. Um how many of you have secondary data within your systems that you have to go and maybe use different tools to get access to or, or maybe ingestion pipelines to get back into open search.

Yeah, absolutely. So a couple of log types that i've heard when i've talked with customers are uh that are common in amazon s3, vpc flow logs, s3 access logs, engine x logs, these logs rest on s3 and are um queried when there are historical or compliance oriented activities that are happening. I know for sure, like i've troubleshooted uh done a root cause analysis that had to go pretty far back. And i needed to go and get access to this data.

So um it it's also applicable for, you know, forensics when you need to go do some security research. Um so some customers tell us in terms of the data size, that primary data is that kind of small comparatively to the secondary data. Does that resonate with, with those of you who raised your hand? Yeah, some people say it's like, oh it's huge. It's giant like the secondary data is petabytes and petabytes of data that's on s3.

What we want to do is we want to extend the view of amazon open search service into s3 so that you don't have to jump between different analytics tools, analytics tools that are often not curated or specific to log analytics. And then sometimes when you do use tools that are specific to log analytics, they don't come with all of the features and functionality like anomaly detection and security detectors and geospatial support, right? That's comes right out of the box with opensearch service, all with no additional licensing fees, enter zero etl integration with amazon s3.

So what we're doing now is we are keeping you within amazon's open search service. You no longer have to jump between tools to get access to your data. You no longer have to create one off ingestion pipelines to get that data into amazon opensearch service. By the way, we kind of we call that kind of stuff around here at aws undifferentiated heavy lifting. We want to avoid that, right?

So now you are able to query data where it rests right from within amazon opensearch service. Let's take a look at the architecture. You see amazon open search service. This is your primary data. This is your important and urgent data, right? Time sensitive data that you want to run alerts on.

And then over in amazon s3, you have your secondary data. This is data that's oh, it's just too big. It's too, it's overwhelming to fully ingest this into amazon open source service. So it's stored in s3

Now you see in the middle, we have glued data catalog. What we've done by adding a new data source between Amazon OpenSearch Service and S3 is, we've created this pipeline that exists between the two and glued data catalog. The tables within glued data catalog represent the data that's on the other side of that pipe. Ok? But when has anybody performed a direct query? Because as this is essentially a direct query, right, you're gonna be able to directly query content in Amazon S3.

When I've performed direct queries, you know, sometimes I'll just hit it and I walk away and go grab some coffee and maybe I'll go walk my dog. Um we know that Amazon OpenSearch Service customers. I'm not saying spoil. I'm not saying prima donnas. I'm saying high expectations. We are blessed with an inverted index that returns query results very very quickly.

So we thought, you know, how could we take the best of this indexing engine and apply it to data on S3? Well, you can see in the architecture, there's that table there in glued data catalog. What we propose is to create a skipping index or a materialized view or covering index. You can use that to accelerate and boost performance depending on your use case. So you can apply a skipping index and very quickly get access to your data. Comparatively to a direct query.

I don't have any other improvisations for materialized view or covering index. So if you have any suggestions on how I should act that out and feel free to let me know. So it's acceleration options that work for your specific use case. Skipping indexes are great. If you're doing deep analytical queries and you're, you just want to jump in and dive deep into that data, you want to build out rich visualizations within dashboards. You have materialized views that can create, you can create complex agg aggregations that will support your beautiful dashboards that you're gonna build. And lastly, if you want to do a um real time analysis on data where you know that you're going to be querying that data a lot, then you can create a covering index and ingest that data into OpenSearch Service and be able to take advantage of all of those out of the box features, enhanced capabilities that OpenSearch Service has all with no additional licensing fees.

But I think I've talked enough. Is anybody interested in seeing a demo? Sweet. All right, let's get started. This is where the praying starts. Let's pretend does everybody like to pretend i like to wear costumes and things like that. Let's pretend that we have an existing OpenSearch Service cluster running 2.11. So we'll go ahead and open up the details page. We'll scroll over to the connections section in the tabs. I've already created a data source, but essentially what you do is you create a new data source. You'd use IEM policies to restrict the what this data source has access to. So you can specify which buckets you want this data source to have access to you specify to what extent you want this data source to have access to Glue Data Catalog databases and tables. And then in addition to that, there is a custom trust policy that you'll need to create. This has all been documented in our documentation. Very verbosely, I know that I am roles can be a little bit tricky. So I wanted to make sure that you guys had some really great examples in our documentation which describes all of the different actions and resources that are required to get this up and moving.

Ok. Cool. So once you've created this new data source, we'll go on ahead and jump into dashboards. This is the data sources, it manifests in OpenSearch dashboard. So when you create it within the console, it automatically inserts it into OpenSearch. I said dashboards, but it's really just OpenSearch for those of you who don't use the dashboards. It's fine. Um so the data source is automatically installed on OpenSearch. So let's go ahead. I can jump in and query data or we can take a look at some of the tables that exist within AWS Glue Data Catalog that are now showing up within Amazon OpenSearch Service.

Let's go ahead and take a look at these tables. So right now what's happening is we're calling out to Glue Data Catalog and Amazon S3. This is um Boni talked about service technologies. This is a services technology. It uses um compu serve compute in the background. It will use the same OCU format as you see within serverless as well as in our our ingestion service. The cool thing is about serverless. is it warms up and, and it will uh meet your expectations for the query that you're running like it'll spin up it auto scale up as well as when you're not using it, it will auto scale back down. So when you're running queries, you're going to be charged for compute. But when you're not running queries, no compute charges cool.

So now you can see that there are three different databases that have shown up from AWS Glue Data Catalog. Well, I like it raw. So let's go on ahead and look at BPC flow logs. And now you can see the different tables that are present within AWS Glue Data Catalog. So go to AWS catalog or AWS logs here. This will load up and you can see I've already had a skipping index installed here for the demo, right? I want to make sure things are nice and fast for you. But in the event that you wanted to accelerate the table though, all you need to do is go over to accelerate table. And this is where you would specify the index that you wanted for your specific use case.

So I called out in the presentation earlier that you know there are different types of indices that you can use. There are different use cases. The skipping index is great for just doing your direct queries. Materialized views are really good for building out visualizations and covering indexes are great for content. When you think that you're going to be executing a lot of queries on this specific data segment, it makes more sense to ingest content into OpenSearch if you know that you're going to query it often, right?

Cool. So this is the workflow for accelerating the data. So we just specify our database and then our table comes up and i'll do skipping index. You can specify how often the index refreshes. So you could have it set to auto refresh or you can have it set to do it at a given interval, i will do manual refresh. And then here you can add all of the fields that make sense for your use case. So let's say that as an example, you have a run book that you know that you will encounter. Um inevitably, unfortunately, when certain events happen, when things go bump in the night. So if you wanted to very quickly get access to this data, you could conceivably set up a skipping index which would allow your engineers to very quickly get access to the data and move on with their lives instead of going and grabbing coffee or, or walking your dog and waiting for a direct query to respond.

Cool. Now, let's go ahead and have some fun. Let's jump into discover and do a few queries here. So i mentioned earlier that i'd love to pretend. So i'm going to pretend that i'm a security researcher and i want to go in and look at the top traffic sources within my VPC flow logs. So pop in query, we expand it. So y'all can see it here. The neat thing about these new data sources and connections with this uh integration with Amazon S3 is we also support SQL and PL. So you're able to search in the languages that you're most kind of familiar with and, and um comfortable with a lot of customers have been asking us for better sequel support that is not just specific to the query workbench, which is where we have our sequel support. Now you'll see us bringing better sequel support and PL support into the experience overall as time goes on.

So the query is running again. The wonderful thing about serverless, you're only paid for the queries that you're running. If you're running an index, right? That's a query. You're gonna, you'll be charged for indexing capabilities as well. Oh, boy, here we go. We've got a bunch of results coming back. Yeah, let's do another one. Let's do a little bit more of a complex one. Let's traffic patterns over time. The other really cool thing about serverless is, you know, the first query that runs does take a moment, right? Because things are starting to warm up, but subsequent queries that run run faster, right? Once the instance is, is warmed up. And so we'll keep the instance open for um a certain amount of time to allow you to perform those subsequent queries. So if you have a kind of a complex run book that you're working through, we'll keep that session open so that you don't have to warm up each time that you're executing the queries.

Cool. So we've got results back here as well. Fantastic. But what if you're a simple person like me, right? I like pictures, big pictures, i like dashboards. Let me show you a dashboard of VPC flow logs that we have set up that's sitting on top of data that was generated from mess three. So what we ended up doing is we created a materialized view and we're using that materialized view to help power a VPC flow log dashboard. So I can make adjustments to some of the options up at the top, you know, to accept apply changes. Now, my dashboard updates. Really, really awesome. So now you again, you don't have to jump between tools. Um you can do everything that you need to do for your analysis within OpenSearch Service.

That's it for the demo. That's uh to wrap it up. We already talked about the wonderful things that we can do. We now can direct query content that's over in Amazon S3, we can build out wonderful visualizations and we could do just in time analysis, really exciting things that you can do with that secondary data that you just didn't have access to that. You know, you had this operational friction or tool friction in the past. That's no longer a problem. You have access to the data that you need to, to keep your business running, to wrap it up overall. Yeah, i don't think the that's working to wrap it up overall though.

I'm, i'm incredibly proud you saw me, you know, hooting and hollering over there. Uh when poverty was introducing some of the newer functionality, i'm, i'm really proud of the things that we've released at re invent. And i'm proud because, you know, our main tenant within OpenSearch is better price better performance and we continue to execute and innovate on behalf of our customers doing that, whether that be introducing serverless vector database engine in a way which will automatically scale up and down depending on your use case and help you build out a i powered applications. Whether that be you heard me get excited about or one i talked about that at the beginning where you have better price performance as well as um durability. And then lastly, you know, to complement it all with zero ETL integration with Amazon S3. Now we're unlocking the power of that secondary data that exists and making it easier for you to keep things running within your business.

Thank you very much, everyone uh for coming out today. We're very grateful. Um i, i'll do, i'll leave you with one challenge. Please fill in the surveys. I know everybody bugs you about that. I'll be another one that bugs you about that. Please do fill in your surveys. We value that feedback and we'll take that into the future presentations. Thanks y'all. Appreciate it.