Modernize and gradually migrate your data model from SQL to NoSQL

All right. Hi everyone. Welcome. And thank you for joining the another breakout session today. Uh I'm actually surprised to see so many people who are interested in making old things better than building new things.

Um today's talk is about modernization and how to gradually migrate data from SQL to NoSQL. Um whether you are enterprise or startup or a medium sized company, you will find yourself eventually in a situation where you have systems, old systems, legacy systems that rely on SQL data and they're getting slower, you have bad performance. They don't let you to innovate, they don't let you to build things quickly to use new things that we announce at Dreamt.

So um the thing is how do we move there? How can we make sure that we can migrate from SQL to NoSQL data and actually adopt new challenges. And today in the talk, we will cover this um this topics.

My name is Alexander Schüren, I'm a Solution Architect at AWS. Um and today with me should be Darko Mesaros, but unfortunately, he has a live stream session on Twitch covering a new service that was announced on a keynote and um that's why he will not join us today.

Um today's topic, today's agenda will be first, we will cover what is modernization journey, how does modernization journey look like for typical customers and how does it fit into rearchitecting systems? And especially considering the database size because it's easy to think about how can we improve services and applications. But as soon as we get state and database and we don't, we can't do it on an on an offline base. Things start to get a little bit difficult.

Um in the mechanisms chapter, we will cover the approaches that we have collected so far. I'm part of the team called Develop Acceleration Team. And we work with customers who are interested in modernizing the application. So the typical question they have is how do I make the transition from the old system to a new system? How do I design things? What are the patterns I can use tooling, best practices and how can I adopt it very quickly? And over the time we collected, we worked with customers, we collected all this experience, all the things we tried out because there are no blueprints for rearchitecting systems yet. And thus, we try to come up with a mechanism that we can share this experience to you. So your modernization journey might be more enjoyable and uh a little bit faster.

Then in the testing phase, in the testing chapter, we'll take a look how can we make sure that when we continuously migrate things that we actually catch the errors, how we can handle failures and at least know when things go wrong and remediate and roll back things.

And uh by the end, we will have a demo where we live migrate hopefully without issues. Um a small how just to demonstrate all these things that we will cover the mechanisms, we will have a q and a session at the end. So if you're interested, please collect your questions. Uh and we can discuss them as well and have a little bit more in the discussion side.

All right. Um modernization. So a lot of customers find themselves in association. Ok. Things are slow. We want to adapt new services, we want to maybe look into servers in event driven architectures. How do we get there? But the issue that they have is they have hundreds of thousands of applications and database sitting somewhere either on, on on premises or in different cloud providers. And they want to move to AWS.

We have invested a lot of time and a lot of effort into finding the best way to migrate the workload. And um the more we do that, the more complicated it gets because we realize that customer workloads is actually highly diverse. So we have by now programs moving mainframes and SQL, Microsoft, SQL specific things and Oracle uh and many different types of workloads. But the general way how to do that is typically to first make a discovery and assessment phase where we collect all the information that we need. Technical information, server names, connections, who is talking to whom? Why is it important? Um how important is it for business and how can we actually move these things um in the future?

And then there is an important part where we have to distinguish what should we do with this application. And uh this is the famous R that we have. So you can repurpose things you can retain. And and my favorite is tire actually deleting code and generating customer value is i think it's the best feeling you can get.

Um but the more we go to the factor to the refactoring side, the harder things actually get. But the more options we have because when we reflect the application, we have to put a lot of effort, we have to understand what is happening inside. What are the dependencies, what will happen if we change things drastically? So you need a robust process there, you need to spend a lot of time and understand it. But the business value that you can generate for the customer is actually the most.

So think about moving to serve this application and adopting, adopting um purpose built databases that will increase the performance and be more secure for the customer. But the way there is much, much harder than just relocating or making lift and shift and based on the data we have collected. The typical way is just to rehosts things or relocate maybe or we put things in container and then we realize at some point, ok. Now it works, uh we can scale actually containers, it's fine. But now we want to uh rearchitect the system.

And the typical case as well is um with the database because while we can split monolith into microservices, there are a lot of patterns like a strangler pattern. For example, where you can provide an API and just route the traffic to a different system. For databases, it's not that straightforward.

And thus only 5 to 10% customers embrace the way of rearchitecting. But we see this number is growing and we have to do more in this area to help customers to rearchitect this system just to recap on SQL. NoSQL.

Popular choice for moving SQL data is NoSQL because a lot of stuff that was designed in the past was for the SQL database. And we put everything in one database and realized after that, that it's, it's not the proper way to um to optimize it.

So for the SQL part, we have um relational data. And on a NoSQL side, on a DynamoDB, it's a key value storage and the advantage for NoSQL is mostly the scaling part. So you can throw more data, you can throw more uh volume of data there. But the performance, the query performance will stay the same or slightly improve uh increase.

Um but of course, there are some disadvantages in terms of queries because uh we have to spend a lot of time to think about the, the structure, the schema for NoSQL database. And uh we have no flexibility compared to SQL.

And when, when a customer comes to us and say, ok, we, we want to move now from uh SQL to NoSQL. We think it's a good idea. We see the value for the customer, it will be performance. We have we reduce cost and there are some business benefit and then there are a bunch of our notes they they face.

So the first thing is how do we design the scheme? How do we do that? Um we will not cover this topic today because this is like an in depth topic and we have lots of great speakers as well as re invent and past sessions as well like Alex DeBrie and other people that wrote written books on that.

Um but other things like ok, when i move the data continuously and this fails, what should i do? How, how do i make rollbacks in this case? Because this is continuously changing process or uh how do i validate? Because once you move outside of the sequels, you don't have the schema. So you have to take care of that. You write the right values that they have the type and so on.

And then you have n boss, which is, i think my personal nn boss is uh sort procedures when you spend a lot of time discussing what you can do better and then suddenly customer drops in. Oh, by the way, we have like hundreds of sort sort procedures and uh we don't know what to do with them.

Um i think this is completely different topic to cover and we will not do it today, but we will think about data consistency and how to do it continuously as well mechanisms.

Um when we start working with customers on migrating things, the the first thing is how do i start? What should i do first? What is the first step i need to take? And it might sound weird but we don't start with the schema. We start with a customer in in mind and this means that customer need to spend um time to understand what would be the value if we design specific data for an application for another system, for an api. And based on that, we can actually derive more information, more requirements on access pattern. How often will it be queried? Um how big will the build the payload? Uh how often will it change? And only when we have queries only then we can start designing schema because this is like completely opposite that we are used to work in SQL space.

Sorry about that. Um on the mechanism side. So when we start, we think, ok, what should we do? We have an idea, we design the schema, we all have all the queries. Now we have the source database and now how do we get that to the target?

And if you have an application with the SQL site, the first thing you need to do is introduce some form of detecting and uh copying the data changes. I mean, most of people probably did it before in the m microservice environment where you want to propagate data changes. This is very common.

Uh but the second thing is that we need to do is to also move code data because many people also with SQL service don't realize that uh the, the the access pattern to the data is not equally distributed. That means that if you, if you look at the data, how often it is queried in the table. For example, if you have billion rows, only a small fraction of this information is queried or change.

So um that's why there is we, we have to, we have to look inside the data and say ok, what what percentage of data is actually not touched and it's called data and we can actually upload it without doing anything. So this is the easiest part of continuous migration in this case.

Ok. So now we have to find out what, what is the mechanism, how can we go and find changes in our system? And the very famous one, probably most of people uh heard about that is change data capture.

Um how many of you used it before or or heard it? Oh, nice. That's good. I think it should be more hands because uh Werner Vogels uh in, in, in a keynote today, actually mentioned how DynamoDB replicates the data exactly with the same mechanism. So they stream the data changes and then they replicate it and apply it somewhere else.

So this is the mechanism we have on a database level side when we want to take a look in the database and say, ok, we want to look in the data inside the database. But then there is another approach that is uh ah sorry one slide.

Then there is another approach, it's called dual writes. Uh who did this one before or heard about that? Oh not so many. Well, that's interesting. Cool.

Um do rights is uh something we want to do on an application level. We don't want to look in the database. We don't want to write queries. We are developers, we are lazy. So we try to modify code, introduce some changes in a secure way and then make ship a cop copy of the data or else.

So let's start first with the change data capture. Assume we have a post database and we have a set of requests that are incoming on inserts updates and deletes. And uh what what's interesting about SQL database is most of the relational database management systems. They have so-called transaction logs and transaction logs is a document that stores information about every data change in the database. This information is super helpful if you want to do rollbacks. If you mess up something, the uh the the system reads the transaction transaction logs and said, ok, i i'll revert the change because i know what changed before.

And the transaction logs are also used for replicating the data. Um for example, RDS uses it heavily DynamoDB uh RedIS use replication as well. Uh for the, for that um so on, on when, when working uh in AWS environment, what really helped us in the past is to take a look at the um AWS Database Migration Service. And DMS is more popular for SQL to SQL migration actually. And it was initially designed in the mind that you can move from different SQL services uh uh service, you can move from Oracle to Aurora and to use more open source technologies.

And the way it works is that you first define the end points. So you, you, you need to configure the, the source endpoint and the target. So you have the communication in a secure way, you need to create the replication instance. This will be an instance in the background in situ instance. And uh you describe in a task which should be done. So this is a document that actually specifies what table should we read? Which column should we take uh as a, as an input and where should we store it?

The interesting bit is that DMS supports not only uh DynamoDB for NoSQL changes, but it supports like way more uh things that you can do and you can combine them as well. So you can ship your data first to S3 maybe or to Kafka or somewhere else. Um and then do something else without uh uh something else with it. It doesn't have to be DMS.

Just, just to be clear, there are open source solutions that support this as well. So for example, if you think of Debezium, you can connect it with the Kafka and then you can process the data and ship it somewhere else and do everything with it. Uh and the same works as well with the Kinesis. So it's more generic approach where you have more flexibility to, to stream the data first and process the high volume of data through Kinesis. And by attaching the Lambda function, you can just do whatever you want. You can do validation transformation within it. You can discard things filter. Uh it's just up to you how to, how do you import it.

So how does the DMS task look like the document? Just, just to give you an idea how, how it actually is inside when you define it.

So DMS task is a JSON document and it's in an area of an area of uh rules and the rules can be different uh can have different types. So there is a selection rule transformation. We don't, we will not cover them today. But the DynamoDB specific rule is object mapping. And this is where we can say we want to pick specific data point in our SQL database and we can map it to a record in a DynamoDB.

So here's an example when we have a, a rule, an object mapping rule and we have an object locate that says we want to have a customer data, we want to collect uh customer names and to insert a composite key, a key that contains two bits of information for DynamoDB, we can combine and they, they come from different rows actually. And this gives you flexibility to, to decide how will my key look like and what additional attributes do i have to put there?

The interesting part is also that it doesn't have to be a table. You can do it on a view as well, which means you can preconfigure how your data should look like in a DynamoDB. And then you can just read one record at a time and you have exactly your key value pairs

And the rest as well, there are some limitations on that. So you can do something like a blog files or large files doesn't work and uh on um types so if you have something like um time stamp, you have to convert into string to a different format and then to convert it back based on, on the information you need.

And we are here not to discuss to say this is the best approach. So in case you came here and, and wanted to know just a single approach how to do it. Uh unfortunately, as always an architecture, there are tradeoffs and this is something we we discovered over the time.

And there is there is no point to say this is the best, but it's valuable to see when should we apply this approach when we have benefit for that? And what are the challenges if we apply that?

So first is you don't need code changes for that, which is amazing because in some cases, some customers had systems that you cannot change really fast. So it takes eight weeks or maybe a couple of months to test every change. So if you want to have some changes on a code level side, like with dual rights, uh it wouldn't work because they just way too slow to roll it out and to test and have iterations. If you're in a situation, that's fine. You can say i don't want to touch a code, just give me the access, read access to the database and we can ship the data continuously. It's not a problem.

The other part is the um the changes are atomic which means everything you read in transaction logs is actually the right correct information that you can replicate somewhere else. So you can focus your failure handling and anything can go wrong with data validation and data consistency. After that, you don't have to, you can rely actually on this mechanism.

Mm because sql database exists for a long time. Now, there are a lot of uh vendors and products that support cdc. Uh and i, i don't have the names here because there are way too many. Um but you can, you can find something that will suit you if you go this far.

Um and the other one is, it's actually suitable also for moving large and, and huge volumes of data if you have the requirement. But there is a but there are challenges. Absolutely.

So if you, if you go this way, you have to create a lot of infrastructure and this means you have to make it more secure, you have to spend more money on that you have, it's, it's complex, you have additional work to do besides your uh shipping the features.

Um the other thing is you can't sample or it's way way harder to sample and segment data. So what we think about continuous migration and this, we will see it in an example uh in the demo, we, we don't want to move entire tables. We're not thinking in, ok, we move data base or we move a table or we move these five tables at once because you don't know how many records there are like thousands or billions of records. It's hard to do that and we want to gain confidence over time.

So we want to focus the migration on very specific set of data. For example, i want to move one customer, just one customer and i want to migrate it continuously. I want to gain confidence that it works and then i can expand it. And with cdc, there are options, it works, but it's um you have to spend more time if you want this flexibility.

The other thing is you're operating on a database level. You might need some somebody who understands sequels who understands your d ms uh uh database management system and um things go wrong. You might need to configure stuff for transaction logs or you might need to look into internal logs of the database to see why things didn't copy properly. Why couldn't we read that and so on?

And for this is this one is specifically for d ms and also other other approaches that we have to provide a document like a an it's not an instruction, it's just a set of rules and these rules will be applied. And because we are in the collaborative space, we are not in the imperative space, we cannot just write code and then this code will be executed and you can do whatever you want. You have specific features in this product that you have to use and you are bound by this limit, you, you cannot escape them.

And there are ways how to make it more generic as i mentioned with kinesis and then you attach the lambda and then you are free to do whatever you want. But if you are in this space, you are limited actually to do that.

The other thing is dual rights. So dual rights is basically on a application level, we don't access any data. On the sql side, we only operate on uh on application level. So for us, it actually doesn't make any difference whether it's sql or no sql. On this side, we just blend it out and say whatever data is there we rely on be it relation, object, relational mapping or any other function. It will get the data from the database and everything else we will do our own. Ok?

So let's take a look how it works. So we will have an example here. So let's say we have a service and it writes data to database. The first thing we need to do, we have to add the dual right capability. So when we process orders, we um we first write it to the database and then we have to add additional function that says, ok, please ship a copy of this data to a new service or a new piece of code, whatever it is that can write it to no sql database. Ok? That's fine. So now we're just writing the copy and this is the mechanism. This is the first step and we don't rush, we just select, for example, one customer or one region or product type.

And then we run it for a little bit, a couple of days, maybe a week. And then we collect the information and we start thinking about, ok, how many rights did i do? How many of them are different? Um and there are some other mechanisms how to gain confidence. So we can have a good feeling over time. Ok? We are doing the right things that it is correct and we can expand it.

Then if we have a good understanding, we think, ok, i think, i think i'm doing a good job then uh we change the read priority. So instead of reading from the um sql database, we, we make a switch in our code and say now every read goes to um to our canary, let's say as a low sql database. So in this case, we just write uh just read it from the new database. And then chances are we did something wrong. So you can read from the old database there as well. This is the crucial point where we have to get our confidence here helps us to make sure that we switch the source of truth.

So in this case, before we change the right property, everything, the source of truth is sql database. So whenever we read something in no sql, we have to double check. Is it there as well or not? And over time again, we will run it and we collect the data and we'll find out that both records actually match and we are in a good spot. So we can continue, sorry.

Well, um exactly in the background during this whole process, after the, the changing the weed priority, you can start thinking about moving the data um to um moving the cold data to the no sql database. Ok? And then after a period of time, again, we gain confidence on, we can write to a different system, we can read from a different system. Now we ship or we flip the switch from what is the priority? What is the source of truth for this specific use case? Again, we are always operating on, we are migrating one customer or we are migrating a subset of data. And this will help us to understand that the mechanism actually work and then we can expand it.

So we just write to nosql and the old database is stand by. It's not only only for only this customer, only this portion of data, only this segment. Um and everything else goes exactly as it was for everybody else.

Here's an example, we have worked with customer. Um i can share, i can share the name but i can share the details a little bit so they had a huge landscape of corporate uh center sequel servers. So they had over 1000 servers, each server designed for one tenant. So they had tenant based isolation on the database level and they had a huge load actually on the on the sql server. So they had a front end application pulling data all the time. It's like the same data um was pulled, but they couldn't catch it properly.

So they decided to go the better route and just redesign the application and um move part of the data that they have on the sql side to dynamo to be. And what they did is they instrumented the um the application in the ecs cluster and created new application for them. It was uh sync and lambda and dynamo db. So on sync, they could actually replicate portion of the structure that they had on the sequel side, but they can improve it because there we have also types, we have the schema and we have the queries. So we have roughly the same understanding how it works and how it should be shaped.

And once they gain confidence again, they run some experiments, some tests, they collected metrics, they put tons of observable there and then they moved one customer by customer one at a time. And after they switched uh the in the front application, they had a different api call where they said, ok, this type of data i want to get from my a sync and the other, all everything else that goes from the old system.

So do rights are also controversial. If you read a little bit on the internet, people are a little bit scared, but there are some advantages when it works. So first is we can easily, really easily to segment data. So because we, we can use feature flags, we can have branching, we can have different functions, libraries, everything we can use inside the code. Uh we have the options there.

Um and it's less infrastructure actually. So you don't have to any new things. You don't have to configure any im policies, any networking, you don't have to spend tickets and costs and so on and so on. Everything happens inside the code. And if you can change your code frequently, this is the best thing you can find.

Yeah, but the challenges are the inconsistency and this is the biggest challenge that we have actually because things can go wrong. So you can write to this to the old database and then you ship a copy and then suddenly the database um make a rollback, whatever for whatever reason or the transaction fails and we didn't catch it. So you have a just the wrong data on the other side and you have to somehow find out or at least find mechanisms on how to catch that, how to make sure that the data is actually consistent on the other side.

The other challenge is complexity. So if you build more and more functionality around shipping, copy somewhere else, then you have your error handling, then you have some more tests that you have to write. So the lot of code you have to evolve. And this is again, this is not something wrong per se, but it's just the options we have. So if you go this route, we have to spend more time on, on the code level. If we go the other route, uh the other way we spend more time on infrastructure actually.

And this one is also crucial is because you will have a lot of data copying inside the application. You have to take a look uh at the resource you are uh you have assigned to your um system because if it's running somewhere in a container and you expect like how much memory you can use, how much cpu you can use, it will increase. Of course, because you have more activity on the application level

Testing. How do we make sure that things we're actually doing are not broken? So i call it the spectrum of uncertainty. And this is what i, how i feel when i develop things. So i'm usually here somewhere on the left side where i'm pretty not, not really sure that that's, that's the right approach. I try to experiment and i need, i need some, some data, some tests, some validation that i, i did actually good job. So it will be similar for when i run the demo, i will be on this side as well. And then we have the other part where other people are as well where we have. Ok. We, we think that's a good approach. I think we can, we can ship it and we are ready to go to uh to the production and there are ways how to come, how to get there. And this is just a subset of things you can do um to get there.

So first thing you can start very easily just to compare a number of rows very, very easily. You collect the data and then say, how many things did we write for this customer in this database? And how many rows do we have there? Do they match? No, we have a problem. Let's think about that and let's find out then we can increase things and start looking into what are the values we are missing because the the most problem we have there is we cannot compare row by row and we cannot like compare everything. It's it's way too expensive there.

So we start probing, we start experimenting and finding things that might not work can be attribute types. Uh it can be aggregations like min max value. So for example, if you have sales data, you can just sum it up or pick an average and if they match, then you are good to go. So it gives you a good confidence that things are actually correct but not the confidence that you can ship. But you need to add more and more layer by layer. So it works.

And one of the things also that worked really well is randomized sampling. And this is where you start probing and just put some random samples from the source and the day and distribute uh destination. And you pick, for example, 1% of the data at random from the source. And you get the ideas of the records, you roughly can estimate what it is and then you compare the same stuff, what is uh on the, on the target side. So this means you have, you spend less time and it's less cost as well to compare everything. But if you do it more often you get more, more and more confidence.

The goal is not to compare everything, but just to find out whether it's not working. And then the question is, when do we test like what, what are the hooks we can, what are the steps we can do uh on the on the architecture side. So if you have a service and it talks to, for example, dynamo db, the first thing you can add is a validation step. It can be a separate lambda function. It can be um something that uh just checks the types, the values the structure and you make sure that your inputs are correct the issue is that we don't have the schema anymore. On the no sql side, that means that if we need this strict validation, we have to implement ourselves.

And on the right side, we can do monitoring. And this is a really important approach to add more observable and to run reports and to get more data and more confidence that things we are actually shipping are correct, that they are matching that there are no errors and we can generate reports based on, for example, sampling or um other aggregations as well.

All right now it's time for a demo. Ah give me a sec. So in our demo, we will migrate one entity today based on two tables. Uh i have to switch it first

give me a sec ah beautiful.

all right. so we have, we have two objects, two tables in our database and we map them to two objects. in our case, it's a device and an event and each device has a an id number, some type, some status. it doesn't matter actually what, what kind of data is there for now.

and we have a sort of uh a set of events that can happen and our devices emit these events and they send it back to our application and said something changed here. you need to sort this data all the time constantly and these are two tables.

so this means we want to ship the events in our dynamo dynamo db and for that, we have constructed a new event that takes uh an event id, a device id. so it's a combination of data from both tables or to both objects.

um all right. so what we do now we have two simulators. so one is we'll be writing all the time and the other one will be reading all the time. and how we implemented that is that we put some feature flags and we will run deployments to aws with cdk where we switch the feature flags and we change the read and write priority for specifically this read operation. ok?

ok. so the writer is sending events all the time and this is good. oh no, this is wrong one. ok. good. ok. it's going on now, let's take a look at the x ray. so it will take a little bit of time.

so the way we have i can show probably the feature flags first.

um we have two stacks. i've written it in cdk with python and this is a simple database stack with the rest api that we have here. and the python function that handles all the data processing.

and we have a feature flag stack where we have defined this schema that we walk you through quickly. so we have three flags.

the first one is um what is the customer, what, what is the tenant id or the specific tenant? in this case, we want to migrate and in this case, if it matches to any company, then we will apply dual right? and we will ship a copy of the data somewhere else.

then there is another flag where we just can turn off the, the writing itself. so we can first say, ok, we now we want to switch the flag and write data and the the the benefit here, you can switch it back and forth and just collect a little bit more information and data, how it works, how your mechanisms uh are configured if everything is correctly.

and uh if your observable tools show green sign and then we can flip the switch again for the reading operation where we say from now on, for this customer specifically, we don't use the old database. we switch it to the to the new database and this happens without changing the code. so i just provisioned everything. the only thing i change are the flags. ok?

so these guys are running. yeah, looks good. and here we should have a x ray map. ok? wonderful.

so you can see here, i, i'm not a good magician here in vegas. you have many magicians. um i cannot hide where ii i didn't create the illusion to have a sequel or on the back end because for dual writes, it actually doesn't matter if you, you write the sequel to not because you operate on an application level, whether you have things in the sql database or somewhere else. it's, it's irrelevant in this case.

so we write in to the lambda function and it's just in memory storage that we have. and from there we copy the data to dynamo b. so there is no dynamo b yet. we're not migrating everything is great. now, the tricky part life deployment on, on stage, ok.

now we switch the flag to truth and we say cd k deploy feature stack, it will take roughly 40 seconds, i think um to deploy everything. oh, that was fine.

so in the dynamo db, we don't have anything. here's the, here's the tail, make it a little bit bigger. uh nothing there still publishes. it was faster when i was in the hotel. uh maybe i can show something else.

give me a sec. ah here's so what i also did, i said i want to see all the reasons, right? so i changed my code in a way that i emit metrics for every operation i run to the old database and every operation i run to the new database and this will give me an indication. ok. am i doing good or are the numbers different? and this will give me a hint? ok. i need to change something. i need to go into detail. why things are failing?

so we have, let's say for the last five minutes, we have um 75 reads and 82 writes which is good. oh, deployment is done. yeah, of course. i have the new version of cd k. sure. let's take a look. here. we have something. ok. now you ship a copy of very specific customer to a new database and we can take a look and we see there are a lot of events but just for any company, nothing else.

all right. and our metrics should be updated. ah, here we go. successful db rights. the issue here is that we have the rights from the previous run. so here are successful 17 rights to the old database and uh 24 and 17 to the other. so over time we get some, some metrics but nothing failed. luckily, um that we can do and now they equal. so that's good. perfect.

so now everything that goes into the uh old database is copied to the right side and i can also show the code how it's done in a second. i, we flip the second switch where we say for this customer, we want not to read all the data from the old database. and uh if something fails, if we didn't find it in the old uh sql database, we make a fallback in the new database, we make a fallback and read it from the original. ok.

so here is everything good. yeah, this one will not change significantly. so we might also close it. yeah. so we're looking for this number here to change at some point. maybe i can show the code how i have implemented that.

so we have one main function. this is the, the lambda function by heavily using aws. um our open source project aws lambda power tools for python where we instruct here. uh our um i'm the handler to handle all the api codes that i will have. so it's, it's configured in a way that whenever specific api is hit, it will propagate the right objects to my annotations here.

so i have uh get method for devices, i have a uh put method put in post for specific device id on the events as well. so whenever i write something here, i create an event first, i use my store and this is an abstraction that helps me uh to test my my uh store operations. so i can put some different stores so i can store in sequel in dynam db in the code that looks the same, but i just insensate different store in this case.

and then i had the metric because this guy was good. and um otherwise it will throw error and something else happened. but here i can collect metric all the time and say ok, i want to add a metric to my successful reads uh rights. and then i try dual right um to, to copy the data somewhere else. and here as well, i annotate which, which device id did i try to, to copy somewhere else. so it gets me more visibility on the observable side.

and um i use um i fetch first the more device information i need so i can combine the data. and here i use the feature flags where i can say, ok, first, i want to evaluate the flags for am i allowed to write things or not? and if it's not the case, then this whole block will not be executed.

and then the second part is also, is it the right tenant to migrate or not? so it helps also to segment things to scope it a little bit more, a little bit less on, on to migrate less data. and in try, i just use a different store as you can see the dyna mo to be store and i put an event uh in a new object. and then after that, i put a successful metric, right?

so at the end, i should have some nice graphs here. all right. so we can see the reeds match and the right match which helps me to move forward and expand this thing and say, ok, i want this tenant is good. let's try a different one and then you will find out ok. these things are not properly working. i will need to fix something.

all right. um this guy is running. so just to summarize things always starts with customer whenever you want to migrate the data to nosql, it's, it's definitely tempting to start with the entity relationship diagram and draw everything and just come up with some objects and classes and everything but try to go first to the customer side and understand what is the data structure they're looking for. be it an application that is used the front end, the mobile application or be it an api or an external service?

take a look at whether cdc or dual rights fit your case because there are some challenges advantages. they are not equally good or bad, they are just fit for different purposes.

in this case, feature flags are like the most interesting and the most robust concept, i've seen to segment things to, to take just a portion of data. and not to say we think about moving thousands of database at once and we make so much effort to do that. i mean, it's viable for different customers in different use cases. but when you want to scope it down and you want to do it continuously, this is a scary need to kind of like train yourself and and just find out how can we scope it? how can we make it much, much smaller gain confidence with validation?

there are things that will fail. it's it's inevitable, you write things that will fail, you will find out that data types are not matching. there are different order of things that it's it's inevitable. so trying to add more layers of testing, validation and monitoring that will sorry, that will help you to to gain confidence to at some point, expand it in the future.

lastly, this is the common situation. we see also customers, they start missing sql features. this is like, ok, schema is actually a nice thing to have because it enforces everything. i don't have to write my function. i don't have to validate every type and everything. i just write it once and then it works. so there will be features you will definitely miss in sql and be mindful of that. it will teach you what you actually had and didn't pay attention to and you might replicate them if necessary.

and the last thing is absolutely vital and this is also aligned to the, to the uh core messaging that the we of vogels today said on, on the re invent is evolve or die because we have so many old systems and it's absolutely challenging to, well, it's also tempting just to start writing new systems like ok, i'll write new system and we will do serverless and event driven everything, but it's much, much harder to evolve things.

and the thing you write today as a greenfield application, you will have eventually to rewrite it in two years when we have new services or new libraries, something else. so i think from my perspective, it's super vital to take a step back and think about how can we train ourselves to evolve systems that we have right now and be more brave to, let's say, not fear any legacy system not to run away from them and say, oh no, this is a legacy project. i haven't, i don't want to touch it.

um try to try to find ways to improve it. there are solutions, how to do that. there are ways, try to be curious and, and find new ways how we can make business successful. but by turning all systems, a new system and improve over time gradually.

all right, we have time for q and a. if you have questions, we have mike there.