What’s new with database migrations

All right, the uh the music stopped. I guess that means we're uh we're good to go. Uh hopefully everyone's Re:Invent has been going uh splendidly to this point. Uh everyone's looking forward to Replay tonight. I promise we're not gonna go too long. We're not gonna keep you between, you know, learning and drinks. I know where the priorities will come for most people, but you know, don't want to impart upon you my own beliefs or anything like that.

So yeah, let's get started. Uh start uh learning what's new with database migrations. My name is John Winford. I'm a senior manager in the database services organization. I've been with the AWS for many years. If any of you have been to Re:Invent in like the past seven or eight years, you've probably seen me around um I am joined today by Ryan or not Ryan in case that's not obvious that that is not Ryan. So why don't you introduce yourself for a sec?

Thank you, John. Hello, everyone. Uh this is Vijay Kamaji. I'm a principal data, a solutions architect mainly focused on uh RDS Aurora and DMS. Nice to meet you. everyone.

So, uh VJ, I will come back a bit later. Uh in case you're wondering, it's bound to happen. You may have seen it a couple of times. Uh Ryan came down with, uh you know what, on the way here and has spent his entire week in Vegas in a hotel room. So I, I don't envy him at all. Certainly great to have VJ here helping out with the sessions.

So, uh you know, I'm sure you'll enjoy it as we, as we get into it and uh we'll uh make good progress. So what are we gonna talk about today? Quick agenda? Uh why migrate? Um no, we're good. Ok. Why migrate? Uh we don't migrate just for the sake of it, you know, I don't, I don't personally consider it a, a fun thing to do or if that's how you spend your spare time, you know, that's, that's totally fine. But uh generally speaking, there's some typical reasons why people and companies consider migrating and we'll go into these that briefly. You know, this isn't uh this isn't the sales pitch. We'll give you the, the true technical reasons why people tend to migrate and go into that.

Um we'll do a bit of an overview of a typical customer migration journey. Uh talk a little bit about modernization, there's a lot of commonalities in the migration process and uh you know, it's best you sort of understand what these are because, you know, if somebody has encountered it before, it's best you can learn from that for those that you need to do DMS. We'll give a brief overview of the services and the different components. We're gonna go over this fairly quickly though, as we don't want to spend too much time on it.

Overall, what we're here to do today is speak about what's new, I totally appreciate. There's more than 200 AWS services and it's hard to keep on top of what's been happening with each of these. So we're going to go into a pretty uh dense but comprehensive summary about what's changed in the Database Migration Service from last year and how that can be useful to yourself, your company and uh helping with migration modernization, replication, uh data marts, data warehouses, all that sort of fun stuff.

And because I know that probably by this point, Thursday, you know, you've seen a lot of these breakout sessions. Uh death by powerpoint is not something that I think most people enjoy. Uh we are going to spend some time doing a couple of different product demos at the end. So you can see how some of these new features and functionalities work and, you know, give you an idea of how you can actually use the product.

All right. Um ii i don't, i, you know, i just don't think anybody's ever said it, but, you know, our primary objective with uh with DMS here is to make it as simple and straightforward as possible to do a migration. Um you know, the the migrations can be challenging. They don't have to be. Uh as I mentioned before, you know, we've tried to learn from what customers have seen in the past. Uh we've spent a lot of time and effort on iterating and improving our product. The overall objective is to make it as straightforward as possible and as seamless as possible because migration is not what's going to differentiate your business, adding the value on top of your databases and uh creating a, a product that's saleable to the market is really what you're what you're after not migrations.

So you may have seen this slide before. Uh you know, overall, it's a, it's a fairly generic slide we use in the, in the databases world at, at AWS overall. You know, why are people migrating, as I said, um that quote earlier, migrations are fun. That's not the justification for doing it right? There's more to it than that. Uh people tend to start their migrations, especially if they're new to the whole cloud adoption thing. Um starting with the simple ones, they do a lift and shift to easy two. What does that mean? You take your on premises database, you put it on EC2 and you operate it much in the same way you did previously? Yes, there's some, there's some, you know, features that EC2 gives you. You've got uh the power of AWS, you can choose your instance type. I mean, it's a whole lot faster you get an EC2 instance going than it is to go. Um purchase, acquire provision, install the operating system. What have you in the same way, you would have traditionally done it in a data center. But this gets you going, you start to see some benefits. Uh some reductions in costs and increases in innovation from here. People tend to move into managed databases, costs are further reduced here. Um and the velocity of innovation continues to accelerate. But the cost change when you go to managed database isn't necessarily hardware related. You know, if you've been in EC2 and you're going to RDSI mean, there's still overly or sorry, the underlying AWS costs because, you know, we have infrastructure as well in there where you're starting to see the savings isn't on the infrastructure, but in the people capital, you're spending less time maintaining monitoring, patching, you know, all, all this stuff you typically would have to do with the server. You don't have to do it anymore because the managed service is taking care of it. You know, backups, backups are now done with a click of a mouse rather than, you know, running your backup software, monitoring it, offload it too. Well, people don't use tape anymore, but you know, it makes it a whole lot easier when it's a managed service where people start to see the greatest savings and the highest velocity of innovation is when you start adopting cloud native databases like Aurora and DynamoDB. Uh you're breaking up that monolith you no longer have this giant uh oracle sequel server, whatever it is database and you're moving to a microservice architecture that lets you innovate and uh operate on individual components simultaneously without the dependencies that you would if it was a monolithic application.

So typical database modernization journey. Uh most modernization journeys start here with a relational database, not always, but most of the time it isn't just a story of those monolithic databases i was mentioning earlier. Um you know, they do lack agility, they can be hard to scale. Uh there's a variety of access patterns all crammed into that relational database model. That's not it though. It's also the story of start ups and highly agile teams that are looking to, you know, they were trying to get to market quickly. They're like, well, what's, you know, we don't really care about the database, we just need a place to put stuff. Let's just grab the first relational database we can find and shove it all in there. Um they might have leveraged an abstraction layer like an ORM uh the the entire objective of there is like, ok, it's not the focus, let's just use it. And then what happens that start up is successful, it grows, they need to scale and they go, oh jeez, you know, we really made some poor architectural decisions earlier. We probably shouldn't have shoved everything into that database of choice, you know, insert whatever whatever flavor you want. There comes a time when all of these things need to be modernized, the monolith, the start up databases that grew and whatever other scenario.

All right now, let's wind back. Um I want to be clear for most of the session today when I say database, I actually also mean data warehouse, I mean, it's called the Database Migration Service. Could have called it the Data Warehouse Migration Service. I mean, it depends on your scenario. We treat all types of data equally.

Database migrations are not always straightforward. Uh we talked about the lift and shift, you know, this is, this is going to your EC2 platform, we talked about move to manage, this is going to a managed service like RDS or, you know, optionally uh something like Aurora or Dynamo. Uh and then sort of actually Dynamo is more of an example of modernized. So when you're going to sort of a cloud native platform, depending on the path you go with. Uh the tools you use to get there might vary. Um we're gonna drill into the different parts of migration overall though. Let's just have a look at it.

Um when you're migrating in a database, you know, it's, it's it sounds easy like, yeah, just move that. But a database has a couple of different components to it. You've got the, the data in the database and then you've also got the schema that forms the structure of the database. So if you're looking to migrate from, just for argument's sake, a SQL Server system to an Aurora MySQL box or database engine, you've got to worry about actually converting that schema from something that is SQL Server native to Aurora native, right? So there's some conversion part in there and then after that, you then go and migrate the data, but a database doesn't exist um in uh in isolation, it's probably being used by other things, right? It's been used by applications. These applications would also need to be updated to use the new database engine of choice, whatever that is. Uh so there's all these different components that you've got to factor in when doing the actual migration.

And you know, although there is this stereotypical image of it operating in the in the dark basement of an office building somewhere, the fact is we're all doing this to support a business and you've got to think about those business users. So if you're going to do a migration, you can't say, oh by the way, i'm gonna take this super important system that you use offline for the next, i don't know, 34 weeks while i'm doing this, that's not really feasible, right? So you gotta figure out a way to do that migration without impacting your end users uh, to, you know, make it as seamless as possible for them and then all of that, well, before you even get started, do you even know what you have?

Um, I know this sounds like a bit of a funny question. Oh, of course. I know what I have. Believe me. I have worked with so many different companies that have grown hugely over the years and they've got hundreds, if not thousands of databases squirreled away. I mean, I've seen examples of databases that have just been running away on some little desktop machine shoved under a desk in a corner somewhere, right? So, you know, before you're gonna start this journey, especially if you're trying to get out of a data center, you better know what you have because, you know, you may be surprised thankfully at AWS, we have you covered, covered as much as we can.

So how can we help you across that journey? The first set of items are the tools, right? These are the self service tool sets. Um we'll talk a little bit today about AWS DMS Fleet Advisor. So Fleet Advisor is a component of DMS. It's geared toward customers that are looking to move a fleet of databases to the cloud. If you just have, you know, 1510, you're probably not going to use this capability. But the overall idea is is that it will inventory what you've got and give you some information about it so that you can make an appropriate migration plan and really decide, you know, where to start, where to go and, you know, how to make those first steps.

Flu Advisor is a fairly high level analysis. Uh, it will go and as I said, I'll talk about it a bit later but, you know, it'll go and find those databases for you. Some of those databases are going to be, um, your options are gonna be limited. It's maybe the best way to put it. Uh if you've got a packaged application that is certified to work on Oracle, you can't just move that to Dynamo, it's not gonna work. So for those, you don't really need to do a huge amount of analysis. You know, that you need to take that Oracle database on premises and either move it to Oracle on EC2 or RDS for Oracle, you don't need to do a whole lot more than that.

Uh but for those databases that generally speaking, you've probably developed in house, you know, you've got a in-house application that leverages the database. Uh if you want to consider getting away from a commercial system to a cloud data or open source system, uh you can use the Schema Conversion Tool to do an analysis of that database and give you an idea how easy or hard it might be to do that conversion. We'll talk a little bit more about that uh in a while.

Let's go scroll down here to make sure i'm covering all the points. That's fine. That's the wrong button. Obviously. 26. No, that is fun. All right. Got these house. All right.

So then uh talking about the other tool, uh there is, of course, DMS probably didn't even need to look at my notes. Um DMS is like i said, the product that actually moves the data from point A to point B, we'll go into DMS in a lot more information or details soon. Uh it is primarily used for database migration like it's called, but we'll talk about some other use cases later on. There are people that use DMS in an ongoing basis. Generally speaking for feeding data marts, data warehouses, uh extracting information to S3 to do any number of operations on it.

Then we get into the uh the people side of things there just as an army of people at Amazon that can help you. Uh we've got Solution Architects, we've got Professional Services and of course, we've got a massive partner, network partners that are experts in doing this sort of thing, probably operate, you know, in your hometown, whatever that is, you know, there's, there's partners all around the world that are certified in database migrations that can uh that can give you a hand.

And then of course, we have a uh a small specialized team inside Amazon uh as part of the Amazon Database Migration Accelerator. These people can come in help create a migration plan uh generate what we call a solution document, giving you a recommendation on the steps you should take and the potential target for your migration to either uh allow you to get started with a step by step. Like these are the, these are the exact uh stages you should take for your migration or help you get connected with a partner or proserve to go further.

On top of this, we have programs uh programs generally include things like MAP, uh MAP is targeted towards customers doing mass migrations, not just of databases, but of just their wholesale footprint to the cloud. Uh Database Freedom is zeroed right in on uh how we can make that database migration easier for you. And if you need some hands on training to actually uh understand how to do these migrations yourself, Data Labs are a great way to go about doing that as well. If you haven't heard about the Data Lab program, I highly recommend you take a look.

All right. So we are mainly here today to talk about DMS. Uh as I mentioned, DMS focuses on migrating your data from one database to another. It supports a wide variety of sources and targets both relational and non relational. Just to give you that flexibility. If you want to take something from uh say uh I don't know a Postgres system to Dynamo DMS can help do that if you wanted to go from, uh AOL TP to a data warehouse system. Again, it can help here. Uh it really is what we like to call a bit of a swiss army knife. You go from source A to target B, whatever that may be

d ms can help you with that. um, people can also use it to migrate from on premises to ec2 from ec2 to r ds from r ds back to ec2 or, you know, for argument's sake from r ds back to on premises. uh it really does let me, you move your data from anywhere to anywhere else.

it uh it doesn't really discriminate if you will. um it's all about giving that flexibility and choice on where to migrate. um another strength of the migration service is that it does keep things operational while you are migrating. uh it is what we would actually call a logical replication product. what that means is we are reading the transaction logs of your source system to monitor for any changes during the migration. so it means that you can keep using your database. uh however, it is, you want to reading, writing doesn't matter and we'll replicate those sources or those uh that information from the source of the target. and it is in that, that vein that people actually have realized that d ms is useful for a lot more than just a database migration. you can use it almost as a quasi etl tool if you will. and people use it for other purposes, like even uh moving data between regions and aws.

so there's a lot of flexibility there. don't let the name fool you. it is a data movement product focusing on databases uh from point a to point b and probably one of its strongest um benefits shall you say is its cost. if you've gone out there and looked at the uh commercial market for migration and replication tools, they are not cheap. um for us d ms is something that is more of an enabler than anything else. we want you to use it to get you migrating to the cloud and have you leverage the products in the cloud the best way you possibly can. so we, we try to make it as economical as possible. we're really just covering the infrastructure cost for the most part so that, you know, you don't leave that replication server running for 27 weeks when you really only needed it for five minutes. that's the general idea. it's a very, very economical service.

uh letting you move your data around both into and out of the cloud, whatever works best for you. it's obviously, you know what? they didn't give me the new deck. this number is wrong. uh we have now migrated 800,000 databases using d ms. uh well, more than 800,000. so uh there's lots of customers that have uh leveraged the product successfully. uh it's uh as i said, very straightforward to use.

um and it's important also to highlight that last phrase using d ms. this is certainly not the number of databases migrated into uh aws. there are lots of other ways you can migrate. you can use your native technologies, right? so you could do a backup restore, you could use a third party product from any number of vendors to do their migration. this is just 800,000 plus databases migrated using d ms. i'm a little worried now that i see that because there could be one slide missing coming up. but uh we'll see, we'll see when nothing like thinking on your feet.

um demas supports a wide range of sources and targets like it's a big number. now, i admit this slide is slightly misleading because in some cases, we use the schema conversion tool for data movement. uh and what i mean by that is specifically in the data warehouse space. um data warehouses generally don't have a transactional replication stream. um what that means is you can't feed changes from say a uh i don't know, just pick 11 of those data warehouses to, to a target. it's more of a batch and load type thing. so with the data warehouses, we have uh sct uses what we call an extractor agent that lives close to that data warehouse, that'll dump all the data out. shove it to s3 and then get it ingested into red shift.

um so between d ms and sct those are the sources and targets we support. please don't ask a question at the end about a specific version of a specific database on there because i am not gonna remember which one we support, but i can tell you that we strive to support uh as many of them as possible. uh we support quite a number of older versions. you know, generally speaking, there are, there is a cohort of people migrating from unsupported platforms uh to something new. so of course, we, we try to support as many old versions as possible.

um what you will find is we may not support the absolute latest greatest version of database x that was released last week. because again, we're a migration service. if you have just upgraded to the latest version of a database, you're probably not looking at migrating off at the very next week. so, you know, we'll get there but maybe give us a few months. all right on to the next uh fleet advisor.

i talked a little bit about this. uh earlier d ms fleet advisor was a capability we launched to re invent last year. it's geared towards customers that are running large fleets of databases who need the full picture of everything they have. it's a free discovery tool. so there's different components of d ms. the fleet advisor bit is no charge at all to run it. it helps you understand your databases and your environment and provides functionality to discover and collect an inventory of running databases such as your oracle or sql server databases and others.

uh once it's collected all its information, you can look at it, you can organize it by database version operating system, database size, number of tables and so on and so forth and use this information to feed a business case where you can use to decide to decide whether or not to move forward with the migration.

um it's the level of analysis that it does. it's a lot more detailed than say a general server discovery. you know, it does more than tell you. oh, great. i found a server that's got a database on it. it will tell you the database, you've got the version that's on there whether or not it's still in support. for example, whether it's out of support, it'll tell you the number of schemas that are in that database. it'll tell you the number of objects in, in that schema. so how many tables, how many procedures, that sort of thing? and the other interesting capability that it has is it will tell you if it's identified in a similar scheme as elsewhere in your estate.

so this is a pretty common, you've got dev test prod instances, right? if you're looking at doing a migration, do you really want to do it three times you probably want to do it once, right. so, fluid advisor will let you narrow that down and go. well, it says i have 1000 servers but because of dev test pro i really only have 300 for argument's sake. right. so it's a, it's a high level broad brush environment to help or uh capability to help you find uh your databases scheme, a conversion tool.

now, this is the next level of analysis. its overall objective is to help you convert your commercial database and data warehouse schemas to open source engines or aws native services such as aurora or redshift. it works in conjunction with d ms. so remember, you know, d ms moves the data scte converts the schema.

um and generally speaking, you're gonna use it for that heterogeneous uh case it does three main things, it helps run or not, helps it runs an automated migration assessment report. so it connects the source and does a deep analysis of it and gives you a summary of how much it can convert automatically to another database engine.

so say you do an analysis of an oracle database. it's gonna say that i can convert, i don't know, 92% automatically to uh postgres or i can convert only 87% if i went to my sql. now, you might be wondering what's the difference between this and fleet advisor? fleet advisor is like that, that high level analysis, as i said, it'll tell you the number of databases, the types and the objects in there. but it's not gonna say uh we recommend you go to this because we've done this in depth uh evaluation of it. and, and this is sort of what we say. it's more like this is what we've got. and if there is something that you think is a candidate for refactoring to a modernized platform, then you would run sct on it.

the other reason for that is fleet advisor is meant to run in an ongoing basis and sort of monitor uh usage metrics and, and that sort of thing. um it also runs very quickly if you ran sct against 1000 databases in your uh estate, nothing to say you couldn't do it. but that could take a long time to run an sct analysis per database usually takes anywhere from for really small ones. it'll be done in like 30 seconds. but we have seen some very complicated schemas uh containing tens of thousands of procedures and tables and packages and whatever else that sct will actually run overnight for. right. so you don't necessarily want to do that in depth analysis on 1000 databases in your estate all at once.

the other thing it will do uh it will uh help convert it, right. so we do the assessment and then if this looks good to you and you want to give it a shot, it'll convert that schema from whatever it was to whatever new modern target you're looking at now. does it get 100% conversion only in demos? oh, wait, did i say that out loud now? iiii i it actually does a really good job.

um there are cases where you will get 100% conversion if you haven't done anything super complex or crazy in there. uh generally speaking, we've done a uh we've done a pretty thorough um job of emulating capability from a source to target. so say, for example, uh you know, postgres doesn't have a function that is in uh oracle. um with sct, we'll either convert it automatically or possibly we will use this thing. we call an extension pack, we'll emulate that function uh in postgres. so, you know, overall we get, we get the conversion actually pretty good.

uh and it gets better every month. we uh we release a new version every month of sct uh and uh really try to make it as close to automated as possible. but you know, for full transparency, it's not click button done, right? you, you got to spend a bit of time evaluating it. and the other thing it does is it will also help convert application code. so if you have a homegrown app that you've got a bunch of sql statements embedded in an app, it will go scan through that app code and try to convert, i don't know, for argument's sake. the t sql to pl sql in there.

now, if your sequel statements are highly dynamic in nature, uh you know, you've built them up through concatenations and whatever else. well, you know, it's probably not going to do the best job there. but honestly, i've seen a lot of code, a lot of people just, you know, have a sequel file with all the statements in there and it'll, it'll help convert them for you.

um and as i mentioned earlier in the data warehouse space, it will actually also migrate the data. uh so again, it's just a little bit different data warehousing tends to be moved with s cto ltp tends to be moved with d ms use cases.

um d ms can help you modernize, migrate and replicate for any or all of your data needs. uh modernizing, we've talked a lot about that that's converting from one database engine to another and update the uh application code migrating while you're looking to move data from a to b.

uh as i said, that can be between no sql to sql sql to sequel on premises. whatever the general idea is, you're moving data. uh examples of this uh people are migrating business critical applications, they are taking a data warehouse to red shift.

uh another interesting one is upgrading uh minor major versions of database. so say r ds r ds is great. uh if you want to go from version a to version b of a database, you go into the console, you flip the versions, you say apply during my next maintenance window and uh, it'll, it'll apply those changes and you'll, you'll end up press to change with the new database version.

there are some, some features they've either released or releasing soon to make that a bit simpler, but it doesn't, for some database engines, we've had some customers um, use d ms to make it a bit more seamless. what do i mean by that? if customers can't handle that outage during the mo maintenance window of having that database updated, what they'll do is they'll spin up the new version of that database engine, whatever it is. and then they've got their existing production version with the old version, so to speak, and they'll use d ms to replicate the data between the two r ds instances. because then when you want to upgrade, like when you wanna flip that switch and use the newest version, the only you need to take is however long it takes you to update the pointers for your application to point to the new database. so generally speaking, that's a dns update happens in seconds if not milliseconds depending on the scenario.

so people do use it for that other things. uh people will consolidate shards if you've got a big shard system and you know, you don't need to share it anymore with things like aurora. uh you can use it to consolidate that. of course, you could go the other way, right? so people have a big monolith. they want to break things into micro services d ms again because it's a logical replication product. you can take table a move it to aurora, you can take table b, move it to dynamo, table c move it to red shift, you know, whatever makes sense, it doesn't really care. you don't need to move the whole database. you can move parts of it including just the rows if you wanted to. so you can filter down on the rows as well.

other things people do is they'll migrate data either from or to s3. so if you want to take things out of a system and maybe archive awful data and put it in s3 because it's a lot cheaper to have it there than it is in an online database. uh you can use d ms to do that as well.

oh, new since last year. that's why we're all here, isn't it to find out what is new? um there's been a lot that's changed since last year. um this actually probably isn't a whole totally complete list, you know, of course, there's been lots of patch and security fixes and all that sort of fun stuff. well, but overall this at a high level is what we've done.

uh babish was announced to re invent last year, we added babish uh as a target shortly thereafter. so now if you, uh if you want to migrate to bish, you can do so with the ms

um db two mainframe. so we've had db two luw for some time. now, uh that's linux unix windows. if you hadn't heard that phrase before, we also added support for db two on z or z depending on your language of choice. which was the next logical step. there's more to come here as well.

we added uh support for sql server, read replicas. so now you can take the load much. i should actually go into detail on this one

A bit. DMS isn't magic, right? Remember I was saying it's a logical replication product, we have to read from those logs that does add some load onto the database server. So if your database server is absolutely pegged and then you try to throw another DMS task on top of it, you know, it might, might cause some challenges.

So for SQL Server customers, the ability to read from the read replica instead of the primary is it was a very useful feature that we added. Uh last year, we also added support for VPC Endpoints. Uh if you didn't know a VPC endpoint enables creation of a private connection between a VPC and supported AWS services. So you don't need public IPs and that sort of thing pre flight enhancements.

Um what that means is optionally, you can run some uh checks before your migration begins to ensure that it's going to be successful. Uh there's nothing worse than say you're migrating some giant database. Uh you get through the 1st 100 tables and on the 101st table it fails, it fails because of some unsupported data type or who knows what else. So the pre flight enhancements, uh we just put a lot more checks in there to ensure that your migration is going to be successful before you hit the go button.

We've got some enhanced logging. Um you know, going back to that initial statement about migrations are fun. One of the reasons uh they can be frustrating is you encounter some obscure error as you uh as you get through the process. Um, you know, there's, there's all sorts of things that can go wrong and i'm certainly not trying to scare you off. But, you know, there could be problems with the drivers, there could be problems with permissions, there could be problems with your network. Uh it could be that the database servers overloaded, et cetera, et cetera.

So we've tried to make that a lot more straightforward, uh with some enhanced logging to make it a little bit more obvious what the problem was if you encounter it and we're giving you the ability to offload those logs to S3 so that if you do need to share it with us or a partner or what have you, you can go through those logs and remove any PII information that maybe you don't want to share.

We've added a new operating system be beneath the scenes. Uh a L2, if you haven't heard that phrase, that stands for Amazon Linux 2, behind the scenes DMS is just running on an EC2 server like almost everything else out there. Uh we upgraded to the latest and greatest version of AL2 to ensure all the security fixes are there and we support Gen 6 instances. Now for replication, we also added support for IPv6 pretty much uh given these days, it just took a little bit of time to get there. So you can use IPv6 if you want.

And then in the SCT space, uh we can now do an assessment for Babel Fish. So you can uh do an analysis of your SQL Server system and say, hey, what kind of compatibility are you gonna get if you move to Babel Fish? Um in case anybody has looked at Babish before and are wondering what's the difference between SCT and the Bish tool called Compass? Not a whole lot.

Um Compass is maybe actually a little bit closer to whatever the current version of uh Babish is purely because it's developed by the Babel Fish team. We might be, you know, a few weeks, maybe a month behind on the uh assessment. But if you're used to using SCT, we just kind of feel it's good to keep everything in the same place.

We support schema conversion from Snowflake now, as well as Azure. Uh and we added support for Amazon Redshift, serverless uh Google BigQuery also got added in and then we've added support for uh actually ETL conversion from SQL Server Integration Services to Glue Studio. So again, I'm not the Glue guy here, but there's Glue and there's Glue Studio. We're starting to add Glue Studio as a target for a lot of the uh a lot of the conversions for ETL.

We've also got uh Informatica redirection. What does that mean uh for people that are going to or like to continue using Informatica may have a huge Informatica state been using it for lots of ETL in the past. Uh traditionally, maybe that ETL has directed things to a Teradata warehouse for example, or uh you know, in a teaser or Greenplum, you know, any other, any other data warehouse of choice.

Uh this now allows you to go in and automatically uh update and edit your Informatica scripts to point to Redshift instead of your, your legacy data warehouse uh improved code conversion. Uh we now are better at recognizing uh C# and Java code and uh updating the SQL statements in there in a way that we didn't before uh just to make that conversion simpler.

All right, product integration, there are a wide variety of tools one can use during migration. And you know, i didn't say this at the start. This may sound odd. I don't really mind what tools you use at AWS. We just want you to migrate if DMS SCT would have, you aren't the right tools for you use something else that's totally fine.

Uh you know, you may have a license for a big commercial replication product. You know, it's totally fine if you keep using that, but we're here to make it as simple as possible to do that migration, that wide variety can be a bit confusing. Sometimes the use case for each tool overlaps. Sometimes it's not clear if one tool might deliver a better result than another. And the user experience can also be confusing as you transition between different tools at different phases of a migration.

I mentioned the last year we started resolving this by launching DMS Fleet Advisor, which is integrated in to DMS, right? So you start your migration journey by finding your databases and then you start migrating them inside the same console. Now, there are other products out there um at AWS to discover things, right? Uh and we are working towards uh excuse me, providing a seamless experience with those as well.

Uh but really, we're giving you the choice if you're looking at doing a database modernization versus an all up uh estate migration. You know, we're trying to make this as seamless and straightforward as possible. Um yes, we talked a little bit about fleet divisor. As I said, we're trying to minimize the example of a disjointed workflow.

And as I said, at re invent, last year, we launched fleet divisor embedded in the DMS console. Now, this isn't quite what the DMS console looks like today. It's a screenshot of how it looked last year. That little toggle for DMS Studio is no longer there. But you, you get the general idea. Uh you can start with Fleet Avis at the top and then move down to Data Migration as you proceed along.

That's not it though. We're not done. Uh the big announcement that came out on Sunday was we just announced uh Schema Conversion as part of DMS. So we're calling that AWS DMS Scheme and Conversion. Uh some things are just better when you do them all together. So the the general idea is now you can use DMS as a central location to plan, you know, discover your databases, uh do an assessment, then migrate and modernize them. And uh AWS Schema Conversion or AWS DM. Schema conversion just makes it intuitive and simpler to go from one phase to the next.

What does it do? Well today, you know, in full disclaimer, this is, this is a v1, you can convert Oracle and SQL Server database schemas to MySQL and Postgres targets as with the stand alone SCT desktop tool. You start by establishing a connection to your database and then running the assessment report. Now again, we talked about the differences between what SCT does and what Fleet Advisor does. So this is that more detailed analysis you can do uh in there.

Now, what are the benefits of a service? Well, we talked about the integration flows from one section to the next pretty seamlessly. Uh the other thing is you don't need any extra security evaluations done by your internal IT department. So the Schema Conversion tool uh in case it wasn't obvious for the name, we haven't used it before. It's kind of an anomaly in Amazon land. It is a downloadable stand alone Java product. So you download this Java tool, you put it either on your database server near your database server, on your laptop, whatever. And then you run the assessment and you can do the conversion locally that made sense seven years ago or so when we launched it, um people were hesitant about the cloud, they weren't really sure what it was all about.

Uh now people are like, what, what's this thing? I got to talk to it. I don't have admin privileges on my laptop or they don't let us install apps without you know, a thorough analysis. So it makes a lot more sense for us to now put that Schema Conversion embedded into the web service in the cloud. Uh you don't need any extra admin permissions. And on top of that you have the power of AWSs behind there.

So you may have heard me mention earlier that the uh some complex schemas can take a long time to run. Well, now, instead of having that compute power local on your laptop, you've got the power of EC2 behind Schema Conversion doing that analysis. Um you know, probably one of the best things about it is that again, like Fleet Advisor, it's free. So you don't have to pay for anything with SCT uh or with Schema Conversion embedded inside DMS.

Um but, you know, full disclaimer, this is a v1 product. We don't have all the same capabilities as the Schema Conversion Tool today. So we spoil only that limited subset of databases and we don't do app conversion or in line code editing or ETL just yet. But, you know, as with time this, this stuff will come.

So i think it's probably time that we do a bit of a come on demo. So i'm gonna switch over. Let's hope this works just a second. All right. Hey, look at that. It's in technology grand now. I would have done a live demo. I personally love doing live demos, but, you know, best practices are saying, no, don't do the live demos in case you have wi fi problems or something like that. So you get me talking to a video that i recorded earlier.

So here's what Schema Conversion looks like inside uh DMS. Uh you've got this new menu there called Convert. That has these three options, Migration projects, Instance profiles and Data providers. Obviously, Migration projects are what we're talking about. It's a project that we're gonna use to migrate the Instance profiles. We'll, we'll dive into that in a minute. Uh these are just common uh security and settings they would use across all of the different projects that you might have and then Data providers. Well, this is just the information about the source of the target database. Where is it, what's the user name? What's the password, you know, that sort of stuff?

Um and what you'll see as time goes on as we start leveraging these, these sort of concepts a little bit more through the entire migration journey. So we'll go ahead and create a new instance, profile. All these things you do is you, you give it a name, right? Uh what kind of ip address you want to use, as i said, the IPv6 stuffs there. Uh it's very standard Amazon, which VPC do you want the Schema Conversion to be running in which subnets to use? Uh and you can add some extra settings.

Now in here, this is where we're gonna be saving the migration project. We save it in an S3 bucket. Uh you know, it's not like we're saving it to disc or anything like that and then the permissions or the IAM or rather that has permissions to access that S3 bucket. Uh other things we can do. Well, we'll actually, you know what, we'll create that instance profile just now we'll come back to some of the more details.

So we've now got two instance profiles. In this case, i would imagine most people are probably just going to have one. But then the next steps are to go and create a data provider. And again, remember that data provider is just uh the connection to the database. So we're going to create a new one. In this case, we're going to connect to a SQL Server system.

Uh you can give it a description in a n if you want uh standard things, server name, password, user name. You can see in this case, we're, we're obviously using a resource that is in AWS. Well, that's just because when we're building the demos, i don't have access to an on premises system, but you could put the ip address of an on premises system in there and it's gonna work just the same.

So we're gonna choose the RDS one in this case. Uh this is your MySQL target, right? So we're going from, from Sequel to MySQL. Uh right here SSL is generally a good thing to use. But again, for demo purposes, you know, we haven't set this up. Uh and uh yeah, then from there we go and we create a migration project.

So i just give it a name as an example. There's this instance profile again. So these are the VPCs that you're gonna use and the security permissions and that sort of thing. And now you say what we're migrating from this is the SQL Server system. Uh what secrets manager ID we want to use to connect to that SQL Server system. So you have to put the user name and password inside AWS Secrets Manager.

Uh and then also we're going from one to another. So what are we gonna migrate to? In this case, we're gonna migrate to MySQL. Uh you can choose a different secrets ID. If it, if it is different, i would assume the user name and password would be different between your source and your target. Um and then uh you know the permissions to use to access that secret manager.

And optionally in here, you can add a transformation. So uh this kind of transformation, if you, if you know some of the differences between say oracle and postgres, one's case sensitive one's not. So you can say, you know, move everything to lower case, for example. So that's what's happening here now that the migration project is created and we need to actually launch the Schema Conversion.

So right now, we've just kind of given Schema Conversion all the settings. We're going to launch an EC2 instance. Now that will do that analysis and conversion for you. Now, this is a demo. We've cut some time out. Uh it can take a few minutes to launch that replication instance and do the analysis. So, you know, just, just be patient once that's done.

Uh this brings up the IDE let's say looks very similar to a lot of the IDEs. We've got the source on the left, the target on the right. Uh and we're going to choose the database that we want to do an analysis on.

In this case, we'll just choose that one on the top. Go in. And the very first thing you're generally gonna do is run the assessment. So again, the assessment will take a minute or so to run, goes through and eventually again, sped up for demo purposes. Uh you'll get the assessment report result here that tells you how much of it can be converted easily between your source and your target.

Uh you can browse down through the tree and you can see each individual item if you want, you know, you don't have to see it at a consolidated level. You can see it down for your given table or procedure of choice. Uh and just give you an idea of how easy or hard it's going to be to convert a second here, different procedures.

Uh and you can drill into it a little bit more and see uh the original sequel on the left that uh that procedure was done um created with at the end if you want, you can export the assessment report either to cs v or pdf. Uh and you can use it to, I don't know, support a business case or, or you know, say, hey look, this is, this is how much we can convert automatically.

No, that's not supposed. Oh, there we go. Video is just a bit glitchy. Oh, there's, there's the assessment report, sorry, just uh white screen flickering. So this is the pdf uh pdf copy of that assessment report that we were viewing on screen earlier.

Alright, once that's done, the next step, of course is to convert the database. Now, you probably wouldn't do it quite this fast in real life. But you know, for this uh this demo, we're now going to go and convert. So what you'll see is all of the objects on the left are going to be converted to the right, which is now the mysql system.

Uh you can see we've got some case differences between source and targets. The the scheme has been renamed a little bit. It no longer says db o. It says, you know, large db dot db o and that was that transform, we saw uh added earlier. And as you click into each object, you can see this is what the original my sequel or sorry sequel server code look like on the left and the converted code is on the right in uh in my sql format and it doesn't happen for just the tables.

But you'll see in a second as we get down there, actually, it's just showing an example, not everything will necessarily convert automatically. So when that happens, you can click into it and, and see why, why it didn't work. Uh but then we'll, uh we'll go down and we'll look at some procedures i believe. Just give me a second. Yeah, there we go. So it's just showing, you know that that one converted automatically.

Alright. And at this point now we can apply the changes to the target. I guess we didn't show the uh the procedure right in this case. Um so right now what's happening is that generated sql which is just in memory at this point is now being applied to your target database up until this point, nothing has actually happened, but we've gone and applied it there.

So now what you have on your target, my sequel system is an empty database. Well, sorry. It's a database with an empty schema. There's no data in there at all at this point in time. So it will take a couple of seconds to apply. And you can also optionally take the sequel that was used to generate that schema and export it to s3 and view it if you want in a zip file.

So you don't have to let schema conversion, apply the schema to your target database. You can just take the generated results of it to a text file and apply it as you see, fit manually as well. So that, that's always an option. And at the end, the result, you can see we're in d beaver. If you haven't used d beaver before, it's just more of a, a generic sql query tool. You can connect to any number of different types of databases and data warehouses.

You can see the converted objects are now there in the database and you can see the size on the right hand side there, you can see they're all empty, right? Just 16 k of information defining the schema uh with no data in there whatsoever. And that i think is the end of that demo, just switch back for a second. Uh let's hope for the best. I think it's that one. Hey, look at that it works. I like it, right.

So overall uh d ms is out there to uh make it easy to migrate. Uh you can go from any source to any target. You can do the full load, which is the full set of data or you can do the changes. So often a migration will have both that full initial set of data plus replicating the changes that are occurring while the migration is happening. Alternatively, you can just replicate the changes. Excuse me. As I said, we have a breadth of options, lots of different sources and targets to choose from.

Cdc allows uninterrupted use of your systems during the migrations and overall it has the benefit of being a managed service, right? So you don't have to worry about scaling high availability, patching, uh monitoring security fixes all of that stuff aws handles for you. Overall, though one of the biggest benefits of d ms, let's try it again is automation, right? It is a managed service with a managed service comes the ability to use things like cloud formation templates to launch it.

It gives you the ability to have other integrated services like cloud watch events and more than anything else, you have access to an api and a cli to do whatever you want with this. I mean, we've, we've seen examples of customers that have, have migrated thousands of databases. You know, they might, they might be sort of an isv if you will, that has a one database per customer. If they want to migrate it all using the api, you can script that. So you migrate one database, one night, another database, the next or you know, 200 per night if you want.

So that that's one of the big benefits of being a managed service is you get the uh the full scripted ability to customize it. However, you want one such example of how you can do uh automation is with eventbridge. So eventbridge is actually a pretty cool product. It's a serverless event bus service that lets you connect your applications with data from a variety of sources.

Eventbridge was formerly called amazon cloudwatch events. If you used it previously, it delivers a stream of real time information from your applications. Sass products, what have you or aws services to targets such as lambda http in education. Um end points using api s. What have you in this case though, we're gonna use eventbridge to provide a notification of when aws d ms event happens.

Uh you might use it, say somebody created a new replication instance or the replication instance was shut down. But more likely what you're gonna do is use it to highlight, say if something happens, does a task fail? Does a task pause, does stop, does it resume? What have you, you know, you can aws d ms generates a lot of different events that you can use to make it easier to automate.

So i need dj to come up and show an example of how this works in a demo, which actually ironically has started automatically. So let me just bring that back to back to start. Um so yeah, let's uh let's get going with a, with a demo vjl over to you.

Yeah, thank you. Um i want to go back to the presentation ones. Uh no, let's do that one. Oh that's where i am. So i think i need to go to one and then take, so i just have an architectural diagram that i want to cover. Oops yeah, restart it. Yeah, sorry guys.

Uh what i'm going to show as part of the architectural diagram as john showed right uh eventbridge is you can integrate eventbridge with d ms. So in right now, we don't have an architectural diagram. But what i put together for us is i have ad ms instance um task in this case. So which is like my sql to s3. I'm migrating data from my sql to s3. My sql is my source and s3 is the target.

So while i'm doing this migration, uh this migrating the task there is a failure happens, right? So i wanna use eventbridge to capture this event and then automatically try to fix this task. So for that, what i did is as some of the prerequisites that my sql requires for this migration is to enable binary logs for the mysql. And is anyone everyone knows about mysql or you want me to give quick over view?

So if you want to do the migration uh from my sql to any target, we need to enable binary logs. So that is the prerequisite was missing in our case. So what i'm going to do is i will capture the event, the failure event and then take that into a eventbridge. And then i created a rule in the eventbridge which i'm gonna show you using that rule, i will match it with a target which will call a lambda function within that lambda function.

I have created a code, you can use your own custom codes in lambda function to fix any of those kind of failures. So with that, let's get started and see what really happens. So here i have a my sequel to s3 uh task and which is failed. So it is a full load, an ongoing task. And then as you see, the failure is uh this is the event that you can see in the cloud watch logs.

So i just want to show you here because for the my sequel to do the migration, we need a row format, right? So what i'm going to do is now i will show you my source, which is my sql here. I'm gonna connect to my sql and then show you what is the row format of the boundary logs. So i'm connecting to the source engine and then i'm going to log into my sql just to show you what's really happening here.

Now, i'm going to query the variable binary log format and which shows mixed. So for us to do the ongoing replication from my sql to s3 or any target, this needs to be a row format. So that was the problem. That is the reason we are getting the failure.

So what i'm gonna do is now this is the event that got generated uh of that failure that you can see in the cloud watch logs. Uh so this is the event as part of this event, what we have is the source uh here which is aws d ms, right? So i'm gonna use that as part of uh my eventbridge and then even type replication task failed. I'm gonna use both of them and then going to fix this automatically for us.

So you can do that for any of the replication tasks that you, that you have. So this is just one example that i can show it quickly. So now that you have seen that, what i'm gonna do is i'm gonna go to even bridge and then i'm gonna show the rule here. So in this rule, what we have is the source and then even type that i used it from the, i used it from the uh the cloud watch logs that we got, right?

So, so the replication task failed. So i checked the rule and then i will forward that event to the target. So here you have multiple targets, right? You can send an s ms notification, you can send in lambda function depends on whatever you wanna do. It depends on the task, right? You know, in this case, uh i'm fixing the my sql prerequisite. But if you wanna get notified when there is a task fails, you can send it to.

So, in this uh lambda function, what i have is uh if you see uh the cdc para meters prerequisites, right? Uh so right now i'm taking the details and then detailed message uh from the task and then i'm gonna check the prerequisite. So, because my prerequisite was failing, so i have already configured within my lambda function that a re required for the prerequisites for my sql as a source.

Uh and then what i'm going to do is i'm gonna fix that to row. So it's already done in the uh lambda function. And then i'm gonna set that as a role, right? That's what this lambda function does. And then i'm gonna resume the task. It's everything is going to be automated using this lambda function.

So right now what's happening is uh i'm going to enable this rule. Once i enable this rule, i'm gonna start the task again. So right now the rule is enabled, the rule is pointed to a specific target, which is my lambda function. So, what i'm gonna do is i'm gonna start the task again. So whenever i start the task, right, it is gonna fail once it fails my uh rule kicks in and then i'm gonna go to the lambda function and then it will automatically start the task again.

So what happens right now, you see that uh it is running, i'm gonna refresh it uh and it is failed. Once it is failed, what will happen is um now i'm going to go to or the lambda function within the lambda function. What i have is i'm fixing the uh my uh binary log formats and then i'm gonna start the task. So it should automatically fix the task.

So if you see here while that is happening, so my lambda function must have kicked in and then it changed my row format, right, boundary log format to row. So now my task have all the presets that is required and then tasks should run automatically. So if you see uh uh it is raw right now and then if i refresh it, it is starting and then load completed and replication is ongoing.

So this is how you can use the event bridge with d ms, right? It is a pretty good cool uh feature that we have. So if you want to use this right, you have to be at least minimum 3.4 0.6 or higher to use the uh to use the even bridge.

So now going back to the presentation, i think what do we have is the resources that we have, right? Yeah. So that's uh that's pretty much it, apologies for lacking the uh the architecture diagram and probably would have made a little bit clear what was going on.

Um but when you do, if you want to download the slides at the end of this presentation, when they get published on the website, i guarantee you there are gonna be two changes in there. There's gonna be an architecture diagram and it's gonna say 800,000 databases migrated. I i don't know why that didn't make it through to this uh this laptop, but these things happen.

So hopefully, that was interesting for everybody today. Here's some resources uh in case you want to get started with d ms. Um we're pretty much out of time. Uh we'll be happy to take some questions at, at the sidebar afterwards if you want. Um but overall, uh hopefully that was interesting. Hopefully your week here in las vegas has been great and everybody enjoys replay tonight. So thanks again, everyone.