Scaling a multiplayer game into the millions with Mortal Kombat 1

hi, everyone. welcome to this session.

so today you will learn how warner bros games is building and operating video game platform able to serve millions of users. but before we start, i would just want to check something.

please raise your hand if you are playing video games. ok. so let's do it the other way who don't play video games. ok. so we have maybe one. ok. so i will not blame you for sure. and i'm sure you will play video game after this session.

so uh it's very important to know that we are billions to play video games. according to a recent uh study, there is 3.2 billion people worldwide that play video games in 2023. it's more than 40% of the world population and uh over 1.3 billion people will play online games by 2025.

so with this number crazy number, it's really important to think about how you will build a platform, thinking about scale, about security and of course how you keep control on cost.

so i'm super happy and super excited to be honest, uh to dive a little bit deeper and to know what's going on behind the scene uh with building such a platform that serves so many people at the same time.

and uh um and yeah, so this, this is what we will talk about today.

um my name is lionel gin. i'm a social architect here at ws. uh and i will introduce uh steven and elliot from one of ross games. uh steven elliot. it's over to you.

all right, let's see. where's our clicker? there we go. all right. thank you for that hand clap. that's gonna help loosen us up up here a little bit.

um so look, this is fantastic. we couldn't be more honored uh to be invited to come speak and re invent to all of you. uh we know that there's so many talks that you all could be attending right now. so the fact that you took time out to go from one hotel to the next and convention center to come listen to this talk, it really means a lot. we appreciate it and we're excited because we get to talk about the things that we love the most. we love video games. we love solving really hard, difficult technical problems and we love talking about the cool things that our team is working on.

so what we're gonna do, you know, we're gonna talk a lot about um like lionel said, how we scale mortal kombat one, how we get that ready for lots of you and millions of players around the world. but, you know, we're, we're big on talking about sort of the business side of that. right. so we're not just uh um engineers that are just building and trying to solve, you know, engineering problems. we are trying to solve business problems.

so we are going to take us all on a little bit of a journey through sort of the business that we're supporting, how we sort of landed to where we got to today. and then we're gonna do a deep dive into the platform and all the tools and technology that we've been building for mortal combat and all the hard lessons that we've learned over time.

um so let's start off by introducing myself. i'm stephen flannery. i'm the vice president of platform technology for wb games. i've been with the company for about 16 years.

um and i'm responsible for overseeing all of the back end online software development that happens within our company, all of our major technical partnerships and i'm also responsible for our global central it teams.

um and then elliott throw it over to you.

hey, everyone. my name is elliot haass. also super excited to be here at re invent. i'm the executive producer of wb games, new york.

um at wb games, we typically launch on average about 4 to 5 games each year. my job is to pretty much oversee the, the cross functional operations to launch those games. so we have many different teams. we have development teams, we have online teams, we have central tech teams. they all have to work very closely together to get these games out on time and on budget. it's pretty much my job to make sure that all those teams are communicating daily and that we're launching our games on time and with as little server issues as possible on day one, we're gonna talk a lot about that today.

all right. awesome. and then thank you, lionel for introducing us up here.

all right. so let's get into it.

so we're gonna talk about mortal kombat one. you know, we are going to dive a little bit into this product itself. i'm sure a lot of you already kind of conceptually know what mortal kombat is, but it's, it's one of our most incredible franchises and we have this really hard problem of how do we deliver a franchise that's been around for over 30 years. last year, we celebrated our 30 year anniversary that has this resonate with fans. i mean, i've been playing mortal kombat since i was a kid. i don't know if i should say that i've been playing mortal kombat since i was a kid many, many years. so we have all of these fans that's excited about this product and we have to make sure that when launch day happens, that our fans can play that game, access, it, have fun and that their expectations are blown out of the water. so lots of really fun problems to solve there before we get to that.

um, you know, one of the things that we've talked about, um, a lot and every time we do these presentations, um, the number of people that actually don't know that we games or wb makes games or is in the gaming business is actually pretty high. it's actually a surprising number.

so what i want to do is sort of walk through a little bit of the structure of the business. so we can start to set the groundwork for what it is that we do and what our role is there.

so warner brothers discovery, so warner brothers, discovery is a uh media company. it is a merger between uh warner brothers and discovery that happened about uh nearly uh two years ago at this point.

um and warner brothers, discovery is focused on doing one thing, right? we are storytellers. it, it is our job to deliver and create content for all of our fans and um across the world and our fans exist um through lots of different uh countries, through lots of different age ranges. it's, it's a wider range of audiences that we have to serve.

so we're storytellers, we are content creators, first and foremost, everything that we do all the way down to central tech. it's all about how do we create the best content for our players and our fans and with warner brothers discovery, there's lots of different ip. right.

we have just a few of them up here, but mortal kombat is some of the intellectual property that we um we own. uh the dc world and universe. um and then lots of other things like cartoon network and food network and discovery and tnt and animal planet.

so all of these different sort of, you know, uh intellectual properties and these different universes are all under one roof and it's something that we all have access to. so we get the opportunity to create stories and, and new universes and engage our players in all of these different spaces and we focus on a few of those.

so within our company, we do lots of different things, right? so we have all of this uh intellectual properties, things that we're very proud of, but we deliver them lots of different ways. we deliver them through movies and film. uh we deliver through television. um we have a sports network with tnt where you go through and you watch the latest updates on sports. um we also deliver, you know, content, uh news content through cnn and other platforms. so these are just some of the existing mediums that we use to deliver content.

but the one we're gonna focus on is the games division. so within warner brothers discovery, there's a game division, it's called uh warner brothers games. and that's what elliott and i report into

so wb games, um we've been around for a while now. we've been around for about 20 years. right. so, uh relatively speaking, compared to other publishers at our size, we're actually a relatively young game publisher. but, you know, we have again 20 years of, of, of history behind us, of sort of starting our journey building package, good and publishing, package good software to developing the core content that you experience today.

and it's our job, you know, the while the other divisions we just talked about there, it's their job to um deliver um sort of these, you know, tell these stories like one directionally, like they, they are putting it out in film in ways that you can consume it. it's our job to actually focus on the interactive experience around that content.

so while the rest of the company is, is delivering content to you, it is our job to put players into these universes and experiences in ways that they just haven't seen before.

and we've been so fortunate enough to be a part of this company that has produced five franchises, each of them crossing over a billion dollars. so again, really great company has been around for a really a really long time as a whole and we, we had the opportunity to do some really amazing things.

so with wb games, we have this um very unique benefit and also unique problem. we have lots of different studios. so we're comprised of 11 different game studios and each of these studios are distributed all across uh the world. we have a, you know, definitely a concentration in the us and the uk and montreal. but again, 11 different studios and each of these studios, they actually bring something unique and different um to uh all the work that we do.

so, um while we don't have all these studios working on one or two games, many of these studios are actually developing their own titles and they're focusing in the areas and spaces that they're really, really good at.

um as an example earlier this year, um avalanche software, they launched a small title called hogwarts legacy, right? it was one of the biggest games this year and this game was focused on creating this interactive uh rpg and the wizarding real universe, something that our fans have been asking for and craving for years and avalanche software has done such an amazing job of really delivering that and getting that, right.

so that's one type of game that we're working on. we have our team monolith, they're based out of seattle. um they've been working on the shadow of war, shadow of mortor franchises for a very long time now. um and they are currently working on a wonder woman game. so they're focused on sort of these, you know, action combat, you know, third person games.

uh we have tt games, right? whole other audience, their focus is creating content um for, for everybody, right? so uh uh many of the lego games that you've seen out there today, they've been produced and developed by tt games uh including the most recent release, uh lego star wars, the skywalker saga. it was a great reimagining of that lego universe.

so, again, different type of game that we have to think about when we're in central tech.

uh we have rocksteady, right? so rock city, uh they're based out of london. they have developed one of the most iconic superhero franchises in gaming, ever

An amazing trilogy with um with the uh Batman Arkham series, they're currently working on an extension of that universe called Suicide Squad. A game that launches next week. I'm sorry, excuse me next year. Excuse me. Um yeah, completely. Yeah, please. No. Uh whole other title right now. We have WB Games, Boston. So this is a different game than what we just looked at, right? This is a mobile game. It's a free to play game very successful. It's one of those billion dollar franchises where you go in is built in the Game of Thrones uh universe. You go in, it's a strategy game, you build your base, you build your army, you participate in these real time battles on your phone.

And last but certainly not least is Mortal Kombat. Um one of the number one fighting games and fighting franchises of all time. Um Nether Realm just released uh Mortal Kombat one or reining of this title um two months ago. And this is a completely different type of game. So you can see as Central Tech, right, what we have to do is we have to focus on how do we solve each of these unique problems for each of these studios.

So we talked about those 11 studios, uh 10 of those studios are focused on creating content. One of those studios is focused on delivering central technologies that supports all of the studios and that's us, that's W Games, New York. Um that's the studio that and I have been running together for a very long time.

Um so W Games uh founded the studio back in 2016. Uh but the team, we have been working together for well over 16 years at this point and we've had the, the, the, the benefit and the wonderful opportunity of launching over 50 AAA titles as a team, right? So we've been building backend services for a really long time. We've, boy, we made a lot of mistakes. We learned a lot of lessons. We had a lot of tough launches, but all of that has invested into where our studio stands today. So we've been together for 16 years, launched over 50 AAA titles. And uh we have the honor of leading central tech for, for WB Games.

So diving its central tech, right? So we have one simple mission is make great games better. So we think about, uh all of our studios, right? We talked about earlier, all the IP that we have access to, we talked about, um uh the fact that we have all of these amazingly talented studios, some of which have been making games for decades, right. So you can put those things together and they can deliver an amazing title. So our job is to see how do we support all of those studios, uh get really in deep with those teams and help them take their games to the next level. That is our number one job.

And you know, we run into a few challenges today. So first off, uh gaming has become an increasingly connected experience. So whether it's a single player title or a multiplayer title or free to play whatever it is, uh more and more things are happening online to help engage fa uh fans and players as a great example for Hogwarts Legacy, um you were able, that's a single player focus game, but you were able to connect Hogwart's Legacy with another property of ours called the Harry Potter Fan Club. So that we have all of these fans, we brought them together so that we can reward them with new content in the game. We did lots of really amazing twitch campaigns uh to really drive engagement around the viewership for that. A lot of technology behind supporting that. And we were, you know, we were successful in driving that game to be one of the num the number one stream, single player game on Twitch. Ever, right? So single player game, but very, very connected in the way that we engage our players and build our communities.

Um uh we have to delight our fans, right? So our fans constantly are asking like when are we delivering new content? Uh we release this game, when are we getting this new character? So we have to find, how do we, how do we get this content to our players faster, the content that they love that they are asking us for?

Um you know, we're competing for players attention, right? If you walk around this conference, you see all of the different opportunities uh for players to engage uh whether it's um you know, watching television shows or movies or interacting with other games. So we are dealing with a market now that's very different than it was 20 years ago. We have to create compelling experiences and give our teams the tools that they need to do that.

Um and then we need to reach as many fans as possible. Like Lionel said, we have 3 billion gamers out there in the world. So what do we got to do to help our teams reach all of those different audiences?

So when we get down to we central uh W Games central technology, we try to solve this in a few different ways. Uh first off back in game services. So we have a um uh backend game service platform that we have been building for well over a decade. We're gonna do a deep dive into that, but that is really the core of our business is how do we empower our game teams to add and build online features and services in their game?

Uh we also focus on publisher technology. When we think of publisher technology, we create things like tools for them to update and operate the game live, you know, uh consistently tools and uh uh central identity system so that our players can carry their progress from one place to the next so that we can connect our players and reward them for being loyal fans.

Um cross platform support as we talked about with all of those studios, right? We have mobile games, we have console games, we have switch, we have pc games, we have a whole range of audiences. So we need to be able to provide tools and support um so that we can support our teams on no, no matter what platform they're on and then you, we're responsible for sort of building out and creating these amazing technology partnerships, right? We don't want to have to go and re invent the wheel every time, just like we want our game teams to be focused on innovation. We also want to be focused on innovation. So it's our job to go work with different tech partners uh like AWS to help fill in some of these gaps for us.

And we're successful when we tick these four boxes. First off, when our game teams are actually spending the bulk of their time innovating and not reinventing the wheel, right? Like I talked about, we have a backend services platform. You can imagine how much time that would take. If each game studio would they build, run operate, update, secure scale, deploy their own backend service. And that's time they're spending basically reinventing the wheel versus innovating and adding new features that our players are gonna love.

Um we're successful when our players, when we can, they can all play together and do that at really high scale. So when we start these games, when we turn them on, there's millions of players knocking on the door trying to get in. So we need to make sure that when you uh uh at 1201, um when they get millions of people are at the door waiting on that game to turn on that they can actually access the game and have fun.

Um when we release titles that add features and, and content that exceeds our players' expectations. And then ideally, if we're investing properly in central tech, we are making it easier and cheaper over time to deliver real great games that are really high quality. So that's a little bit of background on sort of our company that's a little bit of background on central tech and where we like to focus.

So I'm gonna throw it over to Elliott so he can talk about Mortal Kombat and then we'll get into some deep dives after that.

Thanks so much, Steven. Yeah. So super excited to be here. Obviously. Uh we talked about how I mentioned earlier. We launch about 4 to 5 games a year. Really good timing for reinventing. We just launched one of our biggest franchises, the newest edition of that and that's Mortal Kombat 11. So that's really what I'm excited to talk to you about.

So I'm gonna just take a little run through the game and talk to you about the features that we have in it. And these are the types of features that we support from an online backend perspective. So, but let's talk about just a little bit about the franchise first.

So Steven mentioned Mortal Kombat has been around for 30 years. Now, the first Mortal Kombat arcade cabinet launched way back in 1992. Um we launched the first home council versions uh for Mortal Kombat on the Super NE Genesis Game Gear and Game Boy on September 13th, 1993. So as Steven mentioned Nether Realm Studios, the amazingly talented team out of Chicago Illinois that, that builds Mortal Combat. Just celebrated their 30th anniversary of Mortal Kombat, which is pretty incredible.

Over 25 Mortal Kombat games have been released, that's a lot and over 80 million units sold which Nether Realm and WB is super, super happy about. And as Steven mentioned, we had that slide where we showed the sort of pillars of our, of our company at, at WB uh Discovery Mortal Kombat is a really good example of this and actively right now, we, we sort of refer to it as a trans media franchise, a franchise that spans games TV, movies and comics just as an example, right?

We just released um an animated film centered around Johnny Cage that's available and that you can get and you could eventually be on streaming. We're, we're currently working on a file up to 2020 one's Mortal Combat feature length film. We just released um a mobile title for Android and iOS called Mortal Combat Onslaught right around the launch of Mortal Combat 11 and obviously the big, the big deal we just launched back in September, Mortal Combat 11.

So um we wanted to give you guys a little taste of Mortal Combat, right to see it live. So we have our launch trailer that we're gonna play for you...

[Mortal Kombat 11 Launch Trailer]

Yeah, I've seen that trail about 100 times. I get pumped the same way every single time. So yeah, that, that's, that's Nether Realm Studios in Chicago, Illinois. They've been doing this for a long time. So that trailer is just super impressive.

So a little bit more about Mortal Kombat 11. This is the latest entry in the Mortal Kombat franchise was released on September 14th or just a couple of months ago. And as you saw on the trailer, this is a reborn...

Mortal Kombat universe created by Fire God Liu Kang. It's 23 playable characters, right? So featuring reimagined versions of iconic characters in ways that you've never seen before.

Spoiler alert. If you didn't play Mortal Kombat 11 at the end of that game, Liu Kang gets reborn as a fire god and he uses Chronica's hourglass to create a completely new universe and a new timeline for Mortal Kombat. So this is the setting for Mortal Kombat One, offering new backgrounds, new alliances and just new ways to experience the characters that you've loved for over 30 years.

One of the big new features that we also released is the introduction of Cameo Fighters. So this is a unique roster of partner fighters who assist you while you're fighting with your mainline character, creating expanded gameplay possibilities for all of your fighters and across all the game modes. Cameos dramatically enhance every fight.

There is their team, it's essentially who assist you with special moves, throws and defensive breakers of their own. So now I'm just going to sort of walk through some of the key features that we have.

The first of which is our Campaign mode, right? I mean, Mortal Kombat and NetherRealm Studios is known for their genre defining Story Mode for the Mortal Kombat games. No different here with Mortal Kombat One. And again, the tagline “It's In Our Blood”, it's in our blood to have a really feature-rich multi hour Story Mode in every single Mortal Kombat One game. Ed Boon and team did that again.

Like I mentioned, right, everything is new, the characters are different, the timeline is different. And also that means the villains have new alliances, new backgrounds and new ways to sort of quickly hop in and show you the Mortal Kombat things you're familiar with, but a little new taste and differences and our Cameo characters functionality really slams that home as well.

Multiplayer features, right? This is where we start talking a lot about the features that we help support from a central technologies perspective. The head to head combat that you're very familiar with has to be there and it's there, right? So we have local versus play, head to head with your Cameo fighters on the same console. Of course, that's there. We also have a local tournament mode where you can set up sort of match rules in your own way and run a tournament. You can do that locally.

Online multiplayer. We have our casual versus right. If you just want to get online matchmake with someone who's relatively close to your skill, maybe same ping time you can do that. But our big online modes are ranked matches and we have our Combat League, right? So that is where you compete. Our matchmaking services really need to be fine tuned at that point because we want to find people who are extremely close in your skill level, who are extremely close to your ping and latency. And we define the right matches, right? Because every match that you fight with ranked goes to your rank points, you have nine different rank levels that you can sort of work through. Each one of those ranks in the Combat League brings about new rewards, new skins and new palettes that you can earn in our online mode and we constantly rotate that. We have a seasonal six week schedule for all of our Combat Leagues. And each with new Combat League, we bring new rewards, new skins and new characters.

Invasions. That's uh one of you saw it in the trailer, that's sort of our new extension of the single player mode, right? So fight against invading forces that threaten your timeline. So after the game is over, this is an extension of that, right? Invasions is a dynamic single player campaign where where you can traverse through the realm, sort of in like a board game style, you can with different characters.

There's RPG mechanics, right? So you can level up each individual character, you can level up your Cameo character, you can on the fly, swap between characters, you can enable talismans and relics along the way. And all of that plays into the types of villains and unique matches that you're thrown as as you compete through the board.

And again, Invasions, we have that on a seasonal rotational schedule. We're super excited about that. So every six weeks, we're gonna give you a brand new set of boards that you can work through a brand new set of awards skins and pallets that you can earn.

Gear, right? You've heard me talk about it, but gear is super important for Mortal Kombat, right? So, as you play through all of our game modes and even single player modes, you can earn new gear, you can custom... This is essentially how you can customize your characters. Both mainline and Cameo characters have the ability to upgrade their gear. This is your way to personalize a cosmetic for you to personalize the characters that you love. So if you want to have sort of a unique style of your own online, you have the ability to do that via all the different types of gear that we have.

And again, I'll keep saying that, but each new season, we're bringing out new gear, new cosmetics that you haven't seen before, and we're gonna support this game for multiyear. So we have a lot of new content coming for you. We also have an in game store that has rotating items that you can purchase with in game currency. And again, gear skin and pallets are available there as well.

So that's a little bit of run through the game. What are some of the challenges? Right? This is a good segue to throw back over to Steven, to talk about our technologies and how we support. But what are some of the challenges that we face launching Mortal Kombat One from an online perspective?

Millions of players are waiting to play. No surprise, right? The games have been out there for 30 years. We have a dedicated and built in audience who is literally always there. Day one, if there's an early access version of the game, they're gonna be there waiting to knock the doors down.

We have a heck of a marketing team as you saw with that video, we really get people interested and pumped about the game. We have a whole campaign that we run multi weeks before it launches to get you excited about the different characters that always has people at a fever pitch to play. And again for this Mortal Kombat One, we launch day and day for all five major platforms. So Xbox, PlayStation, Switch, Steam and Epic. We launch on all the platforms at the same time, we had pre downloads for all of those as well too. So it's not just people play at midnight, it takes, you know, 20 minutes to download a build, they can pre download. So literally, it goes from 0 to 100 the second that midnight hits on launch date. So always a challenge for us.

High expectations around online performance, Mortal Combat and the style of game requires high performance and low latency period. Literally, if we don't have that in our online modes, the game doesn't fly, the community can't get behind it. So that is extremely, extremely critical for us.

This required us to really get ahead of our game server set up and configuration. This is where we leverage AWS, GameLift and AWS GameLift fleets a great deal. They serve as the backbone for all of our online game server needs for Mortal Kombat One. We put them to the test.

And essentially the other thing is matchmaking. We need to be well ahead of matchmaking and we need to be able to adapt on the fly, right. We, we have, we could do a couple of betas before the game launches where we can get some of that data to determine what our matchmaking configuration should be at launch. But day one, there's always complaints, there's always sort of changes that we need to make and our technology helps the developers in NetherRealm change that quickly on the fly without doing patches with quick button presses.

So Steven will talk a little bit more about that soon and the game is a platform for delivering regularly new content. I mentioned it, but NetherRealm plans to support this game for multi years. So the days of having box content and that's all you have are long, long gone. We need to support this game for multi years. NetherRealm is constantly pumping out new characters and new content and that is a challenge, especially from an online perspective. We have to help them with that with all the features and needs that we support and Steven, I think that's a segue back to you, my man.

Right. So let's talk about what do we have to build to support Mortal Kombat One? I think it's important to call out that we have been working on the Mortal Kombat franchise since about Mortal Kombat Nine, which came out in, there, came out, came out in 2009. So, you know, our first version of dealing with this problem, man, do we have we had a lot of issues, right? We were, we were really learning what it means to support a game of this caliber. We were really learning what it means to build an architecture that we can stand up.

And then support a title that again at the time, still a lot of the same features. How do we get regular content, regular updates is sort of like online live and social game experiences, not just, you know, put the game in and you're just off and running and fighting. So we've been building on our platform for over 10 years and Mortal Kombat One, honestly, it's been the culmination of a long journey that started with them many, many years ago.

So we have a game services platform that we have been running, operating and building on, like I said for over a decade called Hydra Game Services. This has been built by us here at WB Games, New York. And this is our platform that we use to be allow our game teams to build and add new online features to their game to launch and deploy them to millions of players around the world. And then we give them all the tools that they need to sort of grow and engage those audience for many, many years.

So we made a couple of technology decisions right off of the start. One of those was Software as a Service. So when we first built our product, we had a very customized business. What we would do is we would go build a specific system where you could start with the same code base, but we would go and deploy that and branch that off for each individual title and then design it and customize it to exactly what that game needs. And that worked for us for a really long time, right? It was a great, great process for us. You know, we're able to run a very successful business doing that.

What we started to find was, you know, every time we are going to go branch our code. Well, now we have to keep all of these different branches of code updated. If there's a security change or a bug that comes out that we need to deploy, we need to deploy to all of those environments. If we want to take one of the features from one of our one branches and move it over to the next one. Well, how much did we customize that branch? Can we actually move that over? How difficult is that going to be? Well, now we got to decide if we bring it over, if we're going to go rewrite this feature.

So we got into this model of, we said this is working now, but this is not going to work for us 10 years from now. So we decided to lock down and say we are going to focus on building a product that's going to be completely Software as a Service, which means that our game teams can go to a dashboard. They log into this dashboard, they have access to all of their logs. They can update and change all of their game configuration on the server in real time.

They can download all of their SDKs. We have a C++ and the C# SDK. We have great support and plugins for both Unreal Game Engine and the Unity Game Engine. We also have a Java SDK. We have a Hydra command line interface for our developers that want to work in the CLI and lots and lots of other tools that our teams have access to and then everything is fully documented from the ground up from start to finish.

So this was a, this is a hard decision for us because as we all know, if you technologists, this is harder, it's harder to do this than just to focus on building a, a very specific, very custom solution for each of our game teams. So we had to say, you know, each feature, it's gonna take us a little longer because we need to take our game designs. And then we have to take those, we have to extract out of those. How do we build a generic feature that everybody's gonna use that we can support for years to come but still make sure that this game has exactly the features that it needs.

So we kept that even though we're internal to WB everything we do is completely Software as a Service. And then we said, all right, now that we have this sort of Software as a Service model, we need to start putting more power in our game team hands. So how this used to work is our game teams will come to us and they would say, write me a very specific service and we will go and our team would go write that very specific service for them and we would deliver it to them.

Instead, what we said is, why don't we start creating primitives or let's call them more like Lego bricks of services so that our teams, they can log again into our dashboard, they can look at all of the services that we have available and they can pick and choose the ones that they need. And this model of focusing on building these sort of baseline primitives that are all built to be plugged together.

This right here is just really a snapshot of the features that we have. We have dozens and dozens of features in our platform, but this snapshot is player data. So if we want to track thousands of stats for each of our players, I think in Mortal Kombat, there's two or 3000 stats that we're tracking per player. We have a matchmaking system. We have leaderboards for ranking. We have clans and friends and messaging for creating sort of the social features.

We have segmentation and event scheduling so that we can sort of live, operate our game and get this content out on a regular basis. We have tools for fine grain permission control so that our game teams can go and decide who which developer or QA person should have access to what and then we have everything available in a full monitoring dashboard so our game teams can go and they can view exactly what's happening in their game environment in real time at any point in time.

So this is just a snapshot of the features that we have. We have dozens of these that we've been building for for many, many years. So what we've ended up with is a system that has dozens of primitives, thousands of endpoints and APIs that both our game teams use to sort of stick together or customize or that we do for them.

Um we built the system because of this investment so that it's flexible enough to support all of the studios and the game titles that we talked about there before. Self service has been built into the platform's DNA. So everything we do, we always say, how do we make sure we're going to support all these studios, all 10 of these studios within Warner Brothers and some of our external partners, we need to make sure that we are always putting the power back into the developer's hands and our system is supporting dozens of live titles and titles and development.

We release code every two weeks. So, um again, there feels very much as a software as a service product and it runs entirely on AWS. So when we first started thinking about our business and what we needed to support, um you know, we had some decisions to make of like, you know, what platform are we going to go with? There's a couple of pillars that were important for us, for AWS that made us go here.

First off the scale, right? We need to make sure that we can deploy servers all across the world quickly to support these millions of players. We need to make sure that security was at the forefront. It's something that we think about in our API design and the way that we deliver and develop our products. So we want to make sure that we were partnered with somebody who also provides us with the tools to improve and increase our security. But that we're also focused on their own security.

Uh we needed that flexibility. So for us, you, you heard a lot, we, we build a lot of our own tools, right? So we don't necessarily use a lot of all the management or the managed services. So we need to work with a partner that's gonna give us the flexibility to work and develop our way and that we needed to go with a partner that was gonna help us monitor and manage cost and make sure that was as efficient as possible.

So our transformation journey, we actually started off cloud agnostic, which means that because we were working with lots of different publishers in the game industry, we needed to have a system where we actually weren't using any of the managed services from the provider. We might need to deploy something at AWS for a client. We might need to deploy at a completely different provider depending on who that client was working with.

Um and maybe even at the time, a lot of our providers even had their own data warehouses. So we might need to actually deploy in their space. So everything we built, when we started our platform, we were using no managed services. We built our own firewalls, we built our own load balancers, everything from scratch to that, we can put our system wherever it needed to be.

So, with AWS, what's been great is since we joined Warner Brothers in 2016, we've been able to take that platform and we moved it wholesale into AWS. And we've been able to over time, you know, pick and pull and use the features that we thought were going to help fill some of the gaps for us.

So over time, here's what we use today. We use a lot of EC2. Um and we use a lot of EBS volumes attached to EC2. Most of our infrastructure actually runs on EC2, but then you see all these different edges of, of features that we need.

So we want a really great DDoS and security and protection. So we use AWS Shield and Shield Advanced. Um we have this problem with game servers, right? So we want to be able to run dedicated game servers, which is a different workflow than running a game service. So we, you know, either we built it ourselves or we work with a partner like Amazon to help fill that gap.

Uh we use all their monitoring and logging for all of our security. Um and the list goes on. So this is just a snapshot of some of the features that we use from them.

So when we look at the design of Hydra, I want to talk about this really quickly our environment setup. So we have a concept of there, there's two levels of environments, we have system environments and we have Hydra environments. A system environment is a full deployment of Hydra. So the way that we deploy our application, so Hydra is built from the ground up to be multi-tenant, right?

So we want to be able to have multiple games running on a single environment at any point in time. Now this works for a lot of our different current, like our sandbox or our developer integration environment where there's not a lot of changes or spikes in traffic. We have all of our developers using that same environment, but we also have this problem when you're launching a Mortal Kombat or Hogwarts Legacy with, you know, these regular updates, really unpredictable traffic at times, we need to make sure that we can deploy separate infrastructure to support all of those titles.

So in our system environments, they all they all share the exact same code base, but we're able to deploy those out separately. And within the system environment, we have what's called a Hydra environment. So the system environment we control is what we control, we manage and we operate the Hydra environment is what we give our developers if they want to separate their configuration and their data.

So our developers, some of them might decide we want to have a different Hydra environment based on the branch of the game that we're working on or we want to have a separate Hydra environment that's just for QA or a separate Hydra environment. That's for certification. So we give them all the tools that they need to build out and adjust and create these Hydra environments.

So we're gonna dive a little bit into one of these system environments. So this is what Hydra looks like. So we have two entry points and two hour stacks. So we have a developer entry point and then we have our player. So from our developer's perspective, they interact primarily with Hydra through our developer dashboard. This is a lightweight dashboard is built in Vue.js. It does not have any database, it is built to be a window into the rest of Hydra so that our game teams can use this and adjust and update their game configuration at any point in time.

Alongside this developer dashboard, we also provide them with a developer API. So again, how do we put power back into our game team hands? So because this is a lightweight, basically everything that our dashboard can do. Our developers have a developer API and a developer key so that they can do this directly through the, the, the command line interface or they can tie into some of their infrastructure and deployment that they monitor and operate. Or maybe they want to build their own dashboards, right? We don't care, we want to be able to support developers do all of these different pathways.

So as a player, you have two entry points, first entry point is over HTTP TLS. This is your standard sort of request response API and all of this is going primarily to our service stack here. Their other connection point is they have a websocket connection to our real time environment. So our real time environment is for any time we need to distribute and a game event low latency, bidirectionally, this is the environment that they're connected to.

So with an Hydra, one of the things you're gonna notice really quickly is that this is not a microservices architecture. So we have done the microservices architecture in the past, really successful for us for a while. And one of the things that we found with that architecture is that when you have a few microservices, it's really easy to manage. But once you start having 40 50 60 different microservices again, if you're building on platforms and games for a really long time, it becomes really complicated to keep those updated, it becomes really complex to add a new one to understand the data modeling and how data is gonna flow between each of those microservices.

So what we decided to do was invest in one big monolithic service and we said we're only gonna build a separate service if it needs a different workflow than that primary, you know, REST API request and response.

So as an example, we have our server side code system, which our game teams use to write code that we host in the cloud for them so that they can support custom editions to the game, right? That's a different workflow. We have our matchmaking system, right? This is not your standard sort of API exchange. Instead, players are requesting to get into a pool with all of the other players requesting it to get into a match. And then our matchmaking algorithm that our game teams can adjust on the server at any point in time. It is churning through those players as fast as possible to find the best match and keep people matches as quickly as we possibly can.

Uh we have a task system that we use to run and manage long running tasks and routines, right? Things that go out of that different workflow. As an example, if we need to interact with millions of profiles at once, say our game team wants to roll out an update that's gonna add a particular profile stack to every profile, right? That's millions of players. What we're obviously not going to do is go and make API call and update billions of player, millions of players of records, millions of player records at once. Instead, what we're going to do is we're going to put that into a task and that task and that process is going to turn through those events over time.

So again, it's, this is a different workflow. So that's sort of the the design of our system. We focus on building one monolithic service architecture. This allows us to add new features really quickly. It allows us to make sure those features stay connected and tight throughout the integration. Um it's made easier for us to manage and read the code there and then we have pathways for having some type of work flows when we need them.

Um so, you know, we're gonna go over to the service side, we'll talk a little bit about the database layer. So our service layer, right, this is the piece that managing all of our player data, our leader boards, our matches our events maintains all of our developer configurations. This is written in Python and we use three databases.

So we have MongoDB. This is our primary data store that we use for keeping track of all of our state, full data for our events like sessions or leaderboards, we use Redis, right? Anything where we are going to deal with transient data or if we need to sort on something really quickly Redis has been a great tool for us to do that and support that in the past. So it's a core part of our infrastructure.

And then we use Elasticsearch. Elasticsearch indexes all of our MongoDB events so that we can do a couple of things. A we want to provide more visibility back to the game developers and us. So every time an API call is made into our system, we put a, we log that record and we make that accessible to the game teams so that they can go through, they can filter those events using a very complex log viewer. It helps them with debugging, advancing and developing their features and services.

Uh we also use that data so that we build all the different operational telemetry and metrics that we need to keep an eye on our system, right? How many players are playing right now? What modes are they playing? Um you know, what are they spending their money on? Um where are we having issues? Are we seeing a spike in errors anywhere we want to be able to have that information in real time.

So Elasticsearch has been a great tool for us for that. And then we also use Elasticsearch for search features. We index all of our MongoDB records so that at any point in time if we need for whether it's a game facing feature to do a find a player, find a friend or some sort of matchmaking. Uh we use Elasticsearch's search features and, and functionality for that.

And these are things that we build, right? So we run and we manage all of our own databases. Um none of these are managed services. Uh big part of that is, you know, I, I think if we were going to start this again today, we would look a lot closer at the technology that's available that it's come such a long way just over the last five years back when we started talking about, you know, how we're gonna, how we're gonna manage and run our own databases.

Um the managed services at the time, they just weren't where they needed to be to support the scale of games almost to the extent that with some providers, they actually said this is a great managed service for a low an app that has a little bit lower traffic. But what with, with what you guys are doing, you guys should really think about just continuing to run your own.

So we've just invested so many years, over 10 years in running optimizing and deploying databases. Um this is just where we spent time. We're really good at it. Now, we can run these at really high scale at a really low cost. Uh but I think if we were going to do this again tomorrow, we would actually take a different look at all the different managed database options that are out there today,

Service side code. So this system, this allows our developers. So as we talked about, our system has lots of different primitives and our teams can take those and put those together however they need to. Uh but what we run into is that typically gets them, you know, we say like 80 to 85% of the, the, the path that they have to go to, right?

Um but there's that the remaining 10 15% that game needs. It's it's custom, right? Each of our games are different, each of our games are unique. All of our game teams need to solve different types of problems. So we need to, again, we have tools that are in our hands. How do we put tools in our game team's hands so that they can go and they can advance and update our API to their specific needs anytime they need to. Here is the remaining transcript formatted for readability:

So we create, this system is running on Node.js as our runtime engine. And our developers have the opportunity to write Javascript and deploy it to the cloud.

We have a concept of SSC event hooks. So this means that we have lots of different events, hundreds of events that we track in our system - log in, log out, join a match, leave a match, add a friend, right? So when these events happen our team, our game teams have the ability to tie into these event hooks and add additional code or logic to it.

As an example, somebody might want to say when a player logs in, I also want to reach out to this external service to see if they're banned and if they're not, you know what? Because they're signing in for the fifth time. I want to go reward them something in the inventory system or I want to let this player know about these friends that have signed on.

So this is all custom logic that our game teams can use.

They write in job script, uh they can add it to the system. Uh it is uh the sources is git. So our, most of our game teams use github. Uh we pull in that source. Um we stage it in our system, we create a snapshot for it. So our game teams can look at that snapshot and have it ready to go for when they're ready to deploy it.

And at any point in time, right through our dashboard, they can decide to warm up the servers and get things initialized and deploy their code at scale. So this means that our game teams because of the power of this tool, they move so much of their game logic into ssc instead of the game clients. So if they need to make changes to the game, well, after it's released, balance changes, a workflow changes, whatever it is, they do most of that through ssc without having to worry about building a new patch and running that through all the first parties.

And then our real time system, um you know, a real time website, website connection. Um this is all about low latency, bidirectional communication. Uh the goal here is to allow our players to communicate with each other in real time or with the system. We use this system, it's written in java, we use this system for sending system messages, right?

So um maybe there's a new event that's, that's releasing, maybe there's a new notification. Uh maybe there's a new item in your inventory. We want the user to know about it right then and there without having to call out to an api uh maybe we uh and then our game teams, they want to have things like chat so our players can chat with each other, chat within clans, chat, within groups, et cetera. So we built this sort of real time system to support all of those different workflows.

So these are things that we built. Um and then we're gonna lean into a system that um we didn't build, but we tie into. So hydra uh has a direct connection because the way we built our system, remember we're doing all these lego bricks. So if we can tie in with a bunch of features internally, additionally, we can tie in with features that exist outside of hydra as well.

So within our system, our game developers have the ability to have our players match, make join a match and then go spin up a server on game lift. Right. So could we build dedicated game servers? Absolutely. Right. We have all the tools we have all the expertise to do this. But if we don't, if us building that is that going to really help us move the needle and deliver a better game when we have a team at aws that is really investing into building great tools and technology here. And we just watch how they do developed and advanced this tool over many, many years. And gamelift has become a powerful tool for us.

We used it for some smaller deployments, but this is our first time trusting such an important part of our business in a managed server with aws. And it's been an incredible journey. We're really proud of what we have here. Our players have had great experiences, low latency, great auto scaling so that we can scale up and down to our player needs for when we have that rush on day one or when we're at one of our valleys, you know, throughout the um the addition to local zones have been fantastic. We've been using that to make sure we can get closer and closer to our players so we can uh provide lower latency connection between all of them.

Uh the d os protection is built into the service. So it is a well rounded service that ties directly into hydra that is really managing an important part of our infrastructure and our player experience and is doing it really well.

So now we move over to and then we run a long time, we're gonna move on to how we uh get ourselves ready for scale. And i think that's it.

So as you guys can see, right? Um we have big games. We have a lot of uh a big piece of technology in hydra. Um our format is we don't just throw that over the wall at our development teams and they figure it out. We actually work with them. We are there working right alongside of them. We have a technical account management team who helps them build hydra, helps them design out their features.

Uh and preparing for scale is the other big half of, of our bread and butter, which is we need to make sure that we an infrastructure set up for day one. All those challenges i mentioned earlier with traffic waiting there at the midnight launch. We literally need to be prepared for that.

So, you know, we fail as a central technologies team if we're not ready and prepared for the anticipated traffic on day one. If our online servers are down, players are angry, our development teams are angry and literally the game loop, the main game loop of all of our titles fails to exist. So we pride ourselves in preparing for scale as early and as often as possible throughout the entire life cycle of the project.

So one of the first things that we made sure no single points of failure, we removed all single points of failure throughout our entire stack, all of our services are distributed across multiple replicas. Our mongo db database runs completely shouted. Our reddest and elastic search databases are all clustered.

As steven mentioned earlier, we launched a crap ton of games. We had a lot of success as we had a lot of failures as well too. We turned every failure into a lesson learned. A lot of those lessons learned pointed at no single points of failure. So this is one of the major thing that we put a lot of development time behind about five or six years ago, our low testing method.

This is something that i'm super proud to talk about. So our technical account management team factors into here a lot. So again, we don't just have our game development teams use our technologies and go figure out the launch on their own. We work side by side with them in order to be ready for launch.

We have a whole low testing method that we do for every single individual title. So we we perform a full api review prior to the launch of our titles. What's an api review? We will literally play the game that the team has developed with our technology, we will record that traffic in, we will analyze it and monitor it. We've launched a crap ton of games. So we have a lot of best practices, a lot of lessons learned that we tell them about.

So we're giving them, hey, don't use this api call, use this over here, batch these early in the project. Not just late, not when there's only two days left and they have no time to to fix their game. We write low tests based on typical user behavior. So we take that traffic that we can forward through our a power review process. And we literally build a low test script that we will run and simulate to the traffic that we expect.

On day one, we use forecasts for player distribution and scale targets. So we force our publishers and our analytics team to tell us what are the trends, how many people are going to be there on day one for mortal combat one, we give, we make them give us those data that data as early as possible.

So we combine those two efforts. We put a p a reviews and low testing. Actually on road map, we have a best practice. We want to be low testing your game three months before any user facing launch happens so that we have time to actually find something, find an issue and have you correct it in time so that it doesn't bring the game down when there's millions of players playing on day one,

We run all of our low testing applications across multiple ec2 instances, right? So we spin up a low test specific system environment. And if we expect 500,000 concurrent players to be there, day one, we will run our low test script simulation to that number. We will see where we fall over. We will see how much mango db we need. We will see how much elastic search we need.

We run a lot of different types of low tests, burst tests, endurance tests tests that run 12 hours, 48 hours. We always find edge cases when we do that breaking point tests. We want to make sure we can handle the day one traffic and then how much further can we go before the infrastructure breaks down? We want to know that going into launch. So we know how to prepare.

And after that's all done, we literally take a snapshot of that low test environment and we spin it up for that game specific system environment. This is our tried and true uh format. We've been doing this for all of our games to great success.

This is just a little bit of a diagram. You can sort of see our low testing method, but um our biggest launches just we have had over millions of concurrent players playing. At the same time, we've had to low test to those values. We've had upwards of over 300,000 api requests per second at, at launch time for some of our games. So those are some of the values that we've low tested.

Sure. This is a, this is a really important slide because um we are using our java sdk when we are running this low test. So our goal was again, like elliot said, we have had some big games, knock our systems over and we go back and we scratch our heads and we go every time we, what did we miss? Like, what do we do here to get this close to a real world environment without having 500,000 people play?

So our lowest client is actually running the job of dk. It's actually connecting to our website connections. It's actually like sending events and connecting to the api. So we are running our real sdk and hundreds and hundreds of low test clients simulating that traffic to our system. It is really expensive, but it is what we do. It's an investment for us to make sure that when we are going out there for launch, we know that we can stand up to 500,000 million, 2 million, whatever it is, players they want and we can prove that and then replicate that environment and get ready for those actual players.

So just to wrap things up here, um you know, working with aws has been great, right? So we've been able to meet all the demands of our game teams. All those four bullets that we talked about earlier.

Um we provide our players with reliable online experience. There's wood up here, i would knock on it. But because of all of these investments for well over a decade, we feel pretty confident, knowing. We know we know what we have to do to make sure that game is ready to launch. And the only way, the only thing that's going to knock us over is if you know the number of players quadruple and surprise us, or if we skipped an item on our checklist, like we know we have to do.

Um we spend a lot of our time focused on innovating on products because we're using so many more of the aws services we can deliver at a global scale and then because of aws and are constantly working with them, we are able to stay right on the cutting edge.

So what's next for us? Um you know, we fo we focused a lot on sort of our online service technology. We want to start investing more into engine technology. We have some plugins that sort of start this, but we start to think about with our game team again, keeping them focused on innovation. Are there any problems that we can solve for them end to end, deliver them a full plug in? Um all the way that connects with all the different game services and providers that we have to help them solve problems.

Um we want to make sure that we're bringing more and more players online, especially with our ip that our communities are going to be safe and secure. So we are looking at ways that we can use uh a i and machine in order to make sure our communities are safe and that our players know when they can show up to our games, they can be themselves and feel comfortable.

And then last but not least, we just are kind of rolling through our last big slate of games and we're starting to prepare our next slate of games. So the next conversation is, hey, we know what it took to deliver all of these games over the last year. What is it going to take for us to deliver those games, improve the quality and do it at a lower cost of the game developer for years to come?

So, uh i want to thank you all for joining our talk. Hopefully you all got something out of this. We really appreciate you listening. Thank you.