Improve your mobile and web app quality using AWS Device Farm

m 202. Hopefully you're here to learn about Device Farm today. If you are not, then you're probably in the wrong session or saw the wrong room. So just make sure you evaluate, make sure you're in the right session.

Well, without further ado I'll go ahead and introduce myself real quick. My name is Sam Patzer and I'm a Solution Architect covering Riot Games. Uh so my customer is pretty awesome. So I'm gonna let them talk and introduce themselves. Uh and then yeah,

Hi, I'm Alex Lachowski. I'm a Staff Software Engineer at Riot Games. I work at the central tech facilities uh building centralized tech solutions for all of the games in Riot Games and I'll hand it off to Niel.

Hey, everyone. Good morning. Uh my name is Nikil. I'm the Principal Cloud Engineer for Front End Web and Mobile. More specifically, I look at Device Farm in a specialized role and have been with the team since the launch of the service. Uh I enjoy making customers navigate their challenges, technical product commercials. Uh anything that serves towards improving the quality of their apps, happy to be here and I'll hand it back to Sam.

Awesome. Thank you, everyone. So first I'm just gonna go over a quick agenda for the presentation today. So that way, you know, when to pay attention, when to fall asleep. Definitely like during my time. Uh so first of all, I'm just gonna give a level set. This is a 200 level conversation slash presentation. So I just wanna make sure everyone knows what Device Farm is and why would you care about using something like Device Farm? After that, I'm gonna kind of talk about who Riot Games is and where do they get their start? And what are they doing today? And then afterwards I'm gonna hand it off to Alex to talk a lot more about kind of what does Riot do when it considers testing as a whole at Riot and then dive into more details around what they're doing with Device Farm and how they implement it at Riot Games and specifically how do they test games. And then afterwards they're gonna kind of talk about the decision making behind why they just choose Device Farm and some common questions that they get when they talk to their engineers about Device Farm because the game teams obviously don't run the Device Farm. And afterwards, I'm gonna hand it off to Nick Hill to talk about some of the new features and recent launches. that Device Farm has had because there's a lot of cool new things that were coming out today.

So the level set. What is Device Farm? Uh Device Farm is a DevOps tool that allows you to test validate and debug both mobile and web apps. There is no provisioning needed and it's just a one click button and you get to take advantage advantage of the cloud where it scales out to the needs that you need all in parallel. So you can run multiple tests at the same exact time.

What it does, what this does is it enables your teams to be able to test across hundreds of real devices. Let me click that clear real devices. So these are real devices in the cloud across thousands of versions that you're able to run these tests on. So what's really nice about this is allows you to, for example, when iOS releases another map, minor version or major version, you're able to continue to test on previous versions as you see fit most enterprise customers nowadays support anywhere between 3 to 4 major versions back. So what that means is that if we're on iOS what 19 or 18, I lose track at some point, you have to continue to support backwards as well. So you have to go iOS 1817 as well. And then on top of that, you have to make sure all your features work. It all sounds great when uh WWDC for iOS or Android's uh Google, next project, they announce some new features that are enabled. Well, obviously you need to make sure that you have backwards compatibility. And a lot of times that means that you have to make certain uh actions available on older iOS versions and make sure those actions are still actionable on those older iOS versions.

And to have a device or multiple devices that are running all those different minor and major versions, it actually can be a big headache because what happens if one developer, you know, clicks the update button by accident. Next thing you know, you lose that device on that version and maybe that's the only device on that version you have. So there's a lot of different considerations. What's nice about this uh service is that it just manages all that for you. So all you have to do is just install the app and then you run the app as uh with your tests and then it cleans up the device on your behalf. And what it does enable you to do is no updates. So you're actually able to continue to test on those versions in the previous. Also, you can automate all your tests as well.

So for those that have ever built a mobile app or front end web app before you know the pain of opening up Firefox and opening up Chrome and opening up Inter Explorer to make sure that the animation isn't broken or the CSS it looks horrible, you know, all that sort of thing or same thing with mobile apps, you know, wanna make sure it looks good on a tablet, make sure it looks good on the the phone that's massive, the phone that's small, make sure it fits with the bezel and everything else like that, that all can be done automatically. Now rather than you having to plug in every single device, open every web browser and test every animation if you can automate that, that is what Device Farm is able enabling you to do with all this, you also get all the logs back as an artifact.

What does that mean is that you're able to then automate and be able to take a look at those logs and actually debug or send that information back to the engineering teams. So that way you can enable your engineering teams to continue to iterate or even debug and go down the in depth as well.

What one of the big things we always are asked about is like what happens if for example, uh we're all familiar with Android devices, there is about 5 million different flavors of Android. We have Samsung uh flavor, Google, flavor, every single different type of flavor and all those introduce their own bugs or own issues. So for example, sometimes you know Samsung releases their own version update for Android that might introduce a bug that you might have to go solve later on and you can actually go test that and debug it with remote access as well. So you actually can then dive into that one device, be able to actually test to see where that issue is coming from, get those logs down, send those to the engineers or the engineers can be doing this themselves.

Now, as far as why would this, why would anyone care about this? Um the and the biggest thing is just around making sure that the quality of your app is continuing to uh exist throughout the entirety of your app. A great example is when iOS decided, or Apple decided to add a bezel to all their phones. Now, everyone who saw that was like, wow, that's really cool as a user. But as a programmer, everyone knows that's a headache because now you have to work around the bezel. Now, depending on the view that you're using with iOS, it might, you know, overlap with the bezel all of a sudden you're hiding your titles because of the bezel and all these sorts of things that, you know, as a user you don't think about. But as a programmer, it's a big headache and you might not catch those because you might not have the newest phone or there might be just a bunch of different issues that may arise. So that's where Device Farm comes in as well.

Now, extrapolate that out to all the different devices and versions out there. You can see why something like Device Farm could be very, very helpful. And if I forgot to mention this, this all can be done in parallel. So what's super nice about this is that while you're running all these tests, they don't have to be plugged in individually, you can run it across hundreds of devices. All the exact same time, get back all the results of the same exact time and then just collaborate and be able to ship off the errors if any errors to anything else. And you're good to go.

Now, one of the other common questions we always get is that I don't have tests written or I don't have tests a team to write tests yet. Today, I can't use Device Farm. Well, Device Farm does have built in things like a built in fuzz. So those aren't familiar with the fuzz or is is it just randomly clicks around on a screen, navigates randomly through your app and just make sure that things just don't break, which can be very helpful because you know other things that do happen and something that uh Device Farm tells you is like what your CPU utilization is. What is your memory usage and what is like your thread usage? You don't want, you know, a bunch of like navigating back and forth between your uh device uh menus, a bunch of threads will be spun up that aren't closed. You can have a device that overheats a device or an app that crashes and those things not always sometimes appear on a automated uh like desktop with Xcode or something like that because Xcode does better thread management. Uh oh, you're also working on a device that has 10 times more memory than your iPhone does. So it can handle way more threads. So these are things that running on real devices actually can enable you to do better.

So that is what Device Farm is. It's like a 10 minute spiel on Device Farm. So hopefully you guys understand what Device Farm is at a very high level. So that way when Nick Hill and Alex talk about it a little later on, you're not like uh what is Device Farm again, that is what Device Farm is.

So why should we care about who Riot Games is and who is Riot Games? So Riot Games uh or should say Riot Game at this point was founded in 2006. Uh they released their first game in 2009 called League of Legends. It was two teams of five battling to destroy the opposing team's base or nexus with that. They decided that what would be awesome is to make this a competitive sport and that was the release of uh LoL eSports. So LoL eSports became a global leader in eSports with an annual League of Legends world championship featuring the eSports teams of 12 international leagues. Uh and is one of the most popular and largest gaming sporting events in the world with last year or I guess this year's uh eSports ha hitting over 6 million concurrent viewers.

After that Riot Game became Riot Games adding an s in 2019 and releasing four new games. So on top of uh League of Legends, they also released uh Wild Rift, which is a mobile version of League of Legends and some sort of kind of coming into where Device Farm starts to exist a little bit at Riot. And then on top of that Team Fight Tactics as well, which is an auto battle chess game. Uh that's where uh Device Farm really takes the shine. And then Le uh Legends of Runeterra and then Valorant which is a first person shooter.

Now Riot is becoming more and more like a media entertainment co uh company where they're now releasing not only just games but also other content as well. So they have a lot of different uh things like uh brand new comic books as well as board games, Arcane, which is a fantastic Netflix series that I recommend to go watch as well as uh they now have uh their own independent studios that work with Riot's IP to release awesome games uh like the Legend of Runeterra, which is a brand new game that is released this year.

So without further ado, I'm gonna hand it over to Riot to talk a lot more about what they do with the Device Farm and all the awesome things that they do.

It's how we know a game is good and fun to play. But there's an entirely different side of testing that I'm going to be calling functional testing. It focuses on the technical side of things.

To dive in, first I'm going to go in with gameplay tests. It focuses on things like mechanics - is the game fun? All of these things are great and you need to focus on them to make sure you have a game players can enjoy.

Take us back to EVO 2023. This year, we had the first gameplay demo of Project L. Tons of people tuned in, were able to play Project L, and we got a lot of good feedback. You know, how great does it feel when Darius's axe swings down? How does it feel when the characters are introduced, the stylized graphics? These are all done by humans and tested rigorously by developers and QA alike, as well as playtesters like yourselves.

But now you know you have a fun game, how do you know it works on everything it needs to work on? That's where functionality testing comes in. Functionality testing involves things like performance testing. Does it run on this device? Does it overheat? How many threads are allocated on this device versus your developer PC? You need to understand how your game will run on the plethora of devices your customers use.

I'll tell you a quick story about a QA person who started out being asked by the game team, "Our game is fun to play. We know it's fun. The levels are working. QA, can you make sure it works on this device?" For the story, it's an Android device.

So QA takes their phone out and tests on Android 10. Yes, it works on Android 10. Now Android 11 is released. "Here QA, test it on this new phone." Now QA has to run the same tests on two devices. Android 12, 13, 14 come out. iOS devices too. You can see the QA's desk stacking up with tons of devices and cables - this poor QA can't manage it all.

So they hire another QA. Parallelism and scaling was thought of as a human solved problem - just hire more people to test on more devices. It became distributed across people.

Time to rewind - how did this happen at Riot? Once we put the S in Riot Games, testing became a big problem. We needed to test on all devices. Every game team had the same manual process - acquire a physical device, set it up on a desk, connect it, investigate issues. Not scalable.

So how did we solve it? We advanced to a centralized solution that all teams use so we don't acquire devices and pile them on QA desks. We designed a high level service that takes API requests, puts them in a queue, allocates devices, runs tests, and stores results.

What types of things are we testing? There are multiple things, which I'll dive into next on how we did this at Riot.

We have a centralized service for all game testing, with an API ingress point so teams can use it automated or manually. Requests go to a workflow request Lambda that puts them in a queue to prioritize order. The queue interacts with DynamoDB tables to get the right devices in a clean, ready state.

There are two major underlying device farms - on premises devices we gathered centrally, and cloud device farms like AWS Device Farm. We like the cloud farms because physical devices don't scale - hardware fails, cords trip, etc. The cloud is fully managed so we don't worry about that.

The workflow decider Lambda takes a request and decides which devices to use on premises or in the cloud. It ensures they are in a clean, ready state. Then a mobile state machine kicks off the tests.

Metrics and analytics are gathered during tests. We post events through Amazon EventBridge like "device running test", "device done", "gathering artifacts", etc. These are picked up by the workflow update Lambda to update status in the workflow request table and Kafka stream.

Why is AWS Device Farm best? It seems counterintuitive to give up control as a game team. But with device logs, screenshots, metrics, and video, you can see test results.

Game teams have to create a custom functional testing build that can be invoked through Appium or another framework. This allows parallel testing at scale across devices.

We have a centralized device catalog saying what devices you can use, their specs and availability. This is mirrored in Device Farm's filterable options.

Teams want to know - can this connect to my internal game servers? That's where VPC support comes in. You can connect your whole mobile test network to your VPC, both on premises and in the cloud. We have the device farm in one AWS account, and other solutions in VPCs as needed.

And then we have the cloud solutions and all of the AWS Device Farm devices are in separate AWS accounts. And these and through a VPC configuration during the call to the AWS Device Farm, we can say during this test run, please connect to our VPC during your test run so that this device appears to be directly on our internal network.

And this is all done safely and securely. And it makes sure that things like unreleased game servers, unreleased games can all be very easily tested on the cloud with no issues. And all of this culminates into game teams being able being to do things like nightly tests, uh you know, regression testing, scaling, testing, load, performance testing, all of these things are now possible. Whereas before it was, hey, can we get QA to slot in a time slot for us to test our game? Now you just make an API call.

And so to recap, here are the common questions that we get:

Can it run my game? How do, how does, how does your framework run my game on the cloud? And for this, we had a, we can run your game through a command line as long as your game accepts command line inputs. We're fine.

How does it run my game? Well, it runs it through a service called Device Farm Controller that then routes all of your requests onto the device itself, whether it's on premises or in the cloud and it runs your commands. Exactly as you'd like.

Can I connect to my game servers? This, we covered in the VPC section? A huge concern for a lot of, a lot of DM teams. You, you want to be able to connect to your internal things. We have an entire VPC configuration that's customizable and safely and secure, whether it's on premises or the cloud.

What artifacts and metrics does it provide? Sam touched a little bit on this. It provides things like CPU metrics, load metrics, but we also, but through the power of AM and AWS Device Farm support, it also does things like screenshots, videos and log bundling and all of these things can be separate separated based on configurations that the game teams provide.

And finally, you know, game teams as usually is the one of the last questions that they ask is, is this secure? Um you know, hey, now that it actually runs my game, does this is this actually safe for me to run my game? And the answer is yes, it's all within the virtual private cloud that you set up in AWS. None of this is on is ever going to the public internet. It is all internal. So everything is very, very secure.

And then even after security. Can it scale now that they know that it runs my game and it can run it securely. How does this run at scale? This is I think where the big benefit of functionality testing comes in and where things like Device Farm help immensely. You don't want to be scaling up your physical on premises device farm, everybody knows it's a pain. It's expensive cloud lets you do it by just sending an email. Hey, can we access these 20 more devices? Hey, we want to access 50 more devices on our, on our private cloud.

And with that, I will hand it off to Nikel to go more in depth about the future of AWS Device Farm. Thank you, sir.

Alright. Thank you, Alex. It's been a wonderful journey working with Riot uh with your talented team and just seeing how what they have come up with and I'm looking forward to the things that we feel in the future.

Uh before I just jump into new features and saying, here's what it is new, here's how you do this. Let's take a step back. The title of this session is improve the quality of your apps. And while Sam and Alex did touch on a lot of, you know, like, how do you approach this? How do you think? What are the different aspects if you say i want to improve the quality of my app? Is it just go out test on multiple devices and it's done. Is, is that what it is for most customers? That's how they start? Well, I got to test it on multiple things. Uh I have several frameworks that need to be invoked. But end of the day, if you're not testing on multiple devices, all those other questions become less relevant at the start.

So one would ask, well, ok, how do I go about this? Uh am I just starting off with manual testing? Am I doing full blown automated testing? What is the baby step here?

So in that view, one could take an approach saying, well, let's start with the approach that Alex's team took in his rewind slide where they said everyone go out, buy a device. Good, good, good, great. We're doing great. Oh, new release came out. Oh, there's a new Pixel, oh, there's a new Samsung and things start to get a little unmanageable at that stage.

So what is the other approach? Well, most folks want to reach a mature model saying here's an end to end flow where you are doing something and the entire pipeline works and you know what has changed. It could be between versions of an app could be just a single line of code, it could be just a splash screen that you changed.

So most people in this room will agree that it is an end to end mature model that they want to go to on top of it. Uh you want to have it secure and you wanna have it fast. But in all of these details, there is a simple, you know, when you talk to other customers, when you just talk to people, it just comes down to plain three English sentences.

When they say, can I do this somewhere? Can I do this on my own? Can I do this on AWS? And those simple three things are let me connect to my private end point. It could be anything, it could be public, it could be private, it could be a mix. It could be an on premise. So that's like let me connect, I have databases or whatever it may be.

Second is let me run it from anywhere. I have a local machine, you know, I want to reproduce something. Uh oh, I have a CI CD system. Oh, I have this legacy system that's there. So that's number two, let me run it from anywhere.

And then number three is let me customize it. Sure it runs. Sure I can connect. I wanna make some changes to it. Right.

So let me connect, let me change it and let me run it from anywhere, right? And one approach uh which most folks take is there's hyper focus on. Uh let's just get one stuff running and then we'll figure it out along the way, which is not a wrong approach at all. But I think these three can work in parallel independent of each other.

So let's work on each one of these pillars and just say, firstly, our point was let me connect to my end points. So Device Farm uh has a VPC and I feature that Riot Games has used. And this gives you an ability to securely connect to any private endpoint. This could be on premise, it could be on, on an AWS VPC uh or it could be on another uh another vendor.

So using the VPC and solution, we can put not only your devices, not only you can put an iPhone and a Pixel in VPC, you can also put the test host or the host machine to which the device is connected inside the VPC. So when you are running a test, either manual or automated, you connect the device to the host machine, you run some commands, but at times you may want both of them on the same VPC, you may even want both of them on the same subnet. So this is what that feature enables.

Furthermore, when you're running an app on a device, it may be a game app, it could be a streaming app, it could be a messaging app. It may have different traffic types. You know it could be web sockets, you're chatting, it could be streaming something. The connectivity feature will support all traffic types. There's no restriction on those and this feature today exclusively works with private devices.

So on Device Farm device offering. We have two types of offerings. One is public devices which are shared devices. Someone say, hey, i need you know an iPhone 13 running iOS 17 i 16, they can go in and find that device when they are done. Device Farm make sure that they clean all the artifacts, including app test, any logs, any screenshots, anything that you may have produced, not only on the device but also on the host machine.

So once uh those are done, any other person who's waiting in queue uh will get the device. Now there are multiple instances of each one of these devices behind the scenes. It's not just one instance. So if you get one, we'll have multiple of those and private devices. In contrast are dedicated devices. You don't want the guess work, you want your dedicated uh capacity.

You may also have features such as hey, i need to install MDM profiles. I need VPN on my device. I might have developer certificates that are exclusive to this. I might have SSL certificates that connect to my private end points. Private devices will enable those features.

And so VPC works really well with private devices today. So we got the, let me connect to my private end point. Now let's say let me run it from anywhere.

So for let me run it from anywhere going back uh Device Farm today can be consumed via multiple uh starting points. One is the most friendly. One is your AWS console. You go to the console access Device Farm, the other is your AWS command line interface, your CLI and then multiple AWS SDKs available in multiple languages.

So that's one point of consumption. And then there is a rich uh ecosystem of plugins that we have developed. For example, there's Jenkins plugins that great old plugins that you can go to those official stores such as Jenkins app store and then you can just download it from there, the plug in store building on top of that, we now have GitHub actions which was newly released just a few weeks back.

"And so this will give you the ability to invoke a device farm in an action directly from your GitHub repo. And that's not the only thing. You would also want features such as, hey, I ran my test. Now, I'm looking at the results. I wanna automatically cut issues, assign it to folks. So the GitHub, uh the device farm GitHub action will also support that. You can automatically create issues. Say, hey, test number x failed. Here's the issue that's automatically created for it. And then these can be scheduled as any GitHub action. These can be done on when you're pushing a code, when you have a new pull request or it could be even invoked on a schedule nightly build, for example, on this GitHub action.

So we got the, let me connect to any anything. Let me run it from anywhere. Now, the third one is, let me customize it. So when you schedule a test on device farm, now, those of you in the room who, who may have heard about this for the first time, I've never used it and we don't have really have a demo which I'll come to, but uh I'll give you a high level 10,000 ft primer. Essentially. When you're saying, I want to test something on device farm. The the system set up that you need is really just two inputs. That's all you just need your app that you need to be testing. It could be a native app, hybrid app or it could be a pure web app. So that's input number one input. Number two is your tests. This could be, you could take the source code, you could take the build output or you could just say, hey, pull my test on the fly from somewhere, you know, pull it from a repo. So that's your input number two, when you give these two inputs to device farm device form goes out and pulls these artifacts, spins up a two host machine on the fly for your android test and a mac machine for your ios test.

And then behind the scenes, it will connect those two, download your artifacts on those host machine form a connection with the devices you selected. So even if you selected hundreds of devices, it will spin up hundreds of hosts for every single device, there's a dedicated host, even in your same session, no host is being shared with any other device. And so there's a one on one mapping between the host and the device on the fly.

Now, when we launched the service about close to eight years ago, we had an ubuntu based ec two instance. And obviously we have upgraded it securities as prime. Uh but as time passed, we realized customers have more software dependencies. They're using more third party libraries, they're using tooling that's not as restricted as it used to be. For example, Alex's example of using am but then they have their own gameplay testing and their own framework. So things are becoming well. So the short version there is you need more control and you need to be able to install software dependencies faster and more easier.

And so we now have a new android test host and this essentially is an upgraded hardware uh runs amazon linux too. But and the inherent benefits of doing that is we have observed on an average for the customers so far, a 32% improvement with the overall test execution time. So you're to execution times and that is a factor of us also putting in more dependencies on these hosts which we realize customers use. For example, customers are might be just migrating to python 3.10. As an example, don't hold me anyone in the room who's really good with python. But you know, i see customers often say, hey, i want 3.63 0.73 0.10 3.11 i have one test branch that runs on this, but the other test branch runs on that. So to manage all of those dependencies, the host already comes pre installed uh with uh dependencies that most of the popular test frameworks might need. But you can now simply use a device farm cli centralized tool chain, the one that's highlighted. And if you look at that screenshot in the example, it's as simple as saying on line 22 device farm cli use node whatever version 16 or 18. And what that will do is not only update the node version for you, but it was also update all of the underlying dependent packages that go with node that your test might need. So it's even reducing further the lift that you need. Because device farms design principle has always been, don't fight the ecosystem. What that essentially means is if you are running and writing a test a certain way on your local machine, we do not want to fight that in front. We want to enable you to run it as is more of a lift and shift. So in order to even further accelerate that we are taking away the boilerplate work which says, oh, i got to change my python version. Wait, i also need to go pip along with it. Oh, wait, i need to also change my python environment with it. So all of those things are taken care by that device from cli told you.

And as also a other benefit is now we have faster manual testing. And what that means is device farm. What sam touched in the start, not only does it support automated testing, but you can also do manual testing. So manual testing traditionally has given a frame rate of anywhere between eight to as much as 12 and while that has been ok. But imagine someone like ryad running it and saying, well, i want every frame, i don't want to lose a frame when my brain is doing some fast paced action. And so not only does the new host provide improved manual testing, but we also have an improved frame rate 25 frames per second on all these devices. So you're gonna get, you're not gonna miss any frames when you're running these.

And am being a pretty popular testing framework cannot go to an uh to a test framework, talk and not talk about am device form comes with out of the box support for am two dot x now. So even if you have some minor versions of am that you need to add, those are all supported. Your friendly device form, cli will switch it with the line, all the dependencies including your wd agents, your chrome drivers for people familiar with the a pm ecosystem, it will do all of that for you just with that single line.

So we covered the three things. Let me connect anywhere, let me customize it and then let me run it from anywhere. Uh third, once you have done all of this, uh we see customers get even more creative, they have even more granular tests. So device farm also supports and rooted android devices and what this is primarily for use cases where if you look at the android for this is because we are talking about android. If you look at the android documentation, there's a lot of documentation in there says, hey, i can give you a deeper battery a analysis but it only works on a router device or in an emulator. That's one. Um and so that's one example on why someone would need routed device, other clever and nice way that i have seen customers really benefit. And this was a learning experience for me as well is i've seen customers install two versions of the same app. So let's say your app package name is com dot example one. And so i've seen customers have version b one and we two of the same package name signed by different stages and then they install both those apps because when you try to install the same package name on the device, it will say, hey, i already have a package, i can install it but with rooted devices you can overwrite that. And then once it does that it takes more metadata into consideration. So once you do that, they'll run one version of the app. Let's say that you say, ok, how long does my login screen take to come up? That's one, then they'll run it on the second app. Oh, log in for this one. Took x seconds more and then they'll compare it. So that's, you can have a comparative things having two versions of the same app on the same device. And that's only possible because routed devices allow you to do that. So that's another example.

The other metrics that we see is granular uh cpu consumption. Something that alex touched on on his team uses when they do load metrics and other things. Uh router devices will give you more ability on that. They'll also give you deeper and more granular control in the file system. And these could be cases where you are taking a binary dump, taking a tcp dump perhaps on the device saying how is it doing? What package is it sending? Uh and then you can take those files, analyze them, say for example in wire shark and just see that traffic is flowing.

There's also advanced cases where we see customers use perhaps a charles proxy sort of intermediate man in the middle proxy where they're saying all the device is sending all the http request, go to this proxy and you're monitoring the uh the traffic that's going in and going out. Uh router devices don't are not really needed for that use case. But you can have additional metrics logged on the device level. In addition to what you could do just alone on charles proxy, though that you can have a timeline where you're saying, oh, that request, https request was sent out this time stamp. Here's what the device did before that. So you can know whether there's a, you know, a a problem in the network from the device to the charles proxy to start with.

So those are all the newer features we are launching. I highly suggest or recommend anyone who wants to get hands on. There's a self service workshop that you can try out. It is really nicely laid out with level 102 103 100. You can get an end to end test an app running, don't have to have any app build or anything. So i would uh suggest trying that out the workshop was today morning. Otherwise i would have also suggested go and do the workshop and that's it. Uh that was the last thing i really appreciate everyone coming out here today, taking time. Uh sam alex and i will be outside after the session. If anyone wants to come in chat, say hello. Have any questions. Happy to help you with those. Thank you."