Running AWS Device Farm tests from Amazon CodeCatalyst

Everybody, welcome to my talk.

Before we get started, if possible, could I get a quick show of hands - who has been involved in a critical incident on a weekend where something's happened? You've been called on a weekend and there's an issue, right? I've had that. You know, it's a tough, tough time for an engineer, right?

Ok, so it's not only frustrating when that happens, right? But the it has significant impacts on your, on your business and the revenue that can, that can happen, right? If you consider Amazon.com, for example, if the 'Buy Now' button didn't work for 10 minutes, the impact on our business would be in, in millions of dollars revenue.

Not just that - customer trust and, and uh is a, is a significant impact too which is important to the company.

Ok. My name is Rob Bradley. I'm gonna talk to you today about integrating Amazon CodeCatalyst with AWS Device Farm.

So our agenda here - what I'm gonna do is talk about shift left in the development cycle where we, we wanna talk about that theory. Then what we're gonna do is we're gonna talk, we're gonna introduce to you Amazon CodeCatalyst and AWS Device Farm.

We're then gonna have a short demo that shows the automation and the we've developed a GitHub action that I can, that I can show to you today and I'm gonna be able to give you some takeaways that you can take away yourselves.

Ok, so as I mentioned, we were going to talk to you about the development cycle here. We have a classic five phase delivery cycle where we have Develop, Build, Test and Deploy.

If we consider a traditional testing, a traditional delivery model here, which is where the peak of the graph is towards the is towards the right hand side - this is where a lot of effort is consumed in that release process.

Let me introduce to you a second line to the graph, which is the shift left principle where what we're doing is we are empowering developers to do testing as they're committing their code. So I want you to focus on this area here. And what we're looking to do with this concept is reduce that cycle down, reduce, reduce the effort by shifting left. So if we shift that effort to the developers, giving them that ability to test their code, we are reducing the effort that we see here.

Ok, so let me introduce to you Amazon CodeCatalyst. Amazon CodeCatalyst is a unified software development environment service. It allows developers to quickly build and develop applications because everything they need is within one unified source.

We have automated workflows in the form of CI/CD pipelines where the developer can see visually where those automations are failing.

What I want to introduce to you next is AWS Device Farm. AWS Device Farm is a remote testing service that allows you to test web applications remotely in a browser. It can improve the quality of your test results because what, what you, what you get or what the developer gets is test artifacts that are generated within Device Farm and those are then sent back to CodeCatalyst itself.

Device Farm allows the engineer to test the websites without provisioning hardware. It speeds up the testing too because Device Farm allows you to test in parallel so you can kick off multiple tests in one go and get the results back at the same time.

Ok, so what we've done with our development process is we have linked the two services together using a GitHub action. What you've got here is you have a developer who commits their code within CodeCatalyst. Then what happens is the action runs as part of the pipeline and then that action invokes the run within Device Farm and then the artifacts that are generated are pulled back into CodeCatalyst for the for the engineer.

So they never have to leave CodeCatalyst in order to get those test results. And we can see here just an example of the two types of artifacts that come back - we have screen recordings which show how the the the browser is working. And we also have a test result file as well.

Ok, so let's get into the demo. Thankfully, Amazon.com didn't let me produce this demo on their website. So I've produced a to do app here, which is just a simple to do application.

What we can see here is the application's functionality. I have it hosted on CloudFront so it's publicly available, but private testing is available on Device Farm as well.

So you can see here my web UI - I create a task. So I'm creating a task with my title. I'm doing all of this manually. This is very time consuming what I'm doing here. I'm entering in the description for the task and then what I'm gonna do is I'm gonna, I'm gonna save that task into my application.

So what we see here, my representation of my task. I can test the UI functionality of the task by clicking on it and seeing how that's behaving. I can also test out different parts of the UI to see how that behaves as well and I can also complete my task as well.

So this is me performing a UI based test manually.

Ok, so let's move on now. This talk is basically designed to show you how you can integrate Device Farm with CodeCatalyst. I'm not gonna go into detail too much on CodeCatalyst. But what I'm gonna do is introduce to you how the service looks.

So this is what you see when you log into CodeCatalyst. What we've got here is - these are recordings. So what we've got here is we have a name of my project. I have the source repositories that are part of my project. So it could be made up of multiple repositories.

I'm allowed to invite members of my team to collaborate with me. So I've got a collaborative experience there. If I move on, I have a central place that I can monitor my issues on my project, I can review my code that exists within CodeCatalyst.

So here we have my source code for this repo, I can also generate and review, pull requests against the project. And what we're gonna dive into here is I can see where my CI/CD pipeline exists as well.

So if we have a look at workflows, this is what a classic workflow screen looks like in CodeCatalyst. See the name of the workflow. We can also see the status of the previous workflow runs. If we expand this, we can see the runs that are associated with the commit ID and I can see when that actually happened.

So let's dive into this run and we'll take a look at what that looks like. One of my favorite features about CodeCatalyst is this visual representation of what the pipeline looks like. So you get a very nice visual representation, I can see green ticks against each of my actions.

I have an initial workflow source which is where my code is being pulled in. I can see the duration of how it ran. And as I go down, you can see a sequential flow with parallel actions that can happen at the same time. So here we have two parallel actions occurring on the pipeline. These could be classically compile source code, build source code.

You can see this build package here. What we can do is we can review the source that was used. We can also have a look at the logs that were made available to me as well and the reports and the configuration as well associated with that.

Ok it's got some more stats about the way the configuration looks and again, more detailed logs about how every step ran in the development pipeline.

Ok, so the action that I introduced to you earlier is a Device Farm action that we're gonna look at on the last three boxes of the pipeline here. What we can see here - the first action in the pipeline goes off to Device Farm and creates a project for me. My developer hasn't had to go to AWS Device Farm and create a project. This action is taking care of that, meaning that the developer is not leaving CodeCatalyst. They're getting that unified development experience.

One of the outputs of this action is my ARN for my Device Farm project. And that is passed to the next stage in my pipeline which is where my tests are being executed from. What we can see here are the logs again, it's it's exporting the ARN as a environment variable that CodeCatalyst can use.

And then if we move on, we can see here, let's move on. If we zoom in a little bit, I can see my next action is actually the run action for the tests. And here's where you can integrate your own testing framework. I've got examples on PyTest or G depending on what you wanna use for your test wrapper.

Here we are, we're pulling in the environment variable from the previous step. And then finally, I've got my execution of my tests.

Ok, we're just moving to the final stage. The final stage of the pipeline is where I'm pulling my resources and my artifacts from Device Farm into CodeCatalyst.

AWS Device Farm produces screen recordings of what the test, what happened in the test, but it also produces test reports as well, which we can see here. So let's dive into those test reports and have a look.

So my test report here contains three tests that occurred - should complete a new, should create a new task, should complete a task and it should complete a task within 10 seconds. For my example, here, I can see more detailed logs about how that test executed.

What Device Farm has done, what CodeCatalyst has done is it's gone off to Device Farm and it's invoked those tests here. You can see the tests happening on Google Chrome and Firefox and you can see them happening in parallel and get that detailed report.

And as you can see, I have my spec files here as well that denote how the tests performed. So if we move on, we can have a look at what those artifacts look like.

So let's move forward. What I have here is one of the artifact files that's within CodeCatalyst that came from Device Farm. Let's expand that zip and have a look inside of that zip and see what we've got.

So we've got the, the zip file here is the name of my repo and here are my two executions - one for Chrome and one for Firefox. What inside of here, I can see a video output of what actually happened on Device Farm. Blink and you miss it because this happens in about 10 seconds.

So here we see a task being created automatically and the task completing automatically as well. So this was executed remotely on Device Farm.

Ok, so let's move on and I'll talk you through some of our source code that we've got here. This is what your developer might see when they clone down a project related to this.

What we've got here is the layout of the project. I have a backend which contains my infrastructure code which might have a DynamoDB in there. It might have various different infrastructure to make my application work. You can see here, I have a lambda and I have the test folder there that's associated with previous parts of the pipeline as well.

I have a CDK file which defines my infrastructure that gets deployed. And then the next folder I wanna talk about is the frontend. So this is what the frontend we saw when I was creating that task. This was the frontend that was being tested automatically by Device Farm.

As a developer, I might be introducing features into my UI. As I'm introducing these features, I can build tests and allow those tests to execute.

The final folder and this is one of the key concepts of the action that we created was - I would like all of my developers to be able to have a folder where they can bring all of their tests to drop them in that folder and commit those tests. This is just how I have my repo set up, but you could have a different testing framework in place if you want that.

Ok, so next, what we're gonna do is we're gonna have a look at the configuration file. This is my WDIO configuration file.

What we can see here is I'm defining where my tests exist. I can specify how many instances I want to run in concurrency. So I've specified a limit of four, I can specify the browsers that I want to test against. So here I've got Chrome and Firefox.

I'm basically here suggesting I want to use the Mocha testing framework. So I have this control over how I want my tests to run. And then here we can see this is where that environment variable is being pulled in from the action. So that's used by the CodeCatalyst system itself.

And then finally, we're using JUnit to produce test reports for the developer to have back in CodeCatalyst.

Ok, let's move on and take a look at what one of our test specs look like. So this is what one of my test spec looks like - this is 'should create a new task'. I have my URL in this test file but that can be passed in as an environment variable.

Here we have 'describe the to do test' - so this is like the harness of my test that exists. 'Should create a task' - what we're seeing here is various different interactions with the Selenium grid that's running on Device Farm.

So what we've got here is 'check that the browser title equals To Do App' - again, this could be 'check that the Amazon button is working on Amazon.com'. But obviously this contains my code that interacts with the UI.

Ok, so here we go 'should complete a task' and then we have various waits and confirmations that exist as well. So I'm getting this feedback as I'm working within the tool.

Ok, I promise that we would have some takeaways. So we'll come on to those in a minute.

We talked about shifting left in the development cycle and reducing the effort it takes when you have testing later in a development cycle.

Next, we talked about promoting a test driven development - CodeCatalyst really helps here because your engineers are using a unified environment and they're reducing that context switching.

AWS Device Farm is reducing the undifferentiated heavy lifting. So the engineer, the developer hasn't had to think about which, what sort of grid size do I need? What infrastructure do I need in place? That's really helping out here as well.

And then what we've got is - Device Farm provides efficient testing at scale so I can execute tests in parallel on my Device Farm.

And then finally I mentioned just before - AWS Device Farm and Amazon CodeCatalyst reduces that context switching between tools. So I have this single UI, I've got my VS Code. I don't have to go off to Device Farm, set it up. I don't have to go off to other tools as well to set that up too.

Ok, I wanna say thank you so much. You've been a fantastic audience. Here we have a couple of QR codes. The one on the left is our GitHub action that you can take away with you. This will allow you to play with that code as well. Here is the rest of the transcript formatted for better readability:

And then finally, I've got my execution of my tests. Ok, we're just moving to the final stage.

The final stage of the pipeline is where I'm pulling my resources and my artifacts from Device Farm into CodeCatalyst. AWS Device Farm produces screen recordings of what the test, what happened in the test, but it also produces test reports as well, which we can see here.

So let's dive into those test reports and have a look. So my test report here contains three tests that occurred on my web UI - should complete a new, should create a new task, should complete a task and it should complete a task within 10 seconds.

For my example, here, I can see more detailed logs about how that test executed. What Device Farm has done, what CodeCatalyst has done is it's gone off to Device Farm and it's invoked those tests here.

You can see the tests happening on Google Chrome and Firefox and you can see them happening in parallel and get that detailed report.

And as you can see, I have my spec files here as well that denote how the tests performed. So if we move on, we can have a look at what those artifacts look like.

So let's move forward. What I have here is one of the artifact files that's within CodeCatalyst that came from Device Farm. Let's expand that zip and have a look inside of that zip and see what we've got.

So we've got the, the zip file here is the name of my repo and here are my two executions - one for Chrome and one for Firefox. What inside of here, I can see a video output of what actually happened on Device Farm.

Blink and you miss it because this happens in about 10 seconds. So here we see a task being created automatically and the task completing automatically as well. So this was executed remotely on Device Farm.

Ok, so let's move on and I'll talk you through some of our source code that we've got here. This is what your developer might see when they clone down a project related to this.

What we've got here is the layout of the project. I have a backend which contains my infrastructure's code which might have a DynamoDB in there. It might have various different infrastructure to make my application work.

You can see here, I have a lambda and I have the test folder there that's associated with previous parts of the pipeline as well.

I have a CDK file which defines my infrastructure that gets deployed. And then the next folder I wanna talk about is the frontend. So this is what the frontend we saw when I was creating that task.

This was the frontend that was being tested automatically by Device Farm. As a developer, I might be introducing features into my UI. As I'm introducing these features, I can build tests and allow those tests to execute.

The final folder and this is one of the key concepts of the action that we created was - I would like all of my developers to be able to have a folder where they can bring all of their tests to drop them in that folder and commit those tests.

This is just how I have my repo set up, but you could have a different testing framework in place if you want that.

Ok, so next, what we're gonna do is we're gonna have a look at the configuration file. This is my WDIO configuration file. Let's press play.

What we can see here is I'm defining where my tests exist. I can specify how many instances I want to run in concurrency. So I've specified a limit of four, I can specify the browsers that I want to test against. So here I've got Chrome and Firefox.

I'm basically here suggesting I want to use the Mocha testing framework. So I have this control over how I want my tests to run. And then here we can see this is where that environment variable is being pulled in from the action. So that's used by the CodeCatalyst system itself.

And then finally, we're using JUnit to produce test reports for the developer to have back in CodeCatalyst.

Ok, let's move on and take a look at what one of our test specs look like. So this is what one of my test spec looks like. This is 'should create a new task'. I have my URL in this test file but that can be passed in as an environment variable.

Here we have 'describe the to do test' - so this is like the harness of my test that exists. 'Should create a task' - what we're seeing here is various different interactions with the Selenium grid that's running on Device Farm.

So what we've got here is 'check that the browser title equals To Do App'. Again, this could be 'check that the Amazon button is working on Amazon.com'. But obviously this contains my code that interacts with the UI.

Ok, so here we go 'should complete a task' and then we have various waits and confirmations that exist as well. So I'm getting this feedback as I'm working within the tool.

Ok, I promise that we would have some takeaways. So we'll come on to those in a minute.

We talked about shifting left in the development cycle and reducing the effort it takes when you have testing later in a development cycle.

Next, we talked about promoting a test driven development - CodeCatalyst really helps here because your engineers are using a unified environment and they're reducing that context switching.

AWS Device Farm is reducing the undifferentiated heavy lifting. So the engineer, the developer hasn't had to think about which, what sort of grid size do I need? What infrastructure do I need in place? That's really helping out here as well.

And then what we've got is - Device Farm provides efficient testing at scale so I can execute tests in parallel on my Device Farm.

And then finally I mentioned just before - AWS Device Farm and Amazon CodeCatalyst reduces that context switching between tools. So I have this single UI, I've got my VS Code. I don't have to go off to Device Farm, set it up. I don't have to go off to other tools as well to set that up too.

Ok, I wanna say thank you so much. You've been a fantastic audience. Here we have a couple of QR codes. The one on the left is our GitHub action that you can take away with you. This will allow you to play with that code as well.

And then finally, we've got our a LinkedIn link where you can reach out to myself and connect. I'm gonna be around at the booth entrance over there so I can take kind of 1 to 1 questions if you want to have a chat about this integration further.

But yeah, thank you so much. Don't forget to rate this lightning talk in the events app. That would be fantastic.