The AWS data-driven perspective on threat landscape trends

Ryan Holland: Thank you for joining us and welcome to today's session on uh data driven perspectives on threat trends. My name is Ryan Holland. I'm from the Amazon GuardDuty team and I lead the security and security engineering and I'm joined by Paul.

Paul: Hey, nice to meet you. I lead the, the Shield threat research team just focusing on analyzing our inbound threats and turning that into actionable intelligence.

Ryan: Yeah, I got it. Ryan. Thank you, Ryan will be back up on stage a bit later. Uh thank you so much for joining us today. Um we're excited. It's a Thursday afternoon after a long week and I'm sure there's some fatigue here. I really appreciate you coming here and supporting the talk and we hope to get something out of it.

Um this is a 200 level talk. So we're gonna look at some high level trends, try and understand what we're seeing from a GuardDuty perspective and from the perimeter of the AWS uh network. And uh there's no doubt gonna be some slides in here that are a little bit technical as well. So on average, it's a 200 level talk.

Um and I hope you enjoy that. What we're gonna cover is how we approach uh throat research, the types of systems that we've built to analyze and find trends. Um and then uh talk through how our defenses on our network are um designed to uh use this intelligence and protect our, the availability. Uh and then again, um from GuardDuty who's kinda like sitting on the inside of our network looking out at what they're seeing from their perspective.

So starting off um our goal, at least my team's goal at Amazon and our permanent protection organization is to uh this is our mission, right? This is we wanna make AWS an unattractive target. Uh we recognize that you can't eradicate threats. But the more work we do to make it difficult to target us uh and more expensive for bad actors to target us means that AWS becomes an unattractive target, which means it's safer for anybody running workloads on top of AWS.

And uh what we do is we have built systems that analyze uh inbound traffic. Uh and, and look at uh sort of the heuristics of what we're seeing there looking for anomalies. And once we find that we have systems that automatically uh work backwards from those anomalies to uh to figure out like what, what data or what indicators in there is interesting. Um and we use that to build features and protect our customers.

Uh the last point is, is important we're gonna touch a lot on this. In the talk, there's a distinction between mitigating attacks and disrupting. Uh mitigating is placing a, a filter, a rule that is dropping traffic that you don't want disrupting is, is efforts that we do to um prevent bad actors from easily, just ret attacking us. And that involves taking down infrastructure and kind of, you know, trying to really get deep and go beyond our network to figure out who is who is attacking us. So we're gonna talk more about that to start off.

This is a scale that we work at. Our team is situated on top of this network, right? It's one team looking across uh this massive scaled network, uh looking at thousands of DDoS attacks every day, tons of web traffic uh that goes through our AWS managed rules and all of this is a data point and a source for us on the dial side.

Um it's not going anywhere. Uh actually, we've seen close to a million DDoS events detected and mitigated this year, which actually over the last five years is pretty consistent.

Um however, for the first time this year, application layer events are the majority of what we see for previous years. Uh since DDoS started network layer infrastructure events are um are were the majority. Uh those are like SYN floods, UDP reflection attacks. Uh those those kinds of attacks that are on the lower network layers are what we would typically see and that's well understood today and easily mitigated.

But where all the innovation is, is on the application side, this is the innovation on the bad actor side, which is also forcing us to innovate um at 40% year over year growth. And finally, the application layer has taken over. And this year uh year to date, we're sitting at 56% of all DDoS events involving application layer.

Uh there are many reasons for this. Our team has um worked backwards and try to figure out, you know, motivations for DDoS. And there are numerous reasons. Uh it could be some script kitty writing and trying to take down a gaming server who eventually realizes how fun that is and then ends up years later targeting critical infrastructure depending on uh the resourcing and funding that they've got. But all of this allows us uh to figure out different tactics and techniques that they're using to DDoS us.

One of the biggest trends we've seen is in the use of proxies, right? A proxy is a server that's on the internet. When you send it a request, it will forward on that request to the destination that you've, that you've uh specified.

Um so if you're a target, if you are the target in this diagram, um and you receive uh requests from a proxy, you only, you don't know where those requests originally came from, right? Because the proxy is, is hiding the original uh source and there are thousands of proxies on the internet. Most of them according to our knowledge are just open proxies. That means they're not authentic. You don't need authentication to use it. Anybody can do it if you're a student and you're trying to get off your student network because you wanna browse something that you're not allowed to. This is a great way to do that. You use an open proxy.

So there's legitimate traffic going through these proxies and they are a great way for bad actors to obfuscate their uh compute capacity that's creating these DDoS attacks, those we refer to as proxy drivers. And so our goal is trying to figure out how do we, how do we figure out where this traffic is coming from? How do we understand where those proxy drivers are situated? These this this attack is um this use of open pro open proxies is very different to a botnet.

So I got this picture here just explaining how a botnet works, right? A botnet is uh in the bots section, there is compromised infrastructure, right? So if you leave an instance open on the internet, unsecured, running vulnerable, patched unpatched software, you're um there's a chance that you will uh vulnerability will be exploited and they will install some malware or some software program on that on your instance. And that then becomes part of the botnet. And these bots are then controlled by command and control server kind of like the brain of the network.

Uh and uh so this, this is uh another thing we see, but more of you, you can do uh uh application layer attacks with this. But more often than not, we're seeing these proxy based infrastructures attacking us uh at the application layer.

So one of the interesting things we do is we track DDoS infrastructure. Um no, uh no doubt you've probably tried this. Uh if you think about your application architecture, you probably keep track of the IPs that are um causing pain for you, right? The IPs that you don't want, you maybe got a block list with IPs that you know about.

Um and we do this except we're doing this across the entire AWS network. So we're looking at all the known offenders um particularly in this application layer space that are contributing to DDoS attacks.

So, for, in a, a simple example here, you've got, you know, two clients sources sending uh contributing to three different DDoS attacks. Uh and so we take these ones that are uh the most prevalent offenders in this space and we keep track of them and they become on our bad list and this has proven to be unexpectedly very effective. And I think that's because we're doing this across our entire network.

Um so actually from the millions of DDoS attacks we see in a year uh 56% of those are application layer attacks. So we are like more than half a million attacks. We're tracking around 30,000 individual sources that are constantly hitting us. Right. And I think the takeaway here is that bad. A most bad actors don't have the, the funds just to find new infrastructure and find new IPs.

Um and so surprisingly, to us, this has been uh a successful tactic just to quickly place some mitigation using this, this list. Um and if bad actors are listening to this and they now go change their IPs that's ok, we'll uh we'll catch you.

Um this list actually is available to you. It's super helpful for us protecting the availability of infrastructure, but it's also in a managed rule group in WAF. So if you use WAF on your ALB or your CloudFront distribution or API Gateway, the list is in the AWS managed IP reputation list and it's there for you to use.

Um there are some other rules in there, but the one that's called the IP DDoS list is the one that has this exact intelligence that we've collected from across our network. So please uh use that um for yourself that's available for your uh use.

Uh just to give you an idea of how we've used this. Um we use this list to protect our management console. Uh we've used it. Uh we've deployed this list across all our CloudFront uh distributions and in all of these situations, we're rate limiting and just impacting the ability for large volumes of requests from these known offenders that we know are bad, uh, hitting your endpoints. And a lot of what we see from there is proxy based attacks, right? So immediately your end points and your applications are protected from that.

Um now, uh an example of how we use this for at the retail store. Uh we had this list deployed during the prime big day deal, uh big deals day. And uh we over uh in October, I think it was two days, we had 12 DDoS attacks and all of those attacks were mitigated with uh with 92% efficacy with just this list, right?

So remember like DDoS defense is a multi layered approach, this is just one layer that we placed and for free, essentially you get 92% efficacy. Um and so I, I really encourage you to use that list.

Um if you are using uh for, if you're using Shield Shield Advanced has a feature called Layer 7 Auto Mitigation. And this feature places the list for you as one of the things it does automatically.

So if it detects an anomaly based on the, the deviation from your baseline traffic, it will place this list and your list. Um uh your application is then uh immediately protected with this ip list in there. Now ip s are a great first quick mitigation, especially when you know that the list is a list full of bad offenders who are repeatedly attacking.

Um but again, we also analyze the traffic and deploy an http signature that um ca captures the attack traffic. So that allows us to have two layers of mitigation um automatically in your wife where now an area that i am very passionate about and something that we've only just recently spoken about in the public space is something that we call mad pot.

Um this is uh how we collect global threat intelligence on the aws cloud. It's a sophisticated system that we use, that has uh sensors or honey pots uh that we use for monitoring and uh detecting just and understanding the noise that's out there on the internet. Uh it's a, it's an integral part of how we collect this intelligence because it allows us to see what your applications are exposed to every day, right?

Anybody can create a honey pot. It's, it's not, it's not that difficult. You create a ec2 instance and you just leave it open to the world, you'll see what happens, you'll get a lot of traffic. Um so it's not um it it's helpful to us because we run on these honey pots uh applications and we respond in a way that mimics um vulnerable applications with unpatched versions and that allows us to potentially pick up what bad actors are doing uh to these types of instances out there.

So um just to give you a bit more detail about my pot, we have um more than 10,000 sensors deployed globally around our, our network. And every day we're getting uh more than 100 million potential threat interactions. So, an interaction is somebody who finds a honey pot and interacts, like sends it aaa payload or a packet and tries to do something on that honey pot.

Um and out of this, um every day, we're able to classify around half a million malicious uh attempts to interact with our honeypots. This is crazy. We're collecting all this data and it's allowing us to have this unique global perspective of the threat landscape of the internet, the noise, the bad noise that's on the internet that you have to face day to day.

And so it gives us insight into the trends um and allows us to learn a couple interesting facts about um the threat landscape. For example, three minutes is all you have. Now, what i mean by this is when we deploy uh well, on average, when we deploy a new sensor that's got a new ip. Uh and it's out there on the internet for the first time, you have roughly three minutes before somebody tries to exploit that, that instance, three minutes. It's crazy.

Uh and that's on average. Um and i know we're all here in vegas feeling lucky. And if you do have, if this does happen to you and it only takes them three minutes to find you, then you, you strike the jackpot. Um because a lot of what we see actually happens quite, quite quickly. And in under a minute, i remember this is an average across a massive data set.

Uh uh so it's, um, i guess you're lucky if you only hit three minutes. But what this does and i'm not trying to create fear here, but i do think that this data helps us reinforce the idea that we have to raise the bar in the systems that we design. We have to think about um the way that our systems are analyzing the logs that we've got and trying to figure out if something bad has happened or if something has been compromised, i don't think that um i don't think that a weekly or daily scan of your logs and infrastructure is enough is what the slide is saying.

Uh we, and um yeah, so i wanted to share that data point with you because i think it's uh it's important for us. Uh we use this data from mmo to, to raise the bar of, of our internal security systems. Um and it's, it's good for you to understand that you can't just sit on your hands uh because there is all of this out there.

So mad pots, not just a honey pot system, right? It's not just like the value that we get is not just by running a honeypot. Anybody can do that. You, as i said, you can build your own thing on ec2. The real, the real gold here is how we extract threat indicators and threat data and insights from the interactions.

Um and so man, part is integrated into our network. Now, we are extracting uh data on the exploit attempts and figuring out, hey, these are signatures that are now commonly being used by bad actors. Uh and we're figuring out these are the types of malware that they're using new versions of malware uh that we are able to download, we're able to reverse engineer, we're able to detonate and collect um insights from the network activity.

This kind of gear this mechanism is powering a lot of what you see and what you are benefiting from in um in guard duty in w in shield. Uh and these are our external services internally. We use this to raise the bar of our own um security posture of our network.

And this allows us to um hold on to that commitment of ours that security is a job zero. Um since its roll out, uh this technology has become a constant part of our uh cybersecurity strategy. And uh we've, we've been running this for years but recently have only just been speaking about it in the news.

Um so there was this article that came out in october, uh a w stirs the mad pot busting bot bo, busting bod baddies. Ah um and uh mark ryland is one of our execs here. He's been helping us work through this. Uh he's done a great job in um allowing the public to know about the security controls that we've built.

Just some examples here. So uh in may uh this, this year, uh there was a nation state uh group that uh a hacking group that's called volt typhoon that allegedly planted uh malware and spyware across uh us critical infrastructure. And by using mad pot, we were able to learn about their tactics. We were able to identify some information on them and allow and, and work backwards from that to uh figure out um customers that may have been in the crosshairs of this group.

Um and we, we alerted the article talks about this, but we alerted those customers and we were able to work with them as well as providing this data to federal agencies that uh who are able to, you know, go above and beyond what aws is entitled to do.

Um similarly, there was another group called sandworm. Uh sand worm is a, is a, a group tied to russia and they tried to uh exploit uh what they thought was a security compliance on our network uh which turned out to actually be one of our honey pots. Uh and so just from that interaction, we captured ip addresses and um signatures of their attack.

Uh techniques. And by taking a step back and looking across our aws network, we could identify again customers who have put, who are in the crosshairs and we could reach out and enable them to uh improve their defenses.

Um so these are just two examples of how we've been able to use this inside and, and to protect aws customers. Now, a big part of what we do is we use this data to work and co-operate with outside uh internet players.

Um so what this action looks like is what we refer to as a takedown. Um when we identify that there are botnets or c twos or proxies or proxy drivers out there, any type of infrastructure that is uh continuously causing harm. Uh we will reach out and say, here's all the information, please. Will you take this down, right?

This is our effort to cleaning up the internet, right? Because we're learning about this infrastructure from mad pot and we're asking them to remove it from the internet, which means that everybody on the internet has a better time. Ultimately, you might think, ok, but they're just gonna rebuild it.

Sure they will, they will rebuild it. Um but this is our, this is us driving up the cost, this is us disrupting them. If we don't do this, then they can just re execute that 50 lines of python and initiate an attack um by disrupting them, they've got to go and build things uh for example, a botnet typically has some redundancy built into it, right?

Um the c if you think about those c twos, the c twos are the brain of the network. Um and the c twos, if you take out ac two, the whole network dies because you can't control it. So what bad actors do is they have a domain name that resolves to many c twos. And so by taking down the domain and having the domain registrar deregister that domain, you now have to go and rein like reupdate all your bots, your, your whole network to point to a different domain, right?

And that's time, that's money, that's down time for their, their ddos as a service. So it costs them that loss of revenue. And all of this means that um when bad actors know that aws is doing this, then they hopefully are gonna think twice about targeting anybody on our network. And that's, that's our goal as i mentioned in, in the beginning.

So a way that we get intelligence on the proxies is by this. We run these honey pots that pretend to be proxies, right? They're not, they're not actually forwarding on any traffic, they're just responding as if they were a proxy. Uh and so this by having this here, our proxy honeypots end up on lists that are out there on the internet.

They end up in tools that are used free tools that are um abused by people to execute ddos attacks. And that's great. Um we're not making the world a worst place because we're not forwarding on the traffic, but we have all the benefits of getting the insight into the compute capacity um on the proxy drivers.

And so we can then initiate takedown requests for proxy drivers. And here is just some information on the, these are basically our partners, right? These are networks out there on the internet where we are tracking proxy drivers.

Um and why by uh kind of handing over the details to them, we're achieving two things um or three things, actually, we're cleaning up the internet, right? Anybody on the internet is benefiting from us because we're taking down malicious infrastructure that's not just targeting aws but uh everywhere.

So the second benefit is that these networks are saving money because their resources, uh these resources and these accounts that are being used by bad actors are, are causing costs and they're able to clean up their own backyard uh and take action based on the data we provide them.

And then thirdly they're able to build mechanisms, right? Once we, by educating them on this is the type of traffic, this is how you find these things, they're able to build mechanisms that allow them to detect this um in their own, in their own way. So we're empowering them to uh clean up their backyard even uh in a in a more efficient and effective way.

So we work with these guys um to achieve that. Here's some examples. Uh so here's one example, this was uh i think this was digital ocean we worked with and they responded to us saying, hey, um we terminated one user, not quite sure what that means. All i know is that they, they took some action which was great.

Um they terminated something and that's excellent. This is what we wanna see, right? We told them about some data and they went off and did it for us. Here's another example uh where we shared some information on them, a bunch of ip addresses and they uh uh they were behind open proxies. So these were the proxy drivers.

Um and again, uh we got some response here saying, you know, thanks for sharing that we've wiped these accounts. This is, this is happening day in and day out. We're, we're partnering with these accounts to do that. And I think that um if there's one way that we can combat this type of malicious activity on the internet is through this collaboration between uh what we're learning and kind of these wider networks.

Now for my favorite, favorite thing, um this is a uh desensitive. Oh, this is a sensitized version of what we found uh on a telegram channel. Um it was a win for us when we saw this. Uh this was some bad guy complaining about how good amazon is uh at tracking botnets tracking open proxies and now we're even tracking spoof traffic, which is true. So, the more i see of this, the more i know that our team is doing a great job of creating uh mechanisms that are uh resulting in a w aws being an unattractive target.

Um and so i, i uh take pride pride in the fact uh that we're able to achieve this mission and over the for years to come, we're gonna be continuing this mission uh to make aws sound attractive target because as i said, i think by doing this, any workload that's running on our network is safer. So i'm now gonna hand over to ryan. He's gonna talk about this other perspective uh from the guard duty.

Thank you very much. Um for those of you not familiar with guard duty, guard duty is a threat detection service that uh customers can enable on their accounts. And we're, we're looking for compromised users, compromised uh instances, any indication that something within the, the customer's account has uh been suffered some sort of a compromise whether it be at the application layer or if it's something with a credential and we have a pretty wide visibility as well.

So we're used by about 90% of the top 2000 customers. We help protect millions of different aws accounts. And in average, any moment, we have over 500 million running ec2 instances that we're helping to protect and, and millions and of s3 buckets as well. So it gives us a pretty good set of visibility and we look at multiple different log sources.

So, uh when we first launched guard duty six years ago at re invent, uh we were looking at ec2 flow logs, dns, logs and cloud trail logs. So that gave us a perspective of the network activity of the instances as well as the user interaction with our control plan through cloud trail. And those, those first two are really what we're gonna look at deeper here today uh as it applies to um looking at the trends with the uh compromised infrastructure as well as some of the other ones like eks audit logs.

After we did the initial launch, we, we, we got requests from customers to kind of expand the aperture and look at other log sources. So we added data events for s3. So we can understand how users were interacting with uh s3 at the, at the data plane. And then as we saw more and more customers moving into containerized workloads, we saw that as an area that needed to get some good threat detection. So we started inspecting eks audit logs and that's the audit logs to the ecs of the kubernetes control plane itself.

Uh and then this year, we added three additional data sources. So the first one was looking for anomalous behavior in aurora, um r ds aurora login events. Uh we added lambda flow log events. And we'll, we'll look at some examples of why here uh later on. And then earlier in uh march, uh we added eks runtime protection that gave us deeper visibility into what was actually happening at a process layer from getting kernel events.

And then this week, we also extended that to include ecs as well as fargate and ec2 as well. So now we have visibility across all the possible compute platforms within uh aws. And when we get these logs, we apply threat intelligence and we, we look at threat intelligence in several different ways.

So we, we partner with some third parties for feeds, we generate some of our own and we get a lot of great threat intelligence from the, the mad pot team as well. Uh we also employ machine learning to help identify anomalous behavior within your individual account. Uh either at your account level or at the user level within your account. When we start looking at things like eks, we're looking for different behavior or anomalous behavior for the cluster level.

Uh and then we also have the ability to do malware detection as well. So when we notice a signal, we notice something from an instance, for example, that is um behaving in a way that is consistent with an uh a malware infection. Customers can enable the ability for us to snapshot that volume, run a malware scan and then capture the malware that's sitting on that uh instance and, and help the customer find where that infection is.

Uh we have 100 and 64 different fighting types. Obviously, we will only look at a few of them here today. Uh but customers can consume these through multiple different ways. So we have a console obviously, but also we send them out of eventbridge, which is probably the most common way to consume those uh that allows you to make automated response.

So you can trigger a lambda function, you can send it some of our partner tools and you can also um investigate those using amazon detective as well. So as i mentioned, we want to look at a few examples of um the data that we're seeing from our vast viewpoint of our customers of what is the activity that we're seeing when uh their workloads are compromised based on the types of findings that we're generating for them.

So by and large, the most prevalent action that we see when an instance gets compromised is uh crypto mining. So it's important to note here too because when you start looking at these percentages, um a lot of times we don't just see one single thing. So crypto mining is the most prevalent. But a lot of times we'll see the same thread actor use multiple uh different tactics.

So they might have it doing crypto mining while it's still sitting there waiting for commands to do nile of service on the when the c two is ready for that. Um some cases also, depending on how it was compromised, we might see one thread actor go and start doing uh crypto mining and another one will make it part of a botnet to do d oscillator or do something else.

So um it's not only um just crypto mining, but this is our most commonly uh seen post compromised activity and it's actually almost very useful. This is a, a very good and easy detect signal when crypto mining takes place.

Um some of the data that we look at too to help enrich our ability to detect. This is what are the ports these crypto miners are running on. So when you're running a crypto miner, it's communicating to uh a mining pool. So that way there, the adversary can get access to the crypto coins that it's mining using your infrastructure.

The most prevalent one we see is moro. So xmr is very, very popular and we see very um strong data around the types of ports that they're using. So 32332222, they use a lot of ports that are pretty easy to remember, but also don't generally have other uh well known or, or typically good purposes.

One of the other things that we see is where are these uh mining pools running and to be fair. It's nothing wrong with running a mining pool. It's perfectly legitimate. So as long as you also are paying for and, and own the infrastructure that's doing the mining, which isn't the case here.

Um but we see also pretty strong concentration of the i sps where the, the crypto pools are, are located. And this gives us some good signal too. So we, we have our threat intelligence, we have some machine learning that can help us identify patterns of behavior that look and appear to be uh crypto mining.

But if we see activity and flow logs that is on some of these well known ports going to some of these well known i sps, it can also help be a signal for us if we don't already know about that mining pool because they're pretty easy to move around. And then the bad guys will set up nuance.

Uh we can then go and automate a system to probe that test if it is an actual mining pool and then update our threat intelligence and make the better uh detections going forward from a customer perspective. If you have guarded dion and you are start to do crypto mining, there's a whole set of different findings that you're gonna start to get from us, right. That'll, that'll give you signal that your instance is compromised and that you're now performing crypto mining. And we get them from multiple different sources depending on um how the crypto miner was configured and how they were communicating with the mining pool.

So flow logs which is um very, you know, connection events. So you're looking at the source ip the the source port, destination ip, destination port and protocol. Um if the ip address of the remote side is a known crypto mining pool will give you uh a cryptocurrency finding from flow los uh that network port unusual.

Again, very useful for us getting that signal around. Hey, we actually didn't know about maybe this particular mining pool ip, but we're seeing that this instance and the way this finding works is we track for every instance in every group of instances, what are the typical ports that you make in outbound connections on so that you actually initiate an outbound connection?

And what are the i sps that you typically communicate with? Um so if all of a sudden an instance has never communicated outbound on port 3333, and it's going to one of those i sps that we just looked at, it's a pretty good signal that it's probably a mining pool. And so that's like another good signal for this.

Uh depending on how the mining uh pool was defined, it might use a dns name instead of an ip. They'll do this often because again ip s are really easy to change. They're really easy to take down. Uh so they use a dns uh host name and then that way it can resolve to get whatever the, the most current ip is.

So from a dns log perspective, you, we can get a, a bitcoin tool finding based on the domain names being queried. And the, and the one below that is another one of our machine learning algorithms where we look at the dn ss requests across every aws customer globally.

And every day, we build a reputational graph um based on all of that data. And it allows us to predict potentially malicious and also known benign uh domains. And so we feed that back into our threat intelligence. And we have a category for that where we're saying based on some of the activity that we're seeing, we believe that there's a likelihood that this is a crypto domain, even if it's not presently on uh known threat lists.

And what we tend to see is we're finding a lot of malicious domains, both cryptocurrency and, and other types of uh bad domains, days to weeks before they end up on unknown threat threat list for customers that are using the runtime protection, so on for either eks or ecs and fargate. And now e two, you'll get all that same uh information. But instead of just getting the ip in the domain, we'll also be able to give you a lot of really rich context about the process itself

"So the process ID, the path to the miner who launched it, what its full process lineage was. So that really helps with the investigation because now, you know exactly where to go.

And then lastly, uh like I said, we have these for um lambda as well. And this was really in response that we started seeing in some cases where accounts have been compromised in order to hide the crypto mining activity rather than spinning up new instances, right? Some of the adversaries started to realize that customers would start to notice if all of a sudden you launched 400 large instances in their account to do crypto mining.

Um so they would try to hide it in lambda. So now we will also process and analyze flow logs coming out of lambda functions and identify crypto mining there as well.

So we did uh a real world test uh in one of our environments just to see uh kind of to that point of time matters like what, what happens if we exposed to something. So we took an eks cluster um configured it and we made a, a mis configuration and unfortunately, this is a mis configuration that we see uh happen in real life on real customer environments and, and that is to uh allow anonymous access.

So in, in ecs and kubernetes, you can bind the, the system unauthenticated or system anonymous user to a cluster role. Um and that means anyone can use that api without providing any authentication.

So that happens at t 09 minutes later. Uh we saw that the uh that was being probed by just generic internet scanners. So just a generic scanner came by and identified that it was probed. And then 45 minutes after that, we saw very specific probing.

So actually querying some of the specific kubernetes api s to see what it could do essentially um on that exposed to. So sometimes people will bind an anonymous role to some less important api s. And in this case, we, we kind of gave them everything.

Um and then four days later, we we saw them actually start to use this information. So clearly, these were automated scanners, they are collecting information out there. And then four days later, we actually saw the first privileged action that started taking place.

So they started using the um the secrets api and kubernetes to retrieve some secrets off the host. They did a privilege escalation by launching a privileged container on the host and then started launching new pods that were um using uh re repos that had a crypto miners in them to perform crypto mining.

And that all happened about 10 hours after they started doing the privileged actions from this type of activity we would have we generated for ourselves. And we did this a whole bevy of findings that you actually kind of walk you through an entire attack chain.

So that initial misfigure generated a finding for us saying you have done, you have made this mistake, right? You have exposed your comin cluster in such a way that anonymous uh access has been granted when that first kernes specific scanner comes along and it starts probing those api s.

Like I said, that generally start off doing some get operations just to see what it is they can do. We start generating discovery findings. So we're kind of walking through the attack chain here when they started requesting secrets.

Now, that's more of an impactful one. So you'll get a credential of access. So this was a successful anonymous access to credentials and then privilege escalation when we saw the creation of the privilege pod.

And then uh lastly, the impact was the creation of new workloads. So the new pods that were being added by an anonymous uh uh user and then with the runtime agent, of course, as soon as the crypto miner was started, we had a finding that there was an executable that was launched that it was a known crypto miner followed by the dns request to the mining pool, followed by the ip traffic as the miner began to uh communicate with the mining pool.

The the second most prevalent action we see is using the instances as a launch platform for for spreading further intrusions. So we see this in about 50% of the workloads that we see. And again, a lot of times this is in concert with crypto mining and and this happens because a lot of times the way the adversary gets in is through some sort of an application vulnerability.

And so what we'll see is they start looking for other in ip s on the internet that have the same vulnerability so it can attack them and spread. And so we see things that we call port sweeps. So a port sweep would be where i'm picking a very specific port.

So i'm looking for a post gra sql database, but i'm gonna go look for post graph sql databases across a very large number of ip s. We also see port scans. So a lot of times this is more looking for lateral movement where they'll start scanning all the ip s inside the vpc to understand what other infrastructure is running in there, understand what ports those have open.

So they can figure out if there's a way where they can find lateral movement onto other hosts, usually with the, the hope of uh increasing their privilege level and then outbound attacks for brute force are very common as well.

So the ports that we see for port sweep. So this is uh some data around when we see these infections and this changes over time, usually in relation to what is the more common or recent uh application vulnerability that we're seeing, adversaries start to target.

So when we got this data post, it was very prevalent, uh we almost always see a lot of outbound ssh. So people are constantly trying to use compromised infrastructure to run brute force attacks against ssh instances that are accepting traffic from the internet.

Um and then the windows management interface. So wm i, we see a lot of those too for brute force attacks, a lot of times either brute force attacks, but also looking for um unpatched windows servers as well to, to use some, some older vulnerabilities.

And then the other ones that we see on there is interesting is like 8088 which is often used as a proxy port. So again, looking to find uh open proxies that they can know about and then start using um for, for these, we have a whole set of findings in this that we get from this data from.

And that is again, like port sweep, like i mentioned where we're looking for a whole lot of ip addresses that a single instance is communicating with and we look at and understand what, what is the response rate.

So it's one thing to go talk to a lot of ip s. But if i'm talking to a very large number of ip s on the same port and a very small percentage of them are answering, it's, it's a good chance i'm just shooting blind across the internet and i'm not actually purposely trying to connect to my or known uh server in that case and then outbound brute force for ssh.

And also for rdp, very good reasons to keep your security groups tight and not leave these on the internet. They are constantly under attack, both from compromised infrastructure and aws, but also from the wider internet.

And then like i mentioned the the windows um management interface brute force as well, denial of service bots. We also see these around around 20% of the attacks that we see. And it's important to note too.

So when we, when we see the denial of service, we're looking at the traffic flows and again, we're, we're gauging how, what is the number of packets out versus packet in? What's the response ratio? How many resets are we seeing? It helps us identify likely outbound denial service.

So this is very different than shield, which is looking for protecting our customers from denial of service. This is more where your infrastructure has been compromised, you've been added as part of a, a botnet and now they're using that infrastructure to, to try to send these attacks.

One thing that we also do for just from an aws wide perspective is when we find those c twos, we can null route them that tries to break this chain as well. So even if your instance does try to communicate to that c two, it oftentimes will fail because if it's ac two, we know about we've null routed it and they won't be able to communicate uh the, the finding types that we, we have for uh this are very simple denial service and we break it down by what, what type of port or protocol, what kind of protocol was used?

So dns based tcp based udp, udp and dns ports were just unusual uh protocols in general. 01 of the last uh kind of trends that we, we tend to see and we're seeing a lot more of actually is as we start to add more and more detective capabilities. Adversaries start to learn what's going to get them caught.

And um you know, like if i'm deploying some sort of a payload that has a minor on there, for example, i need to communicate to a mining pool. So i want to use a well known one. It's gonna have a lot of ip s that'll change over time. Having my first stage include a dns name is very convenient.

That means as the ip s change, the, the dns will just pick up new ones. I don't have to update my, my payload. Well, these guys have gotten a little wise and they realize that customers and even with guard duty, we're looking at these dns requests and we're monitoring those to understand whether or not the dns was going to uh uh no malicious domain.

So they'll try to evade that by not using the preconfigured dns name. So when you configure your vpc, you define your dh dp option set, you tell all of your instances what dns server you're gonna use most often. That's gonna be the vpc provided dns.

Um, customers also can use their own as well, which is perfectly fine. And when they're using their own, they'll probably have something that's looking at those logs too. So they realize that guard duty is now watching all of these dns requests and can flag this activity.

So in order to avoid that, what you'll usually see on the first stage that's downloading the next stage is they'll override the connection by saying to the operating system, use this very specific publicly well known, commonly used uh public dns server.

And then when that happens, of course, there's no log in in route 53 for us to, to see because the dns request went to another, another dns server.

Um what we're also seeing increasingly too is more sophisticated when starting to use uh dns over http and dns over tls as well. So in the latter and even encrypting those requests uh over tls.

So in response to that, we, we created some new findings to help detect this and, and these work by profiling the dns behavior within an account. So, you know, it's perfectly legitimate to use any dns server that you'd like. And so that's fine.

What we do is we look at your account, we look at your instances and we understand, do you normally always use our vpc provided dns or do you normally always use your particular ip address that's associated with your windows active directory server, for example.

And over time we learn about that and then if all of a sudden we see you making a request to uh known doh server or uh publicly known um dns server. That's not one that we see typically using your account. It's unusual.

Um it's probably not what, that's what's configured within the infrastructure and thus it's an override at the operating system level and likely uh someone trying to evade detection uh by, by doing so.

And that's our talk for today. I want to thank everyone for coming and i do hope that you do uh complete a survey, please for us. Thanks."