Driving precision: Creating a tech revolution at PGA Tour events

Ken: My name is Ken Lovell. I and you can, I'll read. So that's, that's my title. I'm responsible for golf technology at the PGA Tour, which is a very random title. I think it's probably uh that there's not a big career path for golf technologists. Uh you know, not with that specific name anyway, but basically, I'm responsible for collecting all the data that's possible to get off the golf course during a professional golf event in the United States and in other countries as well. Now that I think about it. So essentially anything that it takes for us to know what's going on and to plot the ball in three dimensions. So anyone can understand what's happening at any time and we'll go into a little bit more about what that means. But that, that's the big picture.

Ken: So basically this is i'm going to talk for a second about the way that we have done it for the past 20 plus years. And then as we get through the discussion, we'll talk a little bit more, more about what's coming. So there's a, there's a tease for staying for the second half of the conversation. Um but the way that we have done this. We, we, we built something in about 20 a little over 20 years ago called ShotLink and ShotLink was the beginning of going from just understanding what the score is, which I mean, the score is the story, but also going further than that and understanding how the ball got, where it got, what happened with it.

The way that we used to do that or that up until very recently is, um, I'll just kind of run you through a little bit of a timeline on it. Um, we would show up at a golf course about 10 days before the end of the tournament, we'd bring five trucks. We would start by dropping about 15 to 25 miles of cable on the ground. And that was a composite fiber and power solution. So it had copper and fiber in the same cable. So the reason I bring that up is because it was really, really heavy and we would bury that. We ran it in a 63 hole loops all the way around to every, around the golf course. And the reason we did that is for redundancy and allow us to so that we wouldn't have, if anything, if any fiber got cut anywhere, we could just loop it back and we would be ok. And by the way, fiber does get cut and it happens a lot when you, this isn't regular fiber, right? I mean, this is like thick fiber. This is, it's basically, it's instead of standard t fiber, which is sort of the, you know, circum ballpoint pen. This is about your thumb. It's, it's very heavy and it takes a lot of it to get around that far.

So we'd start by putting that on the ground, running it through the entire golf course. And then from there we'd start putting out what we call D-Mars, this custom designed cases that had in them network switches and a whisper generator. And then, and a propane solution that went along with that, we would put one of those at every T and one at every green. And on a long par five, there would be one or sometimes two in the fairway depending on the length, that's the power and the network solution to get it all down.

Then from there, we would put up a wireless network and run the wireless throughout the entire golf course. This is across every fairway on the entire course. And it's the very best wi-fi that you will never ever be and never ever be able to touch. Um, we, we like you, but you can't see it because that's where we collect all our data. So we, we protect that a little bit and also not to interfere with the purposes of why you're actually pulling the data off.

Ken: Exactly. And then once we would put that down on the ground, we would start adding sensors. So the sensors that we used to use included, there was a handheld, there were, there were tablets that were in the fairway and another one around the green, a laser that was in the fairway. On top of that, we had one radar that was sitting behind the T box and then we had an array of three cameras that were set around every green. Those three cameras were, this is the very best technology of seven years ago, which I know doesn't sound like very long. Uh it was actually still very good. Um these were 20 frame per second, black and white, high definition cameras designed for high contrast to pull a golf ball out of a dark surface or light one depending on how, where the sun was. Uh and we would, we would basically spend it took us from uh Wednesday of advance week through Tuesday to set all that up and get it ready to go.

Ken: Yeah, so actually one thing I just want to clarify pretty quickly and especially given where we are, this is of interest. Um one of the things that happened, I mentioned there were five trucks we also put out, um, we put video boards all over the golf course. So any time you see those cool boards that show you what's going on, that was our responsibility to put those out. We had to put power for those network, those and make sure that they get data. Those are all automated and have unique data coming to them. And on top of that, the other thing i was going to mention is that we would have two trucks that became a production studio to trap errors as they came off the golf course and make sure they were correct as quickly as possible in those was a giant set of servers that was collecting all that video data. And then we would take that and run it off the golf course from there. So that is something that's, that's the way we used to do it.

Why we do that, which I think was your question. So essentially this feeds everything it is found to everything that happens at a golf tournament. If you don't have data, you don't have a story to tell. So what we realized is that you can point a camera at something. But unless you know why a guy is taking a shot, it doesn't really mean much. And we built a set of tools that, well, essentially we built a distribution infrastructure. So you take all that information, you put it into a set of distribution infrastructure that's used for both broadcast, for digital, for i for our gambling partners. It turns out people like that to, to, to do that with sports and stuff. So we, we serve all those different constituents and we do it with different, different tools that about custom for each of them.

Let me give you a quick example of why that's important when you watch a golf tournament and i'm going, i'm going a little bit of a spoiler alert here. Not every shot is live because at any given second, once we start playing golf on thursday morning, there are an average of 14 balls in the air, all of them are moving. The question is which one do you watch? And which one are you looking at at a given moment? So a lot of things happen on tape as you tell the story of what's happening in the event. And you've got to think about this and i'm going to say this with no disrespect whatsoever for our friends in other sports. And i mean that it is very well, it's very difficult to put a good sports broadcast together. But if you have a single field and you have one ball and you take one camera and point it at the ball, you will know who won the game at the end of the game. Right. This is a case where you're dealing with 200 acres and 18 playing fields and all of them are happening simultaneously, but at different rates. So it is very much a, a parallel processed sport, I guess so to speak, right?

So you've got to figure out how to do that when you watch on television, a lot of those shots are on tape. That means that when you see the score at the bottom, it has to be accurate to what's on the screen. But we're also collecting data in the second that it happens wherever it's happening. So we had to build some tools that would take that data and line those two things up so that it's time shifted appropriately and, and get things in the right place.

So we built something called the broadcast interface tool. And because we're really good at naming things, we called it the BRINT and that's what we would use. And we ran a line directly from our trucks into the broadcast so that we would be able to put that in and have it with as little latency as possible. And then we put someone in that truck and called them a, a graphics coordinator and wired that directly into the K run machine so that we would have that and make those graphics as available as high availability as possible. And in all honesty, a lot of the graphics, I'd say probably between around 70% of those you never see because in the moment it just doesn't make sense.

So we're producing as many interesting stories as we can that you might not have a chance to look at. It's not because we don't like you. We're just moving very quickly through a live sporting event.

Jefferson: When you look at that, Ken, you talked about a couple of things, one is broadcast media, sports betting. So you talked about sports betting. Is this something that is gaining popularity or has this always been there? And it's just had a heavy requirement on you and gulf tech?

Ken: Yeah. So there was a court case you may have heard about not all that long ago. So sports betting is gaining a lot of traction in the United States. It is a state by state phenomenon and we are here to serve our fans and, and so we have, we have entered into it. It means an entirely new and very different way of collecting. You have to think about data completely differently from, I mean, the biggest single thing you've got two, you've got two problems, accuracy and latency. Right

It has to be there as quickly as possible. It has to be as correct as possible because you are competing against people's eyes. Arbitrage is the real thing. And if someone knows that the ball went in the hole before we tell the operator it did, there's a big opportunity to lose a lot of money. So we have to handle both of those things.

Our SLA for what we deliver in terms of gaming is that we have to deliver every piece of information off the golf course about every shot inside seven seconds, including public internet and response time. And so it has to get from wherever we are to London that quickly and back. And that's just the standard that we have that we're up against.

So when you look at this picture, you describe T box, there's a radar solution, there's a camera on the fairway dependent upon part five, there could be two cameras and then there's another radar on the actual putting green. When you look at kind of the size of these things, how can you describe, how big are they? Can you describe where they're put up?

So basically, you're seeing a little, just a couple of pictures of the different, different pieces here. One of the things that happened - so this is actually one of the things that we've upgraded - is we put a second gen radar unit on here. This is the simplest way to describe it. This is a missile tracking radar unit. So it's gone from a consumer grade unit to something that is capable of much more.

And you'll notice this, we're mixing a little bit from what's coming or what now is on the golf course, we doubled the amount of radar that's on the golf course. So we can paint a picture both from the T and then from the green inbound and you can get a feel for that. It's interesting. Physics is fun.

A consumer graded unit that you may have seen - you'll see this when a lot of our players are warming up on the range, these bright orange units that are behind them, they're about this big, it's a square, flat surface like this. This is the literally the square of that. So go you know, that much bigger than it is. And they also include cameras for their use for calibrating the device.

And they, they're heavy, they weigh about 30-40 pounds. And I, I asked them if they could make them, I, I, why it was so heavy and they said because it's so accurate. So I asked them to make them twice as heavy and they didn't think I was funny.

But that's basically what it's come to and what we put on the golf course now. So, just for perspective - our friend Greg who worked on this project is standing in front of one of the radar solutions, but it's actually small. It's not that big. It's just the way the camera angle, I took the camera angle, but it actually is, I mean, it's heavy but it's, it's not that big. It's like it's literally about this able to track a ball on angle all the way to.

Well. So, actually, let me, let me, I'll go into that a bit more. These, they're fairly impressive units. The reason that we use them and what we like about them is they have the ability from the second - so take the one behind the T box - from the second that the ball is hit within a second to a second and a half, we can measure pretty much every characteristics of every characteristic of the ball. And that includes everything down to the, the axis of rotation. So what axis the ball is spinning at and the, and its spin rate, which then allows us to do with some fun math and figure out where the ball is going to go before it actually gets there.

If you think about it, the inbound is even more impressive. So you've got a radar unit that is sitting behind a green and it's designed to pick a ball, something out of the air, the size of a golf ball, right? Identify where it came from, who it belonged to and where it's going to come with all those same characteristics. So it is a fairly complex problem. But what that allows us to do is then use that information while the ball is still in the air and do a lot of and and present that in a way that makes it more interesting what's happening, it gives context of what's happening but also make predictions about what is likely to happen.

So, Ken, you talked about what's the data coming off the ball? You talked about media using it to track it. You talked about betting. There's another thing you mentioned scoring. So when it comes to scoring again, when Ken talks about this has to be in orchestration. Every time a player is walking up on the green, you have to actually match the name of this is the person hitting the ball. This is where their ball is that they're shot. How are you doing that with AI and cameras because this is the middle part of that section.

So we just talked about the T box. Now you've gone to the fairway part of it with cameras, right. So now again, historically, this isn't the newer things, but what we have done in the past and I'll come to the new stuff here in a minute. But what we've done in the past is that we put three cameras around a green. And what that allowed us to do is so, again, if radar is very good at tracking objects in the air, not very good at tracking them on the ground.

So once you get to a green, that's where the ball is rolling, right, just a different facet of the game, we, we needed a different solution to be able to figure out where the ball was going. We again, this is about seven years ago, started using ML to build algorithms to track the ball as it moved across the ground and then give us ball in motion data.

So one of the things that's interesting about this is that when ShotLink came into existence, we for the first time were able to put coordinates with shots. Ok. So for example, you know, a player hits a ball and then it goes to a certain spot and someone shoots it with a laser and we now have an XYZ coordinate for the first time. That's what we have built this off of for a very long time that has allowed us to create an entirely new set of statistics around golf. And what it means.

But that is also just how the ball where the ball started and finished. Not necessarily anything about how it got there. Once we started putting cameras on the greens, we started playing with being able to look at how that ball moved around the green and see the characteristics of the ball in motion, which was an entirely new set of information.

And we've been working with this for a while now trying to, to get it accurate and consistent enough that we could report on it because it is a very challenging problem to, to track the ball that way and figure out which object actually is the ball when you're looking at a green.

So when you actually have these cameras, it's doing a lot of things. It's separating the background and anything moving from the ball. It's having to actually understand which player and where the ball is. And more importantly, is the ball in a sand trap? Is it on the fairway or is it in the rough because that's all going to sports betting?

Well, and well, it goes to everybody. But, and even then one thing I left out in the process is a couple of weeks before we can show up to the golf tournament. We spend a great deal of time and effort mapping the golf course.

So one of the things that we have to do is create the playing surface itself. And when I say that, I mean, create it virtually. We, we obviously are not, this isn't the team that is going out and, and setting up the golf course. There are some people that are agricultural wizards that do that. I don't understand their jobs, but it's cool.

But what we do is we will map that down to a very minute level we use at this point, we're using LiDAR photogrammetry survey tools and things like that so that we can get a very precise look at what the, what the golf course looks like. Once we do that, that allows us to then say, ok, we put zones around everything that's there and you map those zones and that gives us the ability to then derive a lot from where the ball is and where it's going and how it got there.

Got it. So now we talked about T box with the radar, we talked about fairway, the camera, what it's doing now, let's talk about how you've networked all this together. You, you said 15 to 20 miles miles of fiber.

How many events does PGA put on in the United States? It'll be about 40 between 40-43.

Ok. So 43 events where the PGA Tour rolls up before the tournament starts, lays uh maps the course itself because this is what I learned during this whole process. The course is constantly changing. So this is what I learned when we were on this project together that what happens one year they could move an entire water feature somewhere else. Something could have happened and the course has completely changed. And that is the mapping that you have to do every year to each of the courses.

Yeah. Lakes show up in random places which I know is, it is not a sentence I ever thought I'd say. It just happens. Oddly, golf courses get set up, they change trees fall, they go up. But even then we have to map the edges of the green to a point that you can really figure it out because that is how statistics are measured. That is just a function of how it is mowed in a given. Like, and, and just think about it over the course of a year. How much that could move.

Yeah. So, speaking of mowing, when you look at fiber, what's the biggest enemy of a fiber cut?

Yeah. So, you know, we have weed whackers, squirrels, lawnmowers. It's a lot of lawnmowers. One of the things that is a big problem for us is that we have. So we, we did not start laying fiber in the process until after they put the rope line up. If you've ever been to a golf tournament. One thing you'll notice is that there's a whole bunch of stakes and ropes that are put out to sort of, it's, it's the lines that you can stand in and out of if you're if you're a fan, we run around the greens, we bury it anything where it crosses around a green or where someone crosses it, we would bury the fiber

But along the rope line, we just, we lay it on top. And the reason that we do it along the rope line is because they don't mow the rope line. Literally, that's what we wait for because it just happens a lot that, that, that we would lose that. So if you look on the left picture, that's the buried fiber that's coming out. The middle picture is the the actual line where crowds can't go past. And you see what Ken just mentioned, the fibers laying on top and on the right is just in between holes laid right on to the the concrete basically. Yeah, it will go, we will move across. If, if we find it, we will go through the woods on top things along those lines. If we have a fairly long run to go and when you look at all the requirements you just talked about and we'll get more into kind of the technical requirements.

Fiber at this point when you design this 7 to 10 years ago was the best way to connect, be resilient, be making sure that you actually have the latency in the performance, right? Yeah, i mean, we, we live in, we, we have a lot of redundancy built in. But fiber, i mean, i know that wireless is super sexy, but fiber is pretty strong. You know, it works, you plug it in and it goes and it's really fast. So it was, it was the best solution that we had to do what we needed to accomplish.

And when you look at kind of this solution on fiber, you've had to design this where it's always changing. Where on the first day, good weather, the second day you've had situations where it's bad weather and things go down. So you're, you're changing on the fly. Yeah. Uh yeah, i'll, i'll give you, um, um, there's, there i got, there are always fun war stories, but this is uh one of those operations where it's, it's definitely in the field when we play outside.

So i, one of my favorite stories is about, it's about the team as much as it's about the tech. Um, and we have a sit, had a situation where we were playing. I want to say it was in Greensboro, i think it was 23 years ago. And on a friday afternoon there was a flood, the rain came, came hard, they chased everybody off the golf course, they closed, it waited for, you know, play stop. And it was bad enough that even though we have, i mean, these cases have, you know, moisture temperature sensors, all the right things. They're hardened to, you know, reasonable levels and it doesn't matter outside is outside and stuff's going to break and we lost about 10 of those cases. And overnight the team stayed there, rebuilt them. We played golf on saturday. It was fine, which, which is fine. That's great. Right. Saturday morning or saturday afternoon, a lightning storm comes and they close the golf course and they ran the people off and, you know, because that's dangerous and we lost and we had 15 of those 4, 15, more of those cases get burned to a crisp by lightning. And overnight we rebuilt those cases, put them back out. And on sunday morning, we played golf. So it's just, and it constantly, it evolves in a number of ways over the course of a year, a week and even just a course of play. I mean, it's really used the word when we first met tactical. It is literally a tactical network. It moves every single week. It's changing every single day by the minute based on the weather of the lawnmower and you have to be able to respond and react to it at all times.

The first time i went into one of the operations trucks, it looked like an operating table. There was equipment everywhere and people had screwdrivers everywhere and they were fixing everything and you were actually testing some of the new tech we'll talk about later today. Yep, that's correct.

So now you described that once the encour stuff is happening between the actual t box to the fairway, to the putting green and how it all works. There's an ops center and you had all cool, you know, nicknames for all these people. But what, what's happening here?

So what's going on is, is the way that we, so you've got to keep in mind that no, we're playing outside under imperfect conditions. No system of measurement is going to be perfect. No matter what you do, it will be imperfect. So, and, and we, for a very long time, were much more concerned with speed than accuracy. And that's because when you're dealing with in a pre gambling world with an entertainment product, we want to make sure that fans had the score as quickly as possible. And if it took us a second to correct it, that was ok.

So we would, you, you measure right, you, you measure it, you check it and then you distribute it right? We would measure, distribute and then verify it. And that was how we did it. What that meant was every time we, we built a massive pile of business rules that said that if something looked like an anomaly, a producer, a scoring producer, so this isn't a tv, producer. It's in a very similar environment where they have all these ability to look at things that are going on and correct them in real time.

So i'll give you an example. Let's say that someone's standing. We have, we have a, oh i forgot this part. We, we used to score this with volunteers. So we'd have to train 300 new people to operate this on a weekly basis. There was a set staff, but also a set of volunteers who were working on it, which can make thursday mornings interesting because they do it once a year. They don't practice all that much.

Has anybody been a volunteer at a pga tour event? Yes. There we go. So, radar or tablet. Very cool. So, so for as, as an example, we'll have a volunteer that is staying there with a, with a tablet in the fairway. That's, that's giving us a preliminary estimation on where the ball is going to go. This is how we used to estimate where the ball went and they look at this, they stab an estimated points. We get the right zone in a tablet and then what happens is right after that another person will shoot that with a laser and get a more precise coordinate. Ok?

One of the things that could happen is that you end up with a zone that comes up and it says the they estimated that this was in the, in the right rough and it shows up in the fairway cool that immediately is surfaced on a producer's screen. They will pull up the information that's available and they either can see it, they can talk on the radio or they can figure another way that, that they can know what, what actually happened and correct it.

Most of our errors are correct, corrected inside of seconds, some take a little longer, depending on, on what happens with that. And that's, that's how we used to deal with, with errors like that. So there was an entire group of people that sat in a truck on the golf course and it looked, it, it was air traffic control. That was how we deal with it. They handled everything from obviously scoring, scoring, tech, scoring changes, scoring problems, volunteer management, motivational, speaking to any time that there was a like if there was a, you know, a problem and, and and a player needed a ruling.

So they get a call from volunteer, i need a ruling on sixth fairway ride and they would get somebody over there to ems problems. It was just the operations center for a lot of that where things came in because we were in all those places and you had talked about these d mark boxes and you talked about kind of where they sat. That's where you had power. That's where you had the fiber, you had the actual mass for the camera.

So this is a picture of one of the d mark boxes, the black box at the bottom. Can you describe what is the white panel right there? So that white panel is actually a test that white panel. We we started playing with the technology to look at other ways to power the system. And that's one of the earlier solar panels that we tested as part of the program. And power is super important because you have to, when you talked about redundancy and resiliency, you're doing the network being redundant coming out of both sides. If someone i asked ken, why do you do that? If somebody mows one side of it, you've got the other side to go past power, you've got redundancy, you've got generators. Plus.

Right. So, so think of it this way, everything on the golf course has a battery in it and let me actually back up on that for a second. I'm going to speak in very gross terms here for a second. But there is big power and little power in what we do on the golf course. Big power is things like video boards and trucks and these are the giant generators that make a lot of noise and have a big transfer switch and because it's required, right? And there's little power and this is what we put around those kind of masts where you've got a camera or a switch or a radar unit or something along those lines, it's solid state. It doesn't take a lot of power, but if it goes down, it's very, very bad. Right?

Every one of those devices generally speaking has a, no, not generally they have a battery in them because we make them have a battery in them. And what happens is that that gives us a buffer of 45 minutes to two hours. We never assume more than that depending on you. Got to keep in mind we get asked this question a lot. Why don't you just put batteries in everything? If i did that, i would dedicate two people to doing nothing but changing batteries all day, every day. This is because they die at weird times. They are all kinds of temperature conditions. They have all kinds of different lives. It is just an imperfect solution.

So what we did was we said, let's use the battery as a redundancy. And then we put, and you can't see it here because we're fairly good at covering things up. But there is a generator that you more or less can't hear that sits in there and then a propane solution where we would drop a propane canister next to it. It's all covered up. That is constantly trickle charging the entire golf course. So if we lose a generator at some location, what that means is that somebody's got 45 minutes to put their track shoes on and get to that place and fix it. And that's how we would operate and the way that we handle the propane, the reason for propane, everybody asked this question number one, it's not great. We try to avoid carrying flammable liquids across america's pristine golf courses as much as we can.

They really don't like it when we spill and and the other reason is because it's containerized. Right? So, just like anything else, it's easier when it's contained. So we call up the, for any of those who know what this is. If your ace hardware store that has the giant cage of blue rhino, you know, you know, kind of big propane canisters that it, we would call them beginning of the week, they'd drop off three cages and we're set and we change those out roughly once a day and it just kept us cooking, but it gives us that redundancy level so that we're always operating and it never goes down.

So there is a fleet of golf carts when people, when i was there and the team was there, they're just driving propane tanks during the tournament, avoiding people just replacing all these things.

Yeah, there's i when, when i bring people out and we show them kind of the behind part. I refer to it as and i can never remember it is they take the red pill or the blue one and it, you start things, seeing things you can't unsee anymore that you start recognizing. Why is that there? What is that for? What is this piece for all of the different infrastructure pieces that are there?

We try to actually make them as invisible as we can. But that's one of the things that you start noticing is guys running around with golf carts that have propane tanks in the back of them everywhere.

So here's a moment are there and we'll ask again at the end. But does anybody have questions about the inner workings? Because the next part we'll go through is a little bit of research around why this is so important when you think about the convergence of this technology, whether it's the network that ken talked about, whether it's the power solution, whether it's the actual compute on the cloud and how you're actually transmitting all this um any questions on kind of the as is state of where we are.

So uh what is the data that you're responsible for it? It's not broadcast video. I'm sorry, july, just interested in the um what data you're tracking, what you're responsible for the aws angle.

Um yes, no. So, so, so here's one thing and you said you differentiate, you said not broadcast video. So here's a fun thing. We don't differentiate now to be super clear. I am not responsible for broadcast cameras. Ok? That is a different thing. But if you think about it and we're going to talk in a second and this will make more sense, but data is just video is data, data is video, it doesn't matter. But so think about it as this, we are responsible for every piece of information about the entire golf course. And what you're going to see here in a few minutes when we talk about it some more is that we're building a virtual representation of the entire course. So start with that map. That is a whole lot of ones and zeros that are representing down to the tree leaf branch and rock. What's happening on that golf course or, or it's it's state then layer on top of that. Every person that is moving on that golf course, every ball that is moving on that golf course and precise coordinates for all the places that they moved, take all that information and start turning it into art more or less. And so anything that becomes like as you build a trace, as you build data around a trace, any of that is the data that's happening.

So the most basic level of data is frankly just the score. We have to take a coordinate and translate it into a number that is a score for a given hole for a given player, but then start going down deeper and deeper and deeper and you can get, you can go, it's a rabbit hole, you can go pretty far and figure out what that means. But that is all the data that we're responsible for.

Yes. Um can i just hear more about the laser? Uh i'm wondering about the laser because in my experience, it's tough to get precise location, especially something as small as a ball. So maybe just how do you make that accurate in the old way? I guess before you get into the new stuff?

Yeah. So we, it's a great question. Actually, we've used lasers for a long time. I mean, if you play golf at all, a laser range finder is a fairly standard piece of equipment. Like anything in life, you can find cheap ones and expensive ones and normally quality follows. Right. So, think about it that way. We actually, we test them and we audit them like crazy. So we would go through and also there is a training element here that goes in where we do a whole lot of training. We have a whole set of video training that's online and then in person training with all the people that operate those on a regular basis.

When we started doing shot link before we put the camera system on the greens, which is pretty accurate. We actually use two different types of lasers. We would put a standard range finder in the fairway because there you've got a little bit more room for error and you can find the ball. It's ok. You also have a walking score that is literally next to the ball so that you can confirm things and do things along those lines. And then around the greens, we used a surveying geometer and and that took an interesting amount of training and trying to do that fast is really hard.

So it just came down to use case right? How you know, you can, you can trade time and money for accuracy at different points. And we would just make that decision based on what was needed.

Oh, yeah, one more here. Hi, i'm, i'm curious, you mentioned uh composite fiber being buried around the greens. Uh when it's sort of in the field of play, how do you do that in a way that doesn't disrupt the sort of pristine condition of the course?

Yeah. Um we have hard conversations. Um so the very short version is that we come along and we spade it and so you don't dig it up, you basically spade lay and come down. We are not, we are not bearing something for permanence. You've got to keep that in mind. This isn't afoot trench. We're, we are getting it out of the way and trying to leave it the way we found it over time. That is something that i, i know this is going to sound pedantic, but we get very good at it with time because you just, you practice and you figure out how to do it and, and we've got to have a relationship with superintendents where they understand it.

The even more big picture answer to your question is um, nobody wants to give up the toys and so they're willing to make a sacrifice to get what they want, right? Everybody wants to be able to follow the gulf better than they have. And so they, they go, ok, if this is what it takes, my alternative is to leave it hanging put it underground or not have it. And so, you know, you, you figure out the best way to do that and then you practice, i mean, literally that's what it comes down to is get better at doing that so that it looks the way we want it to look.

Mhm. Does it, uh, the question is, does augusta let you set up?

Yes. So, augusta for those that don't know, augusta national, that is where the masters has played. Um, it's a lovely golf club in georgia. They, that is so fun fact about professional golf. We actually do not score that golf tournament. They do it themselves. It is one of the few that we don't score and it is one of the few golf tournaments that stays no matter what, at the same place all the time. And they have chosen to invest heavily in putting, they, there is a lot, the amount of the amount of cable underneath of that property is amazing. So they, they, they find ways to push things to bury them and to, and to do a lot of that when you work in the same stadium. So this is, again, that is one of the few exceptions where i envy people that have lines on the ground that are constant. Right. So if you go to a tennis court, you know where the lines are and there is a regulation size augusta, they actually have a little advantage. They're closer to that than, than we are the rest of the time because they own that property. They live on that property. They make it do what they want, they bend it to their will. So it's a slightly different model and very good at what they do back in the corner.

Oh, yes. Yeah, i'll project you, go ahead. What's the question? Is there any trucking on side?

So, just the question is, is there any tracking inside the ball a different way to solve the mouse trap? And number two, do the players get the data or how did they actually use the data?

Yeah. So i get asked the question about the ball a lot. It's a great question. Top golf, their earlier technology was really, well, this was how they built their, the top golf, you know, the, the range piece that they do, they've since moved past it and i'll explain why in a second it's the challenge you have is that they're really, especially on a golf course when we play outside, meaning not at a range, you have two options. Ok. So you've got gps, which means that you've got to have first off, it's not as accurate as you think it is. It's harder to localize than you think it is. And we can go into depth about that. And secondly, that means you've got to have a power source that is too big for the ball. A, a golf ball by the way is you may not realize it but a ridiculously fine tuned piece of equipment like it is something that is very, has very high tolerances and the people that make them expect perfection. I'm not joking. If you go to a title factory, it is something worth seeing. So there's that, that rules that out. The second option largely is rfid, right? So you put an rfid chip in the ball, lower power. It's easy to do at that point though. The problem is i'm going to have to put a sensor network underneath the entire golf course to make it worthwhile. So you sort of run into a problem of measurement versus projection, you know, like broadcasting it and we haven't seen a good solve for that. I'm not saying it won't happen in the future, but between manufacturing it, measuring it just figuring that piece out. It's, it's not something we found it, it is a better solution to look at other ways of measuring and leave and leave that piece alone in the future. Who knows? We rule nothing out? But that's where we're at today.

Your second question, how do we get to the players? So the first thing is the players get nothing during live play. I, i don't think you were, you were implying that, but i just want to be very clear about that. They, they don't get to have that data because it could influence what they, how they play. And the choices they make, they don't get to do that. It's against the rules. So that is something that we work very closely with the rules, staff and all of our, our team that has that in mind to make sure that we are well within those parameters

After that though, you're absolutely right. More and more data, more and more players, more and more interest. Some of them are more and more of them are actually hiring outside help - analysts, people that think about data this way - and we provide all of our players.

So the PGA Tour is actually a membership organization. Since they are members, we provide this data to them in pretty much whatever form they ask for and help them. We give everybody the same set of information and we give it to them raw and then they can fold, spindle and mutilate at their will.

And if you think about on the usage of data side, it's also practice rounds too. Players can get access to the data during the practice rounds. It's just during the field of play that it's not available to them.

Yeah, I'll let you take this one and then we'll come back to that because we'll talk more about practice rounds and some other stuff. There's some goodies here.

Yeah, so as you hear some of the questions, it's so interesting because everybody wants the ability, as you said, the toys, right? The ability to read the data, see the data, it enriches the game. It absolutely creates better engagement.

So when you look at the kind of left-hand side, one thing Eccentric did was we surveyed 1000 different industries and of those CEOs that we talked to, we just wanted to figure out what's the investment that's been put into the infrastructure. You talked about the convergence of the technology versus the expectation of the network versus the digital transformation that you're getting out of it.

So on the left hand side, you're seeing some of the results of this, that really steep hockey stick is really what the impact of digitalization in all these industries is - so incredible, at 162x, basically saying that this is what everybody wants.

The middle line at 40% above is basically how much the network capacity has had to actually keep up with everything that Ken just talked about. And that line at the very bottom that almost looks flat at 2x is actually the investments that have been made in the infrastructure like the network.

So you could look at the very positive side of things to say what little investment has been able to create such a massive explosion of digitalization. And the other side is, you know, we can't really keep up with what is being asked from media, from broadcast, from betting, from the actual fans if we don't change this. And that's what we'll talk about - the second half of this meeting is what are the things that the PGA Tour is doing to change that?

Now, what's interesting is, of that spend on the left hand side, you see that purple and gray bar, 46% of that spend is going towards the legacy network, the legacy network that has tech debt, that has security holes, that has issues. They can't keep up just to keep that running to keep everything going on at that PGA Tour event, almost half the budget is being spent on just keeping it running.

So there's not a whole lot left on innovation, on modernizing, on actually creating a more efficient solution.

And then, what's the answer, right? How do you come out of this situation where the network is an inhibitor at this point, the investments in the infrastructure is lacking and we're not getting the actual output that we require?

Well then, the discussion became, how are you creating a digital core and that digital core - how do you look at the network connectivity? How do you deal with cloud and edge because the edge of the course is where you have to pull all of the data off very quickly?

You said 6 to 7 seconds round trip from the course to AWS and back. No, no, no, not to AWS - that's right - AWS to London, London has to come back and be acknowledged, right? Six seconds from course to AWS to London, come back and go. It's 500 milliseconds off the golf course - that's what the objective is, right?

So 500 off the golf course, 6 to 7 round trip all the way - we like to leave some buffer in there if we can because there's an SLA associated, there is an SLA. Yeah, there's a penalty payment associated if that fails, right?

Yeah, I mean, so this becomes a very mission critical situation because there's a payment tied to it, right? And then what are you gonna do with kind of data? And all those great questions around, you know, how do you look at the data? How do you separate the data - is video just another piece of data? And how do you actually use this in different places?

And then ultimately, how do you wrap cloud and security around that? So being able to reinvent your digital core has been almost the only way you've been able to keep up with this.

Yeah, so not to jump too far ahead, but basically what happened was that about 18 months ago, we made the decision to basically burn that all to the ground and start over and build it from scratch and say that we had squeezed every piece of efficiency we possibly could out of the legacy infrastructure and we had to start over.

And if you think about it, what we do has more or less four layers to it, right? There's a bottom layer of just logistics - you cannot do this unless you think about it as a logistics problem, not just a tech problem. You've got to get this stuff out and get it back every single week. So the circus has to go up and it has to come down.

And then the second layer is the software itself, right? So that living in a server in a truck - servers don't like trucks, I don't know if you guys know that, but it's bad.

And then, so we had to think about that. And the third layer is network - literally network. And we think of network as three things: you've got middle of the golf course to the edge, edge to the cloud, and then cloud to distribution from there, right?

And then the fourth is the one that we already think is what the job is, which is those cool fancy sensors that we put all over the golf course where you get into cameras and things like that.

We had to rethink every single one of those layers and then the network was a big one. And is it - well, we can talk more about what we ended up doing and the convergence of all of that so that you can actually send it to broadcast, do the sports betting, share with the fan experience.

One way is certainly I can use the word "burn it down to the ground." Another way you could look at it is, you know, kind of stabilize the current environment - how do you actually leverage and modernize the current tech in pieces and waves - which you actually are doing, right?

So actually when you say burn it down, I would say from a design perspective, but from a deployment perspective, you're trialing things of the new solution right now before the new tour starts in January.

Yeah, there is an old and fun meme, you know, "I don't often test, but when I do, I test in production" - that was our life this year. And I wish - I'm incredibly proud of the team that did it because we literally did do that every single week.

We built the whole new infrastructure, it was built in parallel while we continued to operate, meeting the SLAs we had to with the legacy system. Once we did that, we started dropping things in.

So we'd put - you saw the solar test that was going out there - that's the power system. We did a new network system, some pieces of that went out, we tested that at various locations. The scoring system itself, we put new sensors out that were the radar and new different cameras that we can talk about.

We put that down starting at - you know, we did one hole, then we did four holes, we stayed at four holes for a long time and then we did nine - so little by little, we were pushing those things into production and the advantage is that we do this every single week.

So you have a lot of reps to figure stuff out and you can see where things are going to break. And so yeah, we made the decision that it was all going to be brand new and we made the decision we were going to do it very quickly. But what we also did was say we're going to stage this and use this strength to test ongoing and recursively over time.

And then once you actually test recursively, then you just step on the gas pedal and go - do three fully accelerated because your season is not changing. Your tour is January in Hawaii and that's when you go live. Yeah, it's gonna be fine. I'm very confident it will be.

You know, one data point that we found out of that research that was really interesting is that 87% - so you heard Ken talk about the AI that he has built, all those pieces that have happened - and what's super interesting is that of those 1000 enterprise executives that we researched, 87% have already said that the growing demand of data and AI systems has already outstripped their current legacy networks.

Like that's a pretty staggering piece. I mean, I'll give you guys an example, I'll give you - I'll throw a number at you really quick. In the last, over the last 2 to 3 years - I would have to pick the actual date - but we, we've gone from, we have multiplied by 10 the amount of data that we're pulling off the golf course. And so that just, that's a lot of pipe that's got to get built big fast.

"So we're now going to shift to what is new, what, what can everybody expect in this room? January in Hawaii. It's going to go fine, no issues, but maybe first talk about your partnership with AWS. What are you doing with AWS? We talked about it going to AWS cloud and then going to be, but what's that relationship like they're your innovation partner?

Actually, let me, let me back up from there for a second because it's, it's hard for me to talk about AWS as a separate thing because it's folded into everything we do. So, so just take a step back and what we said we wanted to do, given the opportunity, we wanted to rethink how we do everything we do. In other words, question literally everything and assume it.

So the first thing it is, he said, look, we're not going to make the s any harder because it's pretty hard, right? That's a challenging one. So we left that where it is, but we wanted to go from ball. So to increase the depth and breadth of the data we collect right now, as i mentioned, we're collecting an xyz coordinate or we were collecting xyz coordinate for start and finish position. What we decided to do was what we wanted to move to a system where we could get ball in motion from every single, every, from every shot.

Ok. So that means that like just paint a picture for yourself for a second when you have xyz data, that's super cool. And you can do interesting and you can build cool stats off it and it's amazing. But when you get to where you're collecting and storing a polynomial for every shot to describe its path and how it moved now, we have literally moved from algebra to calculus in terms of how we can create information and think about statistics. I have no idea what we're going to do with that yet, but it's going to be freaking cool. Ok. So that's, that's coming. That's, that's option two, option three.

The next thing we want to do is we want to do it faster, right? So how can we reorientate our operations so that we do it more quickly, we can put up more. This is literally built around growing capacity, more people want us to put this in more places all the time. It's not just about shrinking cost, it's about doing it in more places. If we can cut that time down, we can deploy this to more areas.

The other piece we wanted to do is reduce the footprint on site. So how can i do that, get down from five trucks to three or, you know, worst case four so that we can put things in fewer boxes and make it smaller. Every one of these things is, oh, and then also we need to be able to move more globally. We're getting more and more demand in more and more places to do this. So that's what we started with when we did that.

It was cool because it forced us to have some constraints. We realized if we're going to do that, we can't do what we've done before. This isn't an incremental upgrade. And that was where we started out and said, well, first off, we need to rethink how we do production entirely.

So instead of having producers sitting in trucks, super cool. Look at right air traffic controls need and you can see the guys doing stuff and we've got these nifty one way mirrors. It's very cool. It's very fun. But at the same time, that is not scalable and doesn't work very well if we're trying to go to a lot of places and do it more quickly and shrink the footprint.

So we immediately moved to where we said we're going to rebuild all of the the software layers. So take the ops layer out for a second, take the infrastructure out that software layer that's defining everything that comes in and collecting all the data and defining it and putting it in buckets that makes sense that is going to be built completely new and cloud native.

That gives us immediate flexibility. We can make it as big or as small as we want. They'd love to tell me that compute and storage are infinite. You know, it's just, it's completely scalable. So we do that and we, and we have taken advantage of that. What that means is that we have moved all of our production to be in our headquarters in potter beach, florida. There are no more people sitting in trucks doing production.

We put, we now do radio over ip. So it doesn't matter where you're sitting, time and space are a construct in this world, right? So it doesn't matter where you're sitting, you can be talking on, on the radio as if you're local, wherever the golf tournament is happening, but you're sitting in potter beach. What that means is that we can think totally differently now that your, your workday is not. If you're a producer, your workday isn't 5 a.m. until after the guys have done an hour after they're done playing golf, which is a long day, right? No, no. If the case comes out that you can work shifts, you can have a life which is something people like to do. You can also, if we need to add capacity. If something goes weird, if we have a weird format, we can put more people on it or they cannot, all these things make it possible disaster recovery by the way, becomes a completely different thing.

I joke about this, but there's a lot of truth in it. We have guys set up right now where they have dual 42 inch monitors that are set vertically. So you can see as much of the golf course as possible because there's sensors. I'm going to talk about in a second. They're sitting there. But if they had to literally find a starbucks open your laptop and you can score the gulf, ok, we can get it done that way if we have to not optimal, but it works. That's what the cloud is, is not like.

We didn't think of this as lift and shift. There was nothing to lift and shift. We were not doing it that way. We said here is a set of tools we can use. How can we build these things together and start thinking about things as processes and microservices and put this stuff into different places where it makes sense for us to operate that way. How can we take advantage of those and build bigger, faster, longer? So we did all of that piece and that built aw with aws.

Then the network piece we'll talk about in a second. The sensor piece, we went completely different and said, instead of with first off, oh by the way, we take as many humans out of the loop as possible, i forgot that was one of the objectives so stop training volunteers so much. We love them and they still will have a role. I'm not pretending that, but can we do something so that it becomes easier? Like it's just easier to find the ball. What that meant was moving to a different sensor set. What that meant was going from 54 cameras around so three around each green to where we had more than 120 that we deploy on the golf course and going from 20 frame black and white to four k full, you know, so and basically full color um that gives us a whole lot of opportunity to collect data and then start building with a i and ml tools that are built because we use everything that is coming into us with. Again, compute is available for us in a cloud world, start building with that and you can take a i and measure, take a two d sensor, put it in a 3d plane and start projecting everything that's happening in real time across that cool 3d map that we built and show you anything you want to see from any angle at any time for any shot. And then when you add to that, the radar and you start layering these two things together, we can get to a point where video is simply a display mechanism. It is not any more. The, it's not just a way to like measure or do things like that.

One of my favorite things i saw was we were testing and we pulled, we have the radar and the cameras talking to each other. Every sensor is both a source and a destination for information, right? So a player hits a ball, radar picks up that shot it broadcasts, we can put, put a trace on it. At the same time, the camera picks up the ball that scoring can we watched as that ball? The trace went outside the frame? So the ball no longer has to be seen to be measured and shown anywhere you want to put it. Now, you're talking about doing things like taking that shot and spinning it in multiple dimensions. Can you follow it from the side and still see what's doing and put data around it? Now you're starting to tell stories with that. So that's what that gives us. And that's by the way, that is the scalability, that is the opportunity that we had with aws.

But then the ability to converge all of that to tell a story is really, really hard, right? You have to be able to do what's the story you want to tell? What's the actual layers of the technology that you need? How do you find the right partners for each one? And then how do you put it all together and then oh by the way, it's changing day to day, it's pretty cool, pretty easy. No, no, no problem there.

Um when it comes to one thing you mentioned was the network part of it, right? So that was part of the stack that we had talked about and we had worked on together. So extension and pga tour worked on kind of the network layer of it. And you asked me and the team to come in and evaluate the different ways to not have to use all of the fiber because your shot link, i was now, as you said, doubling the amount of radar increasing the amount of cameras. So that's just more fiber, right? And that then as, as the gentleman asked before, that's more digging, that's more transport costs, that is more operational challenges.

So one thing that i was super impressed with was you actually had a lot of hypothesis on what you wanted to look at, maybe opinions but hypothesis. What were the things that you wanted us to look at before we ultimately kind of decided and designed the january solution?

So here's the, the this is a very practical problem said we, we're going to put cameras around the entire golf course. That meant that this is going to sound funny, but we had to go to the other side of the fairway, ok? Because at that point, we're no longer just putting cameras around the green. You're not putting one on each side of the fairway on a par five and sometimes more than that plus one behind the green before that we would run down one side of the fairway and around the green. And that was it, we did not want to run fiber around every hole as well as around +63 loops, right? That's just more and more to your point.

So we started looking at options, obviously, you know, wireless is a thing. Power is still a thing and that's where we ended up with. So talking about the network first, we, we started out looking at what is a, we need high volume, low latency and dedicated like it has to be resilient. And we started laying out all the different options that were there.

So we obviously looked at, we looked at private five g, we looked at hybrid public private five g. We looked at a lot of different elements in that. We looked at some other straight up like fiber alone as an option for some private wireless stuff. And, and we also looked at some other solutions that would do it and where we ended up, what we ended up doing was just looking at what was going to give us, what's this an operational optimization? Something that is cost-effective, but more more than that resilient, right? And something that that would allow us to get and had enough throughput to actually make it work.

And so we built this fund matrix that you know, was very matrix about how that would work. And we tested these for over for about a year, we tested all kinds, well, more than that, even a bunch of different solutions, building every one of those options out and trying different things with them. And then when we did, once we figure that out is we, we realized, look, we, we do know a lot about this. We work in this. We are reasonably intelligent people. We also know that we don't know everything.

And so that was why we asked accenture to come in and basically tell us a, did we ask the right questions? B did we look at all the right options? And what did we miss? Like what did we just not know that we didn't know? And so that's what we work together on and then how can we create a nominal design on one of the, the different selections? What's the business case around that math question you had asked? And then ultimately, who are the partners that could help you deliver this? And that was the fun journey that we were on?"