Accelerate EDI data integration at scale with AWS B2B Data Interchange

All right, good evening. How is everybody doing? Yeah, exciting keynote this morning. Uh you all got to watch Adam's keynote uh some exciting AI innovations.

Um so just a quick show of hands, you know how many of you you have to deal with business partner processes, getting data from business partners or trading partner integrations? Ok. Ok, good. And how many of you are doing it manually today? Any manual process in your organization? No. Ok. More or less. Ok.

Um great. I think you, you guys are in the grades in, in the right session at least because we are here to talk about how we can accelerate some of your EDI based data integrations and make it scale as your workloads grow.

Uh I'm Smith, joined by my colleague, Russ Boyer and really excited to launch this new service uh AWS B to B Data Interchange Just a quick run through the agenda. So I'll briefly talk about, you know why and what are the current challenges, right? Which you know those points you may be familiar with and would like to get that validation. And then that's talk about our approach to addressing those challenges with, right, with the new service that we just launched yesterday, I'll talk about some of the use cases that we are targeting on a broad basis, right? What are the two categories of use cases? Um and then hand it over to Russ who will go into a little deeper dive on some of the features and then of course, no breakout can end without a demo. So we'll have Russ demo and then wrap it up for a Q and A and we can talk more after.

So a little bit more about why um why EDI, right? The big part of is the electronic, right? versus a manual way of doing things, right? Of exchanging files, getting a PO cutting a PO getting inventory data, all of that. And the other aspect of is the industry standard, right? You've got X12 that's widely used in North America and that standardized just makes it easier for you to add more business partners because now in some sense, you're speaking the same language, but I'll come to the variations in E even within X12 that can happen and how you deal with that. But at least, you know, it's a common standard. And if somebody wants to do business, when you say, hey, you speak X12, that's a good entry into the door and to, to sign the deal right, to do business.

So that's one part of it, which is that industry standard the second aspect is you know what to increase your own customer reach, right? Depending on what's your role in that chain, that supply chain, you may want to increase your customer reach or you, when you are as a manufacturer, who are your distributors and you want to increase your reach or as a manufacturer, you want to increase your reach to suppliers. So you reduce your risk. So this just makes you increase the number of trading partners in a much scalable way with EDI, right?

And the third, I think and third and actually more important part is reducing errors with that electronic part, right? Any manual process or you know, think of an example, right? A customer that we were working with, they were like, ok, pick up the phone, we say, hey, here's the purchase order, right? I'm gonna send, I'm gonna be sending it to you and then they send the email, the purchase order goes over email, they call, hey, did you get it? They say yes, ok, then they take that on the other and the supplier takes that purchase order, feeds it into their system, doesn't work, something happened and they call back again.

So all of that just increases the time that it takes to transact, right? And add so much overhead of phone calls and emails and data format errors and ERP issues all of that. So it's just a reduction, just at least aims at reducing those errors. And increasing speed and increasing accuracy, right, then what's the problem and why are we here? Right.

So with the couple of trends that we are seeing, right, as we talk to customers, right? One of the trend is the customers want to move their EDI systems to the cloud, right? As they are moving a bunch of applications, right? They are thinking, hey, ah how do we what happens to that based integrations? Right. Even they also move to the cloud as you retire your data centers, right? That's one thing, one trend that's driving some of the needs, right?

And the second trend that's there is obviously the need for more automation end to end, right? So as these customers move to the cloud, right, they're looking at hey, we want to keep EDI, right? But how do we do that in the cloud? What does it entail? Right?

Additionally, right, a lot of enterprises want to bring the integrations in house, let's say they're doing it externally. So they have more control over that relationship, more control over what's happening with that business partner, right?

So as part of bringing it in house, bringing it to the cloud customers have expressed, hey, here are some challenges, right? First, you need to build a stack, right? You need to build a stack for on boarding, you know, queuing communication storage data lake. So let's talk through a little bit about each aspect of the stack.

The first on boarding, right? Bringing those trading partners, even if though it's a configuration and the initial setup that can take time, right? How do you test? How do you, I remember i talked about the phone call even during testing setting up. Right? You want to, if it, if it's happening in months, how do you bring that down to days or even hours? Right. So that's one aspect.

The second one is no visibility into errors, right? It failed. It doesn't give you a response. You don't know why it failed. All right, let's try the same document the second time it will probably work. It didn't. All right, let's change the document. Let's change the spacing in this document and right. So it's just again, a lot of back and forth and how do you troubleshoot those errors? So that's one part of that stack, right?

The second part of the stack is complexity of EDI, right? EDI has a lot of versions, a lot of document types, even within an X12, even within a purchase order 850 you can, you can construct it in so many different ways, right? And even the whole translator, how do you set it up? How do you mean it as your business grows? The translator can get choked saying, hey, this is a seasonal thing. I didn't expect to see like these many orders in this such a short period of time. So it's just managing that service and maintaining it and keeping it updated, right?

And that's another pain point that customers have experienced and then look the other driver, right? Moving to the cloud, you all saw the keynote this morning, right? With so much innovations in AI and how do you use all of this data and keep up right for your business use case, right? Get insights from this data and and when you keep this in a place in a storage system, let's say on prem all of these files, not easy to use them for with innovations happening in the cloud, right?

So these are these pain points or some of the challenges, right? I kind of sometimes interchange trend and a challenge because a trend can be a challenge if you're not ready for it, trend can be an opportunity, right? If you jump, jump on it. So, so you can look at it either way and that's exactly why we launched the service right yesterday.

So you all are one of the first few people to learn about it, AWS B2B Data Interchange. That's our approach to addressing the challenges with EDI, right? So fully managed service, right? Helps you exchange based transactions, transactional documents with your business partners, right? And the idea behind AWS B2B Data Interchange is to make EDI easy, right? You want to use EDI, right? You need to use and how can we ease, ease that process for you? Right? In some sense, you may want to think about it as I want to use EDI without having to deal with, right? But I'm using so we want you to get the best of both worlds with this service.

Um so here I'm going to talk a little bit about some of the three core features of the service. Uh and then Russ is gonna do a deeper dive into some of these features.

The first one is trading partner management, right? To any B2B service heart of that is your trading partners, your list of trading partners. How do you track them? How do you know what, how do you do business with them, meet them where they are easily, right? That's the goal.

So we have you can easily create your trading partner portfolios, right? And then you know, enter their business name. So it just makes it easy. Trading partner is just a resource right in your cloud, right? And you can save time by assigning like the mapper for multiple trading partners or single. Like it just helps you reuse some of the templates.

So you want to grow your supplier list from 10 to 50 to 100 right? It just makes it easy to assign the map map of profile across one of them instead of having to individually create them by hand, right? So just the idea being behind trading partner integrations, how do i scale up my business and add my trading partners to my portfolio faster.

Um the other feature that the core part of any EDI service is translation, right? You have the AWS B2B Data Interchange works with any communication service, right? Uh Russ will show how it will work with AWS Transfer Family. And how many of you have heard of AWS Transfer Family, which is a fully managed service for file protocols like SFTP FTP which are common protocols used to transmit EDI documents over the wire, right?

So the service will work with any, like even if you're using any third party communication tool, it works as well as with AWS Transfer Family service. As I mentioned, reuse these mappings. You can set up these mappings say, hey, here's my 850 here's, you know, and I want to get it in, I want to get it translated to JSON and, but I want to internally consume it in this format so you can kind of set up, set up these rules and then use them across your training partners.

Automation. I think I mentioned that driving one of the trend. So that's the third one is you can set it up. So that as soon as the file lands in your bucket and EventBridge, a rule kicks off targeting the service, right? And the translator automatically picks it up, translates it based on the rules you've set up and then outputs it to another S3 bucket. So it's kind of like a set it and forget it once you set it up. And that's the idea behind our approach

Monitoring, right, as i mentioned, right? You know, that errors are bound to happen. How can you at least and know what the error is as fast as you can and go fix it? Right. Um so the the transformer throws all the errors into your CloudWatch logs and CloudWatch. You heard like there's so many innovations happening in Amazon CloudWatch where you can you know, search in the logs filter and then even and even track who are my top trading partners. What are the top documents i send or receive? Like what's what's happening so that you get a lot of insights right from your the logs that are being transmitted as these documents are being transmitted.

Um all right. So let's talk about some of the use cases that we are targeting, right? The first category is data lake hydration, right? And I'll talk a little bit more about how we view on you getting a carbon copy of all these transactions and how you can push it into your AWS based data lake, right? And the second part is application integration and this is more as you send and receive your transactional documents over the wire or you feed it directly into your business applications.

So some of the use cases for data lake hydration that we have heard again. This is not an exhaustive list, right? This is just to kind of inspire on what the art of the possible, right?

In manufacturing, right? Let's say you're in a farm pharma manufacturing company and you need to send regular reports to an agency, right? So imagine the ease at which you can do that if all these transactions are in your data lake in the right format. And you have a quick side dashboard and you just download these reports and hit send, right? So that's the idea behind it. So you reduce the manual work and kind of trying to create these reports regularly and you can keep up with the SLA with your with your regulatory agency, you on retail or e-commerce, right? Inventory levels are key, right?

So you may be using an analytics application and as these transactions are exchanged, you can directly get them. Yeah. Uh sure we support them and get which one health care will come to that in a bit. Yeah. Yeah, but that's also another health care is also another vertical that's important like your payer and provider 837 and all that. Yeah, totally.

So monitoring inventory levels and then predicting prediction, prediction is a big deal right now in supply chain. Hey, I know my data coming in and the key is to do it in real time and as close to when the transactions are happening and updating your model quickly, right?

And the third one is if you're in transportation or logistics, right, you're on the hook to get a data, to get a shipment from point A to point B the carriers behind their milestone update. Right. They were supposed to check into your warehouse in location B but they're not there. So at least, you know, you can let your customer know that, hey, this will be late, you know, you can be proactive on top of it or you can even predict and even rate the carriers based on, you know how they're faring. So there's, it just opens up a lot of options, having these transactions directly in your daily and to inspire that architecture just to give you.

And this is a very common architecture that we have also a demo videos and Russ is also going to do a demo as the data, you know, i call it, i call it a carbon copy, right? Your transactions may still be happening on prem. But if you have a listener to push those transactions via Transfer Family over FTP or SFTP lands into your S3 bucket, triggers an event and then B2B Data Interchange will take the rules that you have set up for those transactions converted into say JSON or XML. And then you put it on t bucket, set up a crawler, a Glue crawler that builds a catalog and feeds it and via Athena, you can create it into QuickSight.

So just imagine how beautiful like you've just taken a 40 year old technology EDI and now you have a QuickSight dashboard, right? And all of a sudden like, I mean, I can go on and on like based on the keynote like take Amazon Q right from this morning

And then now you're saying, hey, how many purchase order did trading partner ex send me? Right? Just, just the two worlds are coming together, you know, in this picture for me.

Um again, I think the next few walk through, right? How, how you're building your edi based data pipeline, right? Think about that, right? It's your data that's from your trading partner, your own data lake and it's building an end to end data pipeline across.

Okay. All right. So for application integration, let's talk a little bit about this. This is the transactional data, right? You are in manufacturing, you want to cut a po or you want to get a po right from your like or you are cut up if you're a manufacturer cutting a po to your suppliers, right? How do you do that easily? If you want retail, you want to exchange, maybe you're getting a po right? You retailer and you're receiving a po, right? So how do you do, how do you get those use cases and scale up? Right? As your business grows, right?

So especially if, if you think if you're a small or medium business, you want to do, you want to reduce that friction, you don't want to go build a whole system and you know, just like heavy investment like a cloud based edi makes it much easier for you to do that. You are a large enterprise. Again, you know, you may be paid by your existing system and you want to build it and have a more scalable architecture. Again, transportation logistics, you want all of these transportation transaction documents bill of lading custom declarations, milestone updates, all of these, you can receive them and then directly feed them into your erp again, a similar architecture here except i would say it's not a carbon copy, right?

You can bring your trading partner directly, right to send, submit these documents over transfer families sftp or a two, right? And then as that data is coming in feed again, similarly feed it to your s3 bucket b data interchange set up, right? Uh uses the rules you set, you want json json xml, xml. And then you can use a service like appflow to feed it into your erp or transportation management system, right?

Appflow has an s a connector. You may have your own sap connector. So once it's in its s3, there are a lot of options on how you want that data fed to your business application.

Cool. All right. So with that, uh I will hand it to Russ who is gonna talk a little bit more, go double click into how some of these features work. And um and then we'll have a demo. Here you go, Russ.

All that stands between you and dinner is how fast I do this. So I'll uh I'll try to keep it brief, but I will try to be as comprehensive as possible just so you can understand what we've launched and what to expect.

So Smith already covered, you know, in, in pretty good depth b to b data interchange, ultimately getting data into s3. However, you want to get it into s3, you know, Smith and I work for aws transfer family. So we're a little biased. We think it's a great way to get data into s3. But you may have other pipelines. You may be writing that data directly into s3. You could even just upload the file through uh you know the console into s3 and we're going to transform it.

And so the way that that ultimately works is we are calling a service managed eventbridge rule that is looking in a particular location for the file to show up and is going to execute a transformation against it. An important note does not have to happen that way. We also have an api that you can call and you can point that api at a at a file that already exists in s3 and say, hey, i want to transform using this particular transformer this file in s3.

For the automation, you can just throw it in the incoming inbox and we'll take care of it. For you. And that's what we're going to see in the demo here in a minute.

Optionally, we can map the data. So what do I mean by optionally? Well, we're automatically going to convert the data to either json or xml. But if you want to re resort reorder zoom in and only grab certain fields out of that file, you can apply a map and I'll show you that in the console here in a minute and then we're going to output the file back to s3 for you to pick it up in whatever pipeline you want.

So, you know, as Smith mentioned, maybe that's a glue crawler that's going to feed that data into your data lake. Perhaps that is appflow, feeding it into your erp system or, you know, any other integration where json might be friendly. I'm learning a lot about, you know, oh, I just need the json and then I'm going to do all kinds of things to those files when we had a little workshop about the service earlier today.

So I think we're going to learn a lot about what customers want to do with the service now that we've launched it, they're beyond our expectations. So just really quickly, you know, I like to say, you're going to transform your edi in four easy steps, right? Because that sounds cool. But really it is just four steps that you have to go through to set up one of these partnerships and actually transform your files first.

You want to create a profile and that profile is going to contain important information such as how do you get in touch with the person that you are doing business with? You can create multiple profiles. Maybe you're providing ed i services out to different lines of business. You don't, you know, you, you're not limited in the number of profiles that you create.

After you create the profile, you are going to create a transformer. The transformer is ultimately where you're going to. It's, it's actually really cool. We're going to load the file in. We're going to take a look at it. We're going to pick, you know, what do we want to go to json? Do we want to go to xml? And we're going to be able to map that data and see what the resulting file would look like based on a sample file from there.

We want to create a trading capability. Trading capability essentially takes a transformer and gives it a location in s3 to look for files coming in and apply that transformer. So the thing of the trading capability is really the glue that sort of binds it all together says, hey, i want to apply this transformer to this incoming space or folder and then you're going to create a partnership, a partnership essentially is going to define who you're exchanging files with.

And also is going to allow you to pick the trading capability for that particular partnership. And that's going to give you a specific end box for just that partner. So when the files come in and we fire these eventbridge rules, you know, based on that specific partnership id, we're going to land the files back out in s3 and an outgoing folder that's also going to be specific to that trading partner id.

And that's how you can keep up with different files that belong to different partners downstream and have some nice separation including at your i am role level. For example, in s3, if you only want specific people to be able to pick up specific files, it gives you some nice hooks with the prefixes of the file to say, i only want to be able to expose this data to certain people.

Beyond that, we're going to be able to monitor all the activity related to uh your transformations, all the activity related to each profile, all the part, the partnership activity, it's all going to get logged to cloudwatch and you're able to see that or visualize that in cloudwatch directly. But in addition, we have a nice feed of those logs in the in the console itself under the partnership. And we'll look at that in the console in a moment.

It's kind of hard to get that screenshot to cover all of it and still be able to read it. But you can see it peeking out down at the oh wait, i'm on the wrong screen. Sorry about that. You can see that peeking out down at the bottom uh that log feed and we'll get into that.

So at a high level, we are going to be using aws transfer family to emulate a file, send operation. I'm going to use a aws transfer sft. I mean, I'm sorry. A s two connector that as two connector is going to reach out to an ass two server also transfer family just to simulate the idea of an external business partner sending you an ad file.

We are going to have that file um land into the appropriate folder so that we can kick off the transformation of the file. We're going to have b to b data interchange map that file from ed i, i'm going to do json in this particular demo. It's a little more pars with the human eye to me.

Um we'll download that file and you know, take a look at it, see how it turned out, see what we did to it. Uh we will have already seen that in the mapping. Uh but you know, just so that you can understand and see what that looks like. And then we'll go out and we'll view and cloud watch that logging information, including looking into uh the partner profile. I mean the partnership um logs as well.

So i'm gonna switch over here. Oh success got, got a little scary there for a second. Sorry about that. I'm just going to go ahead and show you. I've got some uh profiles i pre created, but just so you can see the process. I am going to go ahead and quickly pull up a cheat sheet of notes because i need to grab one statement here for later and i'm gonna copy that and we'll just go ahead and go ahead and create a profile.

I'm going to call this profile re invent 2023 the business name we're gonna say b to b data interchange. And we'll do, i, i won't do smith's email address as the primary email address even though it's very tempting to do that. And we will create that profile, we see that profile show up, you know, on the profile screen.

Next we'll go and we'll create a transformer. And as part of creating the transformer, i'm going to name this, the re invent transformer. I'm going to go with this 2 1440 10 format

I have a sample file uh conveniently preloaded here. As you can see, we just select a sample file right out of S3 and we choose it and then we can go ahead and go to next here. Oh sorry, try that again. And then immediately you see that what we're doing here is by default, we've got JSON selected. So it has parsed our entire EDI file and that's what you see represented here. It's, it's all converting to JSON and then the mapping preview that we see on the right, it would just be the full file. So in this case, we would just receive the EDI file, we would transform it directly into JSON.

If we scroll back up, you can see here that I can choose XML. Um I may anger the demo gods by clicking that. So no, I did not. So you can see I can choose XML right here. And uh but we'll go back to JSON because I've already got this filter set up. And so if I go ahead and just paste in this filter, you can see and, and you know, conveniently we have a link out to the JSON, JSON or JSON documentation. That's the default documentation for how to parse this kind of file and how to query this. Like if you want to learn more and you can experiment here, you can make as many changes as you want, apply those, see how it all works because at the bottom, you're going to see the mapping preview. It's just going to automatically show you exactly what the resulting file would look like.

You could reorder things, right? You could just in this case, what we're doing is just, you know, pulling out only specific data that we want. I'm going to go ahead and save that transformer again. It gives me a final preview of what the resulting file would look like. So now if we go in and create a trading capability and we'll call this EDI 214 re invent. Oh, you know, I forgot, I knew I would do one thing. We do have to set our transformer to an active status before it's available to assign to a trading capability.

So let's start that process over and we'll choose the transformer that we just made and here we can browse in S3 and we can actually take a look at the folder structure here. I'm going to go ahead and just choose this inbound folder that I've prec created. And I'm gonna choose that and I'm gonna do the same thing for outbound as well.

So just really quickly to, to help you understand a little bit about what's happening that you do need a bucket policy. If you see over here, we have a copy policy. So this bucket policy is what allows B2B Data Interchange to access the data in the bucket. So once we have selected the specific location specific bucket that we want to load files into the service will dynamically create a policy for you to copy that you can copy and paste right into the bucket policy. So you know, we're doing that for you short of actually creating that, you know, so that we don't create any security concerns with, with your infosec team. So we will give you the policy to apply to the bucket itself.

Another important note, if we go look at the bucket where we're actually, you know, transforming these files. So if we look at permissions here, you can see, I've got this bucket policy applied and that's allowing the service to go ahead and write. If we go look at properties, one other thing we want to look for is that we have a vent bridge set to on. So what the B2B Data Interchange service is doing is creating a service managed eventbridge rule. And you need a vent bridge to be on at the bucket level in order for B2B Data Interchange to be able to do its file processing using a partnership.

So if we back out here to our create capability and we can go ahead and create that capability from here, we'll create a partnership and we'll call that re invent partner. We can choose what profile that we want to bind this partner to and we'll just choose the re invent 2023 profile that we created earlier and we can choose which trading capability is relevant to that partnership and we can go ahead and create that partnership.

You see when we create the partnership, if we actually go explore the partnership by selecting it that each trading partner gets their own trading partner id. So this is how if we look at the bucket itself and we go back and browse the objects and we go to inbound, we'll see that the trading partnership has a folder in the, the selected prefix specific to that trading partner.

So I've pre created a partnership that I've got transfer family all set up to work with. And while we wait on cloudshell to reload, I'm going to show you first the connector that i have created. So this is an AS2 connector and you can see what server I'm pointed at here and this connector is going to go ahead and send the file off to this AS2 server. It's a Transfer Family AS2 server. In this case, we have a defined agreement and that agreement is going to land the files into the appropriate incoming directory.

So I'm going to go ahead and execute that. And you see we get a transfer id. This is just the unique id associated with this file movement. So that transfer id, you know, not relevant to B2B Data Interchange, but just so, you know, in CloudWatch, that's, you know, the log stream would contain information related to this transfer id.

So if we go look into our partnership that i selected here, go ahead and look at the partnership id and it is the one that ends in six f 58. So we'll go into that folder and we should see a file created here and we do. It's this bottom one. That's the file that i just uploaded. That's the, the EDI i file, it does get its name and convention changed slightly by Transfer Family as part of the process.

And if we, if all has gone, well, we will go to our outbound directory and we should see a corresponding transform file. And we do here is our object, our transformed object, we'll go ahead and download that file and i'll open it up and we'll see if we can't zoom in on that a little bit. Hopefully you can see that. So this is our file. We we fed in our EDI file, it automatically transformed it to JSON and applied the map. And in this case, i had the same map applied to the other transformer. So it works the same as we would expect it to.

And if we step back into our partnership in the console and we go down, we'll see all of the information relevant to this particular transformation shows up right in the console on the partnership page. So you don't have to actually go to CloudWatch to see what activity is happening in the partnership. You can see that right in the console. And you see here, you know all of the relevant details of we match the file, we said ok, this matches our trading capability and then we transformed the file and completed that as well.

If we do follow this convenient link here, we can jump out to a CloudWatch Insights dashboard that will load up this data as well. And you can see here, I've done some other test runs. We have a complete log stream of that or we can actually just visit the CloudWatch log stream itself and see the raw logs without having the uh applied the inside dashboard.

Alright. So that is our demo for today. Hopefully, I gave you a little bit of an understanding of B2B Data Interchange and how it works. I'm gonna go ahead and go back to the presentation and let Smither wrap it up. Thank you.

Cool. So I hope you got an idea of how this can help with your integrations and you know the potential and where we can help you with migrating your based workload. Just a quick on regional availability. The service is available in three regions today, Northern Virginia, Ohio and Oregon region a little bit on the pricing per partnership. As you saw r created, each one of those are $8 per partnership, right per month and each transformation step is a penny per transformation. So again, pay as you use, use what you need delete and then you won't incur the cost of it. And for more details, please visit our website on the pricing and the rest of the information.

Um so if you just wrapping it up with a few key takeaways, which i hope you also agree with, right can be automated and help you streamline your business, right? Operations, reduce risk increase revenue potential. Um using AWS for ed helps you leverage AWS innovations, right, in the field of analytics a iml, right? And uh AWS with the launch of this service, AWS now offers end to end capabilities for running your ed workloads. Right?

So thank you again for coming and we'll open it up. If you want more information, we have a news blog on exactly all the steps that Russ talked about. There's also a video if you go to our product page, a demo that Russ gave, if you want to follow along similar things. So these are links that you can follow. And um yeah, thank you again for coming. Enjoy your evening.