lpm算法_在熊猫中实施ip lpm

lpm算法

Longest Prefix Match (LPM) is the algorithm used in IP networks to forward packets. The algorithm is used to select the one entry in the routing table (for those that know, I really mean the FIB — forwarding information base — here when I say routing table) that best matches the destination address in the IP packet that the router is forwarding. I am an old school systems programmer, used to programming in C or even Python, not the new style method chaining model used in the popular Python data analysis package, Pandas. This post is a description of my experiments in implementing LPM in pandas. The goal obviously is to ensure that LPM completed as fast as possible on the full Internet feed, but potentially much larger.

最长前缀匹配(LPM)是IP网络中用于转发数据包的算法。 该算法用于选择路由表中的一个条目(对于那些知道的人，我的意思是我的意思是我的意思是真正的FIB(转发信息库)，在这里我说路由表)，该条目与路由器所处IP包中的目标地址最匹配。转发。 我是一个老派的系统程序员，曾经使用C甚至Python进行编程，而不是流行的Python数据分析包Pandas中使用的新型方法链接模型。 这篇文章描述了我在熊猫中实施LPM的实验。 显然，目标是确保LPM在完整的Internet提要上尽快完成，但有可能更大。

最长前缀匹配(LPM) (Longest Prefix Match (LPM))

Any IP address consists of two parts: the network part and the host specific part. The network part is also called the subnet. The network part is often written as a combination of 2 pieces: the IP network address and the prefix length. An example of an IP subnet written this way is: 192.168.0.0/24, where the network address is 192.168.0.0 and the prefix length is 24. The prefix length, 24 in this case, represents the number of bits used by the network part of the address. The remaining bits in the IP address is used to construct the host part. IPv4 addresses are 32 bits in length. So, in the IPv4 subnet 192.168.0.0/24, 24 bits are used to represent the network part and the remaining 8 bits are used to assign to hosts. Thus, the subnet 192.168.0.0/24 can contain upto 256 hosts, though in reality, the first and last entries are used up to create a special entry called the subnet broadcast network. So, 192.168.0.1 is the first assignable entry to a host in this subnet. IPv6 behaves the same way except that it has 128 bits instead of IPv4 32 bits.

任何IP地址都由两部分组成：网络部分和主机特定部分。 网络部分也称为子网。 网络部分通常写为两部分的组合：IP网络地址和前缀长度。 以这种方式编写的IP子网的示例为：192.168.0.0/24，其中网络地址为192.168.0.0，前缀长度为24。在这种情况下，前缀长度为24，表示网络使用的位数。地址的一部分。 IP地址中的其余位用于构造主机部分。 IPv4地址的长度为32位。 因此，在IPv4子网192.168.0.0/24中，24位用于表示网络部分，其余8位用于分配给主机。 因此，子网192.168.0.0/24最多可以包含256个主机，尽管实际上，第一个和最后一个条目都用完了，以创建一个称为子网广播网络的特殊条目。 因此，192.168.0.1是该子网中主机的第一个可分配条目。 IPv6的行为相同，只是它具有128位而不是IPv4 32位。

An IP routing table (I mean, FIB), which a router looks up to decide how to forward a packet, consists of many of these network address entries. The network address entry is also called a prefix because it forms the prefix of an IP address. To select the best matching entry for an IP address, logically, the router must select all the network addresses that can contain the address in question. Thus, if the routing table contains 192.168.0.0/16, 192.168.0.0/24, 192.168.0.0/28, and 192.168.0.1/32, then all of these entries are valid entries for an IP address of 192.168.0.1, while only the first three are valid entries for the address 192.168.0.4, and only the first two are valid entries for the address 192.168.0.20. Once the valid entries are selected, to select only one amongst these, the routing logic selects the entry with the longest prefix. Thus 192.168.0.0/24 is selected over the entry 192.168.0.0/16, 192.168.0.0/28 is selected over 192.168.0.0/24, and the /32 entry wins over all of them for an IPv4 address (a /128 entry in the case of IPv6). Thus, the IP packet forwarding algorithm is called longest prefix match.

路由器查找以决定如何转发数据包的IP路由表(我的意思是FIB)由许多这些网络地址条目组成。 网络地址条目也称为前缀，因为它形成IP地址的前缀。 要为IP地址选择最匹配的条目，从逻辑上讲，路由器必须选择可以包含所讨论地址的所有网络地址。 因此，如果路由表包含192.168.0.0/16、192.168.0.0/24、192.168.0.0/28和192.168.0.1/32，则所有这些条目都是IP地址为192.168.0.1的有效条目，而只有前三个是地址192.168.0.4的有效条目，只有前两个是地址192.168.0.20的有效条目。 一旦选择了有效条目，路由逻辑将选择前缀最长的条目，以仅从这些条目中选择一个。 因此，在条目192.168.0.0/16上选择了192.168.0.0/24，在192.168.0.0/24上选择了192.168.0.0/28，并且/ 32条目赢得了所有IPv4地址(/ 128条目)对于IPv6)。 因此，IP数据包转发算法称为最长前缀匹配。

大熊猫 (Pandas)

Pandas is is one of the essential libraries for manipulating data in Python. Pandas provides a dizzying number of python functions to implement query and manipulate data. The Dataframe is one of the two most fundamental data structures used in data analysis in pandas (the other being Series). A Dataframe is nothing but a table with rows and columns (every column is a Series). Every column is of a specific data type, such as integer, string, object etc. To those versed in databases, the DataFrame is just like the rows and columns in a traditional Relational Database such as MySQL or Oracle. Thus in Suzieq, the routing table is just a Dataframe. Here is an example of a couple of routing entries as pandas DataFrame in Suzieq

Pandas是在Python中处理数据的基本库之一。 Pandas提供了令人眼花number乱的python函数来实现查询和处理数据。 数据框是熊猫数据分析中使用的两个最基本的数据结构之一(另一个是Series)。 数据框不过是具有行和列的表(每列都是一个系列)。 每列都是特定的数据类型，例如整数，字符串，对象等。对于那些精通数据库的人来说，DataFrame就像传统的关系数据库(例如MySQL或Oracle)中的行和列一样。 因此，在Suzieq中 ，路由表只是一个数据帧。 这是Suzieq中几个路由条目作为pandas DataFrame的示例

Figuring out an optimal way of querying and retrieving data can be a fascinating exercise, though many well understood patterns have come about that can be applied to a data analysis problem.

尽管有许多众所周知的模式可以应用于数据分析问题，但找出一种查询和检索数据的最佳方式可能是一个有趣的练习。

熊猫的LPM (LPM in Pandas)

I wanted to implement the LPM logic Suzieq using pandas. A full Internet IPv4 routing table is 800K routes at the time of this writing. This could easily run into millions of entries with multiple routers in Suzieq’s database. So, I wanted an implementation that was fast enough at such large numbers.

我想使用熊猫实现LPM逻辑Suzieq。 撰写本文时，完整的Internet IPv4路由表为800K路由。 在Suzieq的数据库中，使用多个路由器可以很容易地将其录入数百万个条目。 因此，我想要一个足够快的实现。

LPM in simple pseudocode that is independent of any underlying data structure looks roughly as follows:

独立于任何基础数据结构的简单伪代码中的LPM大致如下：

selected_entry = None
for each row in the route table
   if the given address is a subnet of the prefix:
    if the prefix length of the prefix is larger than the existing prefix length:
      selected_entry = row
return selected_entry

Typically the routing table in most packet forwarding software such as the Linux kernel or in software routers is implemented using a Patricia Trie. Lots of papers have explored alternate data structures to implement a faster LPM. Their goal is to forward packets as fast as possible. In packet switching ASICs, the LPM is typically implemented using a Ternary CAM (TCAM). In Suzieq, I’m not forwarding packets, I just need the algorithm to be fast enough to not bore the human using it or be fast enough for other programs to use it.

通常，大多数数据包转发软件(例如Linux内核)或软件路由器中的路由表都是使用Patricia Trie实现的 。 许多论文已经探索了替代数据结构以实现更快的LPM。 他们的目标是尽快转发数据包。 在分组交换ASIC中，通常使用三进制CAM (TCAM)来实现LPM。 在Suzieq中，我不是在转发数据包，我只需要算法足够快以至于不会让使用它的人感到厌烦，或者足够快地让其他程序使用它。

In Pandas, I have no data structure such as a Patricia Trie. All I have is the Dataframe. I could have tried to suck the data out of Pandas and stuffed it into a Patricia Trie like data structure to query. But this would’ve resulted in too much time to build the Patricia Trie. In addition, in Suzieq its possible to perform the LPM from the point of view of multiple routers in a network in a single query. I needed the ability to support this as well. Constructing multiple Patricia Tries or using a new data structure with modifications to support all the additional requirements seemed too onerous. In any analysis, getting the data into the right structures for analysis consume a significant portion of the time. Caching really helps here, but since the underlying data can change at any time, I wanted to avoid caching the route table in a different data structure. Lots of other possibilities exist, but all involved doing something outside the methods available in Pandas.

在Pandas中，我没有Patricia Trie这样的数据结构。 我所拥有的只是数据框。 我本可以尝试将数据从Pandas中吸出来，然后将其填充到Patricia Trie之类的数据结构中进行查询。 但这将花费太多时间来构建Patricia Trie。 此外，在Suzieq中，可以通过单个查询从网络中的多个路由器的角度执行LPM。 我也需要支持这一点的能力。 构造多个Patricia Tries或使用经过修改以支持所有其他要求的新数据结构似乎太麻烦了。 在任何分析中，将数据放入正确的结构进行分析都将花费大量时间。 缓存在这里确实有帮助，但是由于基础数据可以随时更改，因此我想避免将路由表缓存在其他数据结构中。 存在许多其他可能性，但是所有其他可能性都涉及在Pandas可用的方法之外做一些事情。

To stick with Pandas, the most naive implementation, one that appears most immediately to a programmer schooled in C, is the one that follows the pseudocode shown above. dstaddr is the string containing the IP address I’m trying to find the LPM for.

坚持使用Pandas，最天真的实现(对使用C语言学习的程序员最直接的实现)是遵循上述伪代码的实现。 dstaddr是包含我要为其查找LPM的IP地址的字符串。

dstip = ip_network(dstaddr)
result = []
selected_plen = -1
for index, row in route_table.iterrows():
  rtentry = ip_network(row[‘prefix’])
  if dstip.subnet_of(rtentry) and rtentry.prefixlen > selected_plen:
     result = row
result_df = pd.concat(result)

But this results in a terrible performance, said every thing I’d ever read about programming in Pandas (see this as an example). I had read this enough in multiple places to not even try to implement this to see what the numbers would be. My reading led me to think that the trick had to be to somehow make it a part of pandas natural style of working with data. Maybe implementing IP network as a basic data type in pandas was the right approach.

但是，这会导致一个可怕的表演，说的每一件事情我曾经阅读有关大熊猫编程(见本为例)。 我已经在多个地方读够了，甚至没有尝试实现它来查看数字。 我的阅读使我认为诀窍一定是要使其成为熊猫处理数据的自然风格的一部分。 也许将IP网络作为熊猫的基本数据类型来实现是正确的方法。

Pandas allows users to define new extended data types. I chanced upon a library called cyberpandas which made an IP address a basic data type in pandas. I extended this to make IP networks a basic data type in pandas. This resulted in code that looked as follows:

熊猫允许用户定义新的扩展数据类型。 我偶然发现了一个名为Cyber​​pandas的图书馆，该图书馆将IP地址作为熊猫的基本数据类型。 我将其扩展为使IP网络成为熊猫的基本数据类型。 这导致代码如下所示：

route_df[‘prefix’] = route_df[‘prefix’].astype(‘ipnetwork’)
result = route_df[[‘vrf’, ‘prefix’]] \
         .query(“prefix.ipnet.supernet_of(‘{}’)”.format(dstaddr)) \
         .groupby(by=[‘vrf’]) \
         .max() \
         .dropna() \
         .reset_index()
result_df = result.merge(route_df)

The third line elegantly captures the checking if the prefix contains the address, and the fifth picks the entry with the longest prefix length. The same logic as the naive implementation, but this ought to perform better, I hoped. This also provides the LPM per VRF (think of VRF as a logical routing instance, sort of like a VLAN for IP). This model looks more like Pandas data pipeline code ought to look, no iterating with for loops explicitly over the entire route table. This also led to other benefits in basic route filtering and other IP address operations. So, we released Suzieq with this implementation.

第三行优雅地捕获了前缀是否包含地址的检查，而第五行则选择了前缀长度最长的条目。 我希望，逻辑与幼稚的实现相同，但是应该表现得更好。 这也提供了每个VRF的LPM(将VRF视为逻辑路由实例，有点像IP的VLAN)。 该模型看起来更像熊猫数据管道代码，无需在整个路由表上显式地进行for循环的迭代。 这还带来了基本路由过滤和其他IP地址操作的其他好处。 因此，我们发布了带有此实现的Suzieq。

It all worked well. Until it ran into the full Internet routing table. Donald Sharp, one of the key maintainers of the open source routing suite, FRR, had Suzieq collect the data from a router receiving the full Internet feed and provided me a copy of this data. This algorithm took close to two and a half minutes to perform the LPM!

一切都很好。 直到遇到完整的Internet路由表。 唐纳德·夏普(Donald Sharp)是开源路由套件FRR的主要维护者之一，他让Suzieq从接收了完整Internet提要的路由器中收集了数据，并向我提供了这些数据的副本。 该算法需要近两分半钟来执行LPM！

Investigating the code, I determined that the time taken was caused by two things: converting 800K prefixes into the IP network data type, and then searching through the entire 800K prefixes to find the longest prefix match. pandas had a different model for iterating over all the rows that was faster than manually iterating over the rows as shown in the pseudo code of the first fragment. That was to use the apply function. This code looked as follows:

通过研究代码，我确定花费的时间是由两件事引起的：将800K前缀转换为IP网络数据类型，然后搜索整个800K前缀以找到最长的前缀匹配项。 大熊猫具有不同的模型来遍历所有行，这比手动遍历行要快，如第一个片段的伪代码所示。 那是使用apply函数。 该代码如下所示：

route_df[‘prefixlen’] = int_df.prefix.str.split(‘/’).str[1]
match = route_df.apply(
lambda x, dstip: dstip.subnet_of(ip_network(x[‘prefix’])), args=(dstip, ), axis=1)
result_df = route_df.loc[match] \
                    .sort_values(‘prefixlen’, ascending=False) \
                    .drop_duplicates([‘vrf’])

The same logic as the naive code, but more in line with how Pandas best practices recommended. The fourth and fifth lines implement the equivalent of picking the longest prefix entry over all the selected ones. However, this reduced the time window from two and a half minutes to 1 minute 40 seconds. Better, but still way too long.

逻辑与朴素的代码相同，但更符合Pandas最佳做法的建议。 第四行和第五行实现了在所有选定前缀条目中选择最长前缀条目的等效功能。 但是，这将时间窗口从两分半钟减少到1分40秒。 更好，但是仍然太长了。

Python is not known for being particularly fast, but almost no other language has libraries such as pandas for data analysis. So, stuck with python I was. I had read that the trick to the best performance with pandas was to vectorize the operations. By vectorizing an operation, we’d be reducing it to something that another library, numpy, could perform. In our case, we’d have to reduce the longest prefix match to a set of bit operations that numpy could be used for. The longest prefix match can be reduced to:

Python并不是以速度特别快而著称，但是几乎没有其他语言拥有诸如pandas之类的用于数据分析的库。 因此，我被困在python中。 我已经读到，熊猫获得最佳性能的诀窍是对操作进行矢量化处理。 通过向量化操作，我们可以将其简化为另一个numpy库可以执行的操作。 在我们的例子中，我们必须将最长的前缀匹配减少为numpy可以使用的一组位操作。 最长的前缀匹配可以减少为：

convert the IP network address to a number 
construct the netmask as a number using the prefixlen
if (addr & netmask) == (prefix & netmask), then the address is a subnet of the prefix

In python with pandas, this ended up looking as follows:

在带有熊猫的python中，最终结果如下所示：

intaddr = route_df.prefix.str.split(‘/’).str[0] \
                  .map(lambda y: int(‘’.join([‘%02x’ % int(x)
                       for x in y.split(‘.’)]), 16))
netmask = route_df.prefixlen \
          .map(lambda x: (0xffffffff << (32 — x)) & 0xffffffff)
match = (dstip._ip & netmask) == (intaddr & netmask)
result_df = route_df.loc[match.loc[match].index] \
                    .sort_values(‘prefixlen’, ascending=False) \
                    .drop_duplicates([‘vrf’])

Essentially, the apply line has been reduced to the first 3 lines in the code fragment above. Even though it looks like intaddr. netmask and match are operating on a single value, they’re actually operating on all the rows of the route table. To someone used to standard programming techniques, this code looks a bit strange. But with this change, LPM was reduced to 2 seconds!

本质上，应用行已减少为上述代码段的前3行。 即使看起来像intaddr。 netmask和match在单个值上运行，它们实际上在路由表的所有行上运行。 对于习惯了标准编程技术的人来说，这段代码看起来有些奇怪。 但是通过此更改， LPM减少到了2秒！

Thus, we systematically reduced the LPM performance from close to 3 minutes to 2s for a full Internet routing table. Doing LPM over 6.5 million rows went from over 6 minutes to 4 seconds!

因此，对于完整的Internet路由表，我们系统地将LPM性能从近3分钟降低到2s。 在超过650万行的情况下执行LPM从6分钟以上缩短到4秒！

得到教训 (Lessons Learned)

Just for grins and to make this post more complete, I implemented the naive approach to see how well it’d work compared to the other approaches. To make the code produce the same result as the other solutions i.e. produce an LPM result for each host and VRF in a namespace, I modified the naive code to look as follows:

只是出于笑容并使这篇文章更加完整，我实现了幼稚的方法来查看它与其他方法相比效果如何。 为了使代码产生与其他解决方案相同的结果，即为命名空间中的每个主机和VRF产生LPM结果，我将朴素代码修改为如下所示：

dstip = ip_network(‘dstaddr’)
result = {}
max_plens = defaultdict(int)
for index, row in route_table.iterrows():
   rtentry = ip_network(row[‘prefix’])
   if dstip.subnet_of(rtentry):
      key = row[“vrf”]
      if rtentry.prefixlen > max_plens[key]:
         result[key] = row
         max_plens[key] = rtentry.prefixlenresult_df = pd.concat(list(result.values()), axis=1)

The timing for this was a surprising 1 min and 24 seconds, slightly faster than even the apply method. This surprised me.

这样做的时间令人惊讶地是1分24秒，甚至比应用方法还快。 这让我感到惊讶。

I then went ahead and implemented the same logic using itertuples() method of pandas, instead of using iterrows, which is supposed to be faster. It resulted in this code:

然后，我继续使用熊猫的itertuples()方法实现了相同的逻辑，而不是使用应该更快的iterrows。 结果是这样的：

dstip = ip_network(‘dstaddr’)
result = {}
max_plens = defaultdict(int)
for row in route_table.itertuples():
   rtentry = ip_network(row.prefix)
   if dstip.subnet_of(rtentry):
      key = row.vrf
      if rtentry.prefixlen > max_plens[key]:
         result[key] = row
         max_plens[key] = rtentry.prefixlenresult_df = pd.DataFrame(list(result.values()))

This was a surprisingly fast 9.76 seconds. Much faster than the apply method, and the next best solution to the vectorized answer. This goes against the grain of accepted wisdom, even if you assume that my attempt to make IP network a native data type in pandas was flawed maybe by my implementation.

这是惊人的快速9.76秒。 比apply方法快得多，是向量化答案的次佳解决方案。 即使您认为我的尝试使IP网络成为大熊猫中的本机数据类型的尝试存在缺陷，这也违背了公认的智慧。

“Premature optimization is the root of all evil” is a well-known maxim in software programming circles, a quote attributed to Sir Tony Hoare (the man behind the quicksort algorithm and communication sequential processes among other things). However, I’m also aware of the other famous quote on life by Will Rogers, “There are three kinds of men. The ones that learn by reading. The few who learn by observation.The rest of them have to pee on the electric fence for themselves.” I hoped to have learned by reading, but my observations in this particular case have differed slightly from the popular wisdom. Vectorization is clearly the fastest approach, in agreement with the accepted wisdom around pandas operations. It also is in agreement that itertuples() is faster than iterrows(). However, itertuples() is far faster than the apply() method. Furthermore, from a readability perspective, itertuples is far more readable than the vectorized version or the apply() version.

“过早的优化是万恶之源”，这是软件编程界众所周知的格言，这句话应归功于Tony Hoare爵士(快速排序算法和通信顺序处理背后的人)。 但是，我也知道威尔·罗杰斯(Will Rogers)关于生活的另一句名言：“男人有三种。 通过阅读学习的人。 少数通过观察学习的人。其余的人必须自己在电围栏上撒尿。” 我希望可以通过阅读来学习，但是我在这种特殊情况下的观察结果与流行的看法略有不同。 显然，矢量化是最快的方法，这与围绕熊猫运作的公认观点相一致。 同样，itertuples()比iterrows()更快。 但是，itertuples()比apply()方法快得多。 此外，从可读性的角度来看，迭代程序比矢量化版本或apply()版本更具可读性。

To summarize the results for the full Internet IPv4 routing table:

总结完整的Internet IPv4路由表的结果：

• vectorizing the solution: 2s
  向量化解决方案：2秒
• using itertuples(): 9.76s
  使用itertuples()：9.76秒
• using iterrows(): 1 min 24s
  使用iterrows()：1分24秒
• using apply(): 1 min 40s
  使用apply()：1分钟40秒

The plan at this point is that the next version of Suzieq will ship with the vectorized version of the LPM for IPv4 and the itertuples version for IPv6, till I can get the vectorized version working for IPv6 as well.

目前的计划是，Suzieq的下一个版本将随附用于IPv4的LPM的矢量化版本和用于IPv6的itertuples版本，直到我也可以使用适用于IPv6的矢量化版本为止。

翻译自: https://medium.com/the-elegant-network/implementing-ip-lpm-in-pandas-6983671caddb

lpm算法