我们来剖析下某地数据中心,案例已经去掉敏感信息。
某地数据中心目前核心华为CE12808上联电信出口带宽200G,预留200G出口带宽;由100+台服务器组成的CDN网络节点;另外60+台服务器组成的节点2
华为CE12808是机框式交换机,可以插8块业务板卡
1、首先来看核心上联机房的规划
40GE1/0/0使用40G一分四模块中两个10G上联机房
40GE1/0/1使用40G一分四模块中两个10G上联机房
40GE1/0/2使用40G一分四模块中两个10G上联机房
40GE2/0/0使用40G一分四模块中两个10G上联机房
40GE2/0/1使用40G一分四模块中两个10G上联机房
40GE2/0/2使用40G一分四模块中两个10G上联机房
40GE3/0/0使用40G一分四模块中两个10G上联机房
40GE3/0/1使用40G一分四模块中两个10G上联机房
40GE4/0/0使用40G一分四模块中两个10G上联机房
40GE4/0/1使用40G一分四模块中两个10G上联机房
具体对应的交换机配置部分:
!将以下40GE端口拆分为4个10G端口,因为上联机房,对端使用了10G单模模块,所以需要将端口拆分来和机房做互联
port split dimension interface 40GE1/0/0 to 40GE1/0/4
port split dimension interface 40GE2/0/0 to 40GE2/0/4
port split dimension interface 40GE3/0/0 to 40GE3/0/4
port split dimension interface 40GE4/0/0 to 40GE4/0/4
port split dimension interface 40GE8/0/0
#!创建聚合端口eth-trunk1 并将其修改为3层端口,我方的互联IP1:192.168.10.2/30
interface Eth-Trunk1
undo portswitch
description UPLink_DX
ipv6 enable
ip address 192.168.10.2 255.255.255.252
mode lacp-static
#!将端口40GE1/0/0:1-4 40GE1/0/1:1 40GE2/0/0:1-4 40GE2/0/1:1 这10个端口加入端口聚合组eth-trunk1 ,第一组互联配置基本完毕,这组互联上下行带宽都是100Gbps
interface 40GE1/0/0:1
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE1/0/0:2
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE1/0/0:3
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE1/0/0:4
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE1/0/1:1
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE2/0/0:1
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE2/0/0:2
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE2/0/0:3
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE2/0/0:4
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
interface 40GE2/0/1:1
description UPLink_DX_A
eth-trunk 1
device transceiver 40GBASE-FIBER
#!第二组上联和上面的配置一样
interface Eth-Trunk2
undo portswitch
description UPLink_DX
ip address 192.168.10.6 255.255.255.252
mode lacp-static
interface 40GE3/0/0:1
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE3/0/0:2
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE3/0/0:3
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE3/0/0:4
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE3/0/1:1
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE4/0/0:1
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE4/0/0:2
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE4/0/0:3
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE4/0/0:4
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
interface 40GE4/0/1:1
description UPLink_DX_B
eth-trunk 2
device transceiver 40GBASE-FIBER
!最后是增加2条默认路由 指向电信对端,这样上联就配置完毕
ip route-static 0.0.0.0 0.0.0.0 Eth-Trunk1 192.168.10.1
ip route-static 0.0.0.0 0.0.0.0 Eth-Trunk2 192.168.10.5
这个实例说明数据中心可以通过像华为CE12808这样的多板卡设备来绑定大于8条链路,华为文档说最大支持绑定128条,并且状态都是select,去往ISP的流量将在这20条物理链路上实现负载均衡(CE12800系列交换机的hash取值范围是1~13。缺省值均为1。参照华为产品文档)
因为华为CE12808没有做堆叠(那要投资翻倍),所以现在我们直接看12808下联交换机是怎么连的,
#!新建端口聚合组eth-trunk5 ,用来连接一台IP91.1结尾的spine交换机。将该端口修改为3层端口,分配IP为172.20.0.254/30 和 IPV6地址FC00::172:20:0:254/127(site-local地址),该端口聚合组有40GE1/0/6-7 40GE2/0/6-7 40GE3/0/6-7 40GE4/0/6-7 一共8个40G端口做聚合,上行带宽320G下行带宽320G。
interface Eth-Trunk5
undo portswitch
description DownLink_91.1
ipv6 enable
ip address 172.20.0.254 255.255.255.254
ipv6 address FC00::172:20:0:254/127
mode lacp-static
interface 40GE1/0/6
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE1/0/7
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE2/0/6
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE2/0/7
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE3/0/6
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE3/0/7
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE4/0/6
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
interface 40GE4/0/7
description DownLink_91.1
eth-trunk 5
device transceiver 40GBASE-FIBER
#!对应的规划如下图
#!接下来是华为CE12808下联2台做了堆叠的spine交换机(管理IP结尾 4.250),非常有意思的是,这次是将40GE1/0/8连其中一台堆叠,起3层地址,40GE2/0/8连接另外1台堆叠交换机,起另外一个3层地址;而不是之前那样2个端口做聚合组来和某一台spine交换机互联;这样一来相当于核心和spine之间有2条逻辑链路,而不是像之前的1条;下联交换机的路由表里,也会出现2条等价路由,这样做的好处就是对流经每条链路的流量更加容易实现控制(3层IP)。--------纠错,CISCO_N3K_C3164Q 40G 是一个2层组装在一起的交换机,上层32个40G端口,下层32个40G端口,单芯片1.2T,用了2颗芯片;你可以 把它理解为是1组堆叠形成的交换机,某些情况下和同一个邻居建立2条三层链路,考虑到其中1条跑组播协议,另外1条跑单播协议的需要。
interface 40GE1/0/8
undo portswitch
description DownLink_4.250
ipv6 enable
ip address 172.20.0.252 255.255.255.254
ipv6 address FC00::172:20:0:252/127
device transceiver 40GBASE-FIBER
interface 40GE2/0/8
undo portswitch
description DownLink_4.250
ipv6 enable
ip address 172.20.0.250 255.255.255.254
ipv6 address FC00::172:20:0:250/127
device transceiver 40GBASE-FIBER
#!同理 40GE1/0/9和 40GE2/0/9 又分别下联了2台做了堆叠的spine交换机(实际就是一台胶水CISCO_N3K_C3164Q 40G),结管理IP结尾4.252
interface 40GE1/0/9
undo portswitch
description DownLink_4.252
ipv6 enable
ip address 172.20.0.244 255.255.255.254
ipv6 address FC00::172:20:0:244/127
device transceiver 40GBASE-FIBER
interface 40GE2/0/9
undo portswitch
description DownLink_4.252
ipv6 enable
ip address 172.20.0.242 255.255.255.254
ipv6 address FC00::172:20:0:242/127
device transceiver 40GBASE-FIBER
#!同理 40GE3/0/8和 40GE4/0/8 又分别下联了2台做了堆叠的spine交换机(实际就是一台胶水CISCO_N3K_C3164Q 40G),管理IP结尾4.251
interface 40GE3/0/8
undo portswitch
description DownLink_4.251
ipv6 enable
ip address 172.20.0.248 255.255.255.254
ipv6 address FC00::172:20:0:248/127
device transceiver 40GBASE-FIBER
interface 40GE4/0/8
undo portswitch
description DownLink_4.251
ipv6 enable
ip address 172.20.0.246 255.255.255.254
ipv6 address FC00::172:20:0:246/127
device transceiver 40GBASE-FIBER
#!同理 40GE3/0/9和 40GE4/0/9又分别下联了2台做了堆叠的spine交换机(实际就是一台胶水CISCO_N3K_C3164Q 40G),管理IP结尾4.253
interface 40GE3/0/9
undo portswitch
description DownLink_4.253
ipv6 enable
ip address 172.20.0.240 255.255.255.254
ipv6 address FC00::172:20:0:240/127
device transceiver 40GBASE-FIBER
interface 40GE4/0/9
undo portswitch
description DownLink_4.253
ipv6 enable
ip address 172.20.0.238 255.255.255.254
ipv6 address FC00::172:20:0:238/127
device transceiver 40GBASE-FIBER
#!接下来,需要跑一个路由协议,来承载数据中心数以万计的路由条目(每台spine交换机都会宣告自己底下所连服务器和虚拟机的网段),厂商推荐的路由协议是BGP,可以承载10万条以上的路由,而BGP路由协议是4层协议,需要首先3层可达才能建立BGP邻居关系,这里的规划是核心华为CE12808通过直连接口和所有的spine交换机建立EBGP邻居关系,并且不和非直连交换机建立BGP邻居关系。
本地AS号是65000,所有上面的直连的Sspine邻居AS号是65010 ;CACHE 91.1 AS号是65011;
#!底下这一段是核心和spine交换机,cache交换机建立IPV4 ebgp关系的配置;其中 bfd是双向链路检测协议,用来快速检测链路状态,并通知路由协议收敛,达到缩短原路由协议收敛时间的,这里定义bfd发包最短时间间隔200ms,收包最短时间间隔200ms。但并非是必须配置的。
peer 172.20.0.239 bfd enable
bgp 65000
router-id 115.223.4.254
peer 172.20.0.239 as-number 65010
peer 172.20.0.239 description Spine4_4.253
peer 172.20.0.239 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.239 bfd enable
peer 172.20.0.241 as-number 65010
peer 172.20.0.241 description Spine4_4.253
peer 172.20.0.241 bfd min-tx-interval 250 min-rx-interval 250
peer 172.20.0.241 bfd enable
peer 172.20.0.243 as-number 65010
peer 172.20.0.243 description Spine3_4.252
peer 172.20.0.243 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.243 bfd enable
peer 172.20.0.245 as-number 65010
peer 172.20.0.245 description Spine3_4.252
peer 172.20.0.245 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.245 bfd enable
peer 172.20.0.247 as-number 65010
peer 172.20.0.247 description Spine2_4.251
peer 172.20.0.247 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.247 bfd enable
peer 172.20.0.249 as-number 65010
peer 172.20.0.249 description Spine2_4.251
peer 172.20.0.249 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.249 bfd enable
peer 172.20.0.251 as-number 65010
peer 172.20.0.251 description Spine1_4.250
peer 172.20.0.251 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.251 bfd enable
peer 172.20.0.253 as-number 65010
peer 172.20.0.253 description Spine1_4.250
peer 172.20.0.253 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.253 bfd enable
peer 172.20.0.255 as-number 65011
peer 172.20.0.255 description Cache_91.1
peer 172.20.0.255 bfd min-tx-interval 200 min-rx-interval 200
peer 172.20.0.255 bfd enable
peer FC00::172:20:0:239 as-number 65010
peer FC00::172:20:0:239 description Spine3_4.253
peer FC00::172:20:0:239 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:239 bfd enable
peer FC00::172:20:0:241 as-number 65010
peer FC00::172:20:0:241 description Spine3_4.253
peer FC00::172:20:0:241 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:241 bfd enable
peer FC00::172:20:0:243 as-number 65010
peer FC00::172:20:0:243 description Spine3_4.252
peer FC00::172:20:0:243 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:243 bfd enable
peer FC00::172:20:0:245 as-number 65010
peer FC00::172:20:0:245 description Spine3_4.252
peer FC00::172:20:0:245 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:245 bfd enable
peer FC00::172:20:0:247 as-number 65010
peer FC00::172:20:0:247 description Spine2_4.251
peer FC00::172:20:0:247 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:247 bfd enable
peer FC00::172:20:0:249 as-number 65010
peer FC00::172:20:0:249 description Spine2_4.251
peer FC00::172:20:0:249 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:249 bfd enable
peer FC00::172:20:0:251 as-number 65010
peer FC00::172:20:0:251 description Spine1_4.250
peer FC00::172:20:0:251 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:251 bfd enable
peer FC00::172:20:0:253 as-number 65010
peer FC00::172:20:0:253 description Spine1_4.250
peer FC00::172:20:0:253 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:253 bfd enable
peer FC00::172:20:0:255 as-number 65011
peer FC00::172:20:0:255 description Cache_91.1
peer FC00::172:20:0:255 bfd min-tx-interval 200 min-rx-interval 200
peer FC00::172:20:0:255 bfd enable
#!接下来定义IPV4路由策略和一些邻居关系建立后的行为,maximum load-balancing 18,意思是ebgp打开链路负载均衡,并且最大最大负载均衡下一跳邻居可以有18个, peer A.B.0.239 enable 是激活邻居EBGP IPV4协议栈;peer A.B.0.239 advertise-community 发送community属性,用于将来优化选路规则; peer A.B.0.239 default-route-advertise ;通过BGP协议给邻居下发IPV4的默认路由,其他语句不是必须。
ipv4-family unicast
network 115.223.4.254 255.255.255.255
maximum load-balancing 18
peer 172.20.0.239 enable
peer 172.20.0.239 advertise-community
peer 172.20.0.239 default-route-advertise
peer 172.20.0.239 keep-all-routes
peer 172.20.0.241 enable
peer 172.20.0.241 advertise-community
peer 172.20.0.241 default-route-advertise
peer 172.20.0.241 keep-all-routes
peer 172.20.0.243 enable
peer 172.20.0.243 advertise-community
peer 172.20.0.243 default-route-advertise
peer 172.20.0.243 keep-all-routes
peer 172.20.0.245 enable
peer 172.20.0.245 advertise-community
peer 172.20.0.245 default-route-advertise
peer 172.20.0.245 keep-all-routes
peer 172.20.0.247 enable
peer 172.20.0.247 advertise-community
peer 172.20.0.247 default-route-advertise
peer 172.20.0.247 keep-all-routes
peer 172.20.0.249 enable
peer 172.20.0.249 advertise-community
peer 172.20.0.249 default-route-advertise
peer 172.20.0.249 keep-all-routes
peer 172.20.0.251 enable
peer 172.20.0.251 advertise-community
peer 172.20.0.251 default-route-advertise
peer 172.20.0.251 keep-all-routes
peer 172.20.0.253 enable
peer 172.20.0.253 advertise-community
peer 172.20.0.253 default-route-advertise
peer 172.20.0.253 keep-all-routes
peer 172.20.0.255 enable
peer 172.20.0.255 advertise-community
peer 172.20.0.255 default-route-advertise
peer 172.20.0.255 keep-all-routes
查看下路由表,你会发现通往A.B.2.0/24公网地址的路径有8条,就是之前那8个EBGP邻居,他们都向核心宣告了到达A.B.2.0/24公网的路由
#!具体对应的规划如下图,A1机柜2台做堆叠,B1机柜2台做堆叠;一共8条3层链路
![在这里插入图片描述](https://img-blog.csdnimg.cn/direct/802dfd4aacc34e0daf634243334dabe2.png#pic_center
#!同理开启邻居的IPV6 协议扎,并下发IPV6的默认路由给邻居
ipv6-family unicast
peer FC00::172:20:0:239 enable
peer FC00::172:20:0:239 default-route-advertise
peer FC00::172:20:0:241 enable
peer FC00::172:20:0:241 default-route-advertise
peer FC00::172:20:0:243 enable
peer FC00::172:20:0:243 default-route-advertise
peer FC00::172:20:0:245 enable
peer FC00::172:20:0:245 default-route-advertise
peer FC00::172:20:0:247 enable
peer FC00::172:20:0:247 default-route-advertise
peer FC00::172:20:0:249 enable
peer FC00::172:20:0:249 default-route-advertise
peer FC00::172:20:0:251 enable
peer FC00::172:20:0:251 default-route-advertise
peer FC00::172:20:0:253 enable
peer FC00::172:20:0:253 default-route-advertise
peer FC00::172:20:0:255 enable
peer FC00::172:20:0:255 default-route-advertis
简单来说接入层交换机使用华为的6851交换机,创建业务VLAN ,并配置IP地址,作为底下服务器的网关,通过OSPF 等路由协议将路由重分布进BGP传给汇聚层交换机,汇聚层再宣告给EBGP核心,接入层以下是2层,接入层包括以上都是3层,这样的设计将有效减少2层广播,组播带来的消极影响。
篇幅有限,献上手绘结构图,接入层技术本期先不细讲;下期讲讲数据中心的收敛比的合理设计