RDO Havana Neutron&GRE L3 Layer Troubleshooting

This post follows up http://lxer.com/module/newswire/view/200975/index.html
For Basic Neutron Networking concepts,please, view developer.rackspace.com

Qrouters namespaces external and internal interfaces are mapped to `ovs-vsctl show` output. Qdhcps namespaces interfaces are also mapped to `ovs-vsctl show` output. Traffic control is done via tcpdump on external and internal network interfaces of Qrouter's namespace associated with Network attached via external gateway to router with interface for private sub net having Ubuntu Trusty Server instance running `apt-get install` commands.

[root@dfw02 ~(keystone_admin)]$ neutron router-list
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| id                                   | name    | external_gateway_info                                                       |
+--------------------------------------+---------+-----------------------------------------------------------------------------+
| 86b3008c-297f-4301-9bdc-766b839785f1 | router2 | {"network_id": "780ce2f3-2e6e-4881-bbac-857813f9a8e0", "enable_snat": true} |
| bf360d81-79fb-4636-8241-0a843f228fc8 | router1 | {"network_id": "780ce2f3-2e6e-4881-bbac-857813f9a8e0", "enable_snat": true} |
+--------------------------------------+---------+-----------------------------------------------------------------------------+

[root@dfw02 ~(keystone_admin)]$ ip netns list | grep 86b3008c-297f-4301-9bdc-766b839785f1
qrouter-86b3008c-297f-4301-9bdc-766b839785f1

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

2: qr-e031db6b-d0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:83:90:90 brd ff:ff:ff:ff:ff:ff
    inet 40.0.0.1/24 brd 40.0.0.255 scope global qr-e031db6b-d0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe83:9090/64 scope link
       valid_lft forever preferred_lft forever

3: qg-9c090153-08: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:b2:24:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.114/24 brd 192.168.1.255 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.105/32 brd 192.168.1.105 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.104/32 brd 192.168.1.104 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.102/32 brd 192.168.1.102 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.106/32 brd 192.168.1.106 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feb2:24bc/64 scope link
       valid_lft forever preferred_lft forever

Getting list of all namespaces on Neutron Server

[root@dfw02 ~(keystone_admin)]$ ip netns list

qrouter-86b3008c-297f-4301-9bdc-766b839785f1
qrouter-bf360d81-79fb-4636-8241-0a843f228fc8
qdhcp-1eea88bb-4952-4aa4-9148-18b61c22d5b7
qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-86b3008c-297f-4301-9bdc-766b839785f1 ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

Router's internal interface

2: qr-e031db6b-d0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:83:90:90 brd ff:ff:ff:ff:ff:ff
    inet 40.0.0.1/24 brd 40.0.0.255 scope global qr-e031db6b-d0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe83:9090/64 scope link
       valid_lft forever preferred_lft forever

Router's external interface with the floating IPs

3: qg-9c090153-08: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:b2:24:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.114/24 brd 192.168.1.255 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.104/32 brd 192.168.1.104 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.102/32 brd 192.168.1.102 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet 192.168.1.105/32 brd 192.168.1.105 scope global qg-9c090153-08
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feb2:24bc/64 scope link
       valid_lft forever preferred_lft forever

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show| grep e031db6b-d0
        Port "tape031db6b-d0"
            Interface "tape031db6b-d0"

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show| grep 9c090153-08
        Port "tap9c090153-08"
            Interface "tap9c090153-08"

[root@dfw02 ~(keystone_admin)]$ ip netns exec qrouter-bf360d81-79fb-4636-8241-0a843f228fc8 ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

Router's internal interface

2: qr-f933e768-42: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:6a:d3:f0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-f933e768-42
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe6a:d3f0/64 scope link
       valid_lft forever preferred_lft forever

Router's external   interface with the floating IPs

3: qg-54e34740-87: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:00:9a:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.100/24 brd 192.168.1.255 scope global qg-54e34740-87
       valid_lft forever preferred_lft forever
    inet 192.168.1.103/32 brd 192.168.1.103 scope global qg-54e34740-87
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe00:9a0d/64 scope link
       valid_lft forever preferred_lft forever

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show| grep f933e768-42
        Port "tapf933e768-42"
            Interface "tapf933e768-42"

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show| grep 54e34740-87
        Port "tap54e34740-87"
            Interface "tap54e34740-87"

[root@dfw02 ~(keystone_admin)]$ ip netns list
qrouter-86b3008c-297f-4301-9bdc-766b839785f1
qrouter-bf360d81-79fb-4636-8241-0a843f228fc8
qdhcp-1eea88bb-4952-4aa4-9148-18b61c22d5b7
qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b

[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-1eea88bb-4952-4aa4-9148-18b61c22d5b7 ip a

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

2: ns-40dd712c-e4: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:93:44:f8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global ns-40dd712c-e4
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe93:44f8/64 scope link
       valid_lft forever preferred_lft forever

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show| grep 40dd712c-e4
        Port "tap40dd712c-e4"
            Interface "tap40dd712c-e4"
[root@dfw02 ~(keystone_admin)]$ ip netns exec qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b ip a
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

2: ns-343b0090-24: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:01:8b:55 brd ff:ff:ff:ff:ff:ff
    inet 40.0.0.3/24 brd 40.0.0.255 scope global ns-343b0090-24
       valid_lft forever povs-vsctl showreferred_lft forever
    inet6 fe80::f816:3eff:fe01:8b55/64 scope link
       valid_lft forever preferred_lft forever

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show| grep 343b0090-24
        Port "tap343b0090-24"
            Interface "tap343b0090-24"

***************************************
Now get `ovs-vsctl show` output
***************************************

[root@dfw02 ~(keystone_admin)]$ ovs-vsctl show
7d78d536-3612-416e-bce6-24605088212f
    Bridge br-ex
        Port "p37p1"
            Interface "p37p1"
        Port "tap54e34740-87" <= qrouter-bf360d81-79fb-4636-8241-0a843f228fc8 <=Router's external interface attached here
            Interface "tap54e34740-87"
        Port br-ex
            Interface br-ex
                type: internal
        Port "tap9c090153-08" <= qrouter-86b3008c-297f-4301-9bdc-766b839785f1      <= Router's external interface   attached here
            Interface "tap9c090153-08"
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tape031db6b-d0" <= qrouter-86b3008c-297f-4301-9bdc-766b839785f1                     <= Routers internal interface attached here
            tag: 1
            Interface "tape031db6b-d0"
        Port "tapf933e768-42"   <= qrouter-bf360d81-79fb-4636-8241-0a843f228fc8    <=Routers internal interface attached here
             tag: 2
            Interface "tapf933e768-42"
        Port "tap40dd712c-e4"   <= qdhcp-1eea88bb-4952-4aa4-9148-18b61c22d5b7
            tag: 2
            Interface "tap40dd712c-e4"
        Port "tap343b0090-24"   <= qdhcp-426bb226-0ab9-440d-ba14-05634a17fb2b
            tag: 1
            Interface "tap343b0090-24"
    Bridge br-tun
        Port "gre-2"
            Interface "gre-2"
                type: gre
                options: {in_key=flow, local_ip="192.168.1.127", out_key=flow, remote_ip="192.168.1.137"}
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.0.1"

******************************************************************************
Testing qrouter-86b3008c-297f-4301-9bdc-766b839785f1 namespace
******************************************************************************
L3 routing test on running instance of Ubuntu 14.04 (private IP 40.0.0.5 , floating IP 192.168.1.105) doing ping 83.221.202.254

Capturing packages on interfaces

# tcpdump -i tape031db6b-d0 -vv | tee logdump.int
# tcpdump -i tap9c090153-08 -vv | tee logdump.ext

*********************
Content logdump.int
*********************

19:35:02.343287 IP (tos 0x0, ttl 64, id 11964, offset 0, flags [DF], proto ICMP (1), length 84)
    40.0.0.5 > ns2.donpac.ru: ICMP echo request, id 11817, seq 64, length 64
19:35:02.344662 IP (tos 0x0, ttl 59, id 43283, offset 0, flags [DF], proto ICMP (1), length 84)
    ns2.donpac.ru > 40.0.0.5: ICMP echo reply, id 11817, seq 64, length 64
19:35:03.345161 IP (tos 0x0, ttl 64, id 11965, offset 0, flags [DF], proto ICMP (1), length 84)
    40.0.0.5 > ns2.donpac.ru: ICMP echo request, id 11817, seq 65, length 64
19:35:03.346490 IP (tos 0x0, ttl 59, id 44767, offset 0, flags [DF], proto ICMP (1), length 84)
    ns2.donpac.ru > 40.0.0.5: ICMP echo reply, id 11817, seq 65, length 64
19:35:04.347239 IP (tos 0x0, ttl 64, id 11966, offset 0, flags [DF], proto ICMP (1), length 84)
    40.0.0.5 > ns2.donpac.ru: ICMP echo request, id 11817, seq 66, length 64
19:35:04.348662 IP (tos 0x0, ttl 59, id 46565, offset 0, flags [DF], proto ICMP (1), length 84)
    ns2.donpac.ru > 40.0.0.5: ICMP echo reply, id 11817, seq 66, length 64
19:35:05.353605 IP (tos 0x0, ttl 64, id 11967, offset 0, flags [DF], proto ICMP (1), length 84)

*********************
Content logdump.ext
*********************

19:35:51.434781 IP (tos 0x0, ttl 63, id 12013, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.105 > ns2.donpac.ru: ICMP echo request, id 11817, seq 113, length 64
19:35:51.435691 IP (tos 0x0, ttl 60, id 59160, offset 0, flags [DF], proto ICMP (1), length 84)
    ns2.donpac.ru > 192.168.1.105: ICMP echo reply, id 11817, seq 113, length 64
19:35:52.436598 IP (tos 0x0, ttl 63, id 12014, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.105 > ns2.donpac.ru: ICMP echo request, id 11817, seq 114, length 64
19:35:52.437575 IP (tos 0x0, ttl 60, id 60728, offset 0, flags [DF], proto ICMP (1), length 84)
    ns2.donpac.ru > 192.168.1.105: ICMP echo reply, id 11817, seq 114, length 64
19:35:53.438083 IP (tos 0x0, ttl 63, id 12015, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.105 > ns2.donpac.ru: ICMP echo request, id 11817, seq 115, length 64
19:35:53.439045 IP (tos 0x0, ttl 60, id 62490, offset 0, flags [DF], proto ICMP (1), length 84)
    ns2.donpac.ru > 192.168.1.105: ICMP echo reply, id 11817, seq 115, length 64
19:35:53.883506 IP (tos 0x10, ttl 63, id 31584, offset 0, flags [DF], proto TCP (6), length 152)
**********************************************************************************
L3 routing test on running instance of Ubuntu 14.04 (private IP 40.0.0.4 , floating IP 192.168.1.106) doing `apt-get install xorg fluxbox gnome-terminal`

# tcpdump -i tape031db6b-d0 -vv | tee logdump01.int
**********************************************************************************
22:33:12.066339 IP (tos 0x0, ttl 64, id 12998, offset 0, flags [DF], proto TCP (6), length 52)
    40.0.0.4.40608 > ragana.canonical.com.http: Flags [.], cksum 0xe131 (correct), seq 16007, ack 20856182, win 6835, options [nop,nop,TS val 1137497 ecr 1286593365], length 0
22:33:12.066347 IP (tos 0x0, ttl 48, id 40522, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0x81c5 (correct), seq 20868800:20870202, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.066465 IP (tos 0x0, ttl 48, id 40523, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0x9e67 (correct), seq 20870202:20871604, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.066585 IP (tos 0x0, ttl 48, id 40524, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0xada9 (correct), seq 20871604:20873006, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.066709 IP (tos 0x0, ttl 64, id 12999, offset 0, flags [DF], proto TCP (6), length 52)
    40.0.0.4.40608 > ragana.canonical.com.http: Flags [.], cksum 0xd63d (correct), seq 16007, ack 20858986, win 6835, options [nop,nop,TS val 1137497 ecr 1286593365], length 0
22:33:12.066716 IP (tos 0x0, ttl 48, id 40525, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0x466b (correct), seq 20873006:20874408, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.066835 IP (tos 0x0, ttl 48, id 40526, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0xbd64 (correct), seq 20874408:20875810, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.066956 IP (tos 0x0, ttl 64, id 13000, offset 0, flags [DF], proto TCP (6), length 52)
    40.0.0.4.40608 > ragana.canonical.com.http: Flags [.], cksum 0xcb49 (correct), seq 16007, ack 20861790, win 6835, options [nop,nop,TS val 1137497 ecr 1286593365], length 0
22:33:12.066963 IP (tos 0x0, ttl 48, id 40527, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0x6470 (correct), seq 20875810:20877212, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.067087 IP (tos 0x0, ttl 64, id 13001, offset 0, flags [DF], proto TCP (6), length 52)
    40.0.0.4.40608 > ragana.canonical.com.http: Flags [.], cksum 0xc055 (correct), seq 16007, ack 20864594, win 6835, options [nop,nop,TS val 1137497 ecr 1286593365], length 0
22:33:12.067094 IP (tos 0x0, ttl 48, id 40528, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 40.0.0.4.40608: Flags [.], cksum 0xa2d0 (correct), seq 20877212:20878614, ack 16007, win 434, options [nop,nop,TS val 1286593365 ecr 1137460], length 1402
22:33:12.067299 IP (tos 0x0, ttl 64, id 13002, offset 0, flags [DF], proto TCP (6), length 52)
    40.0.0.4.40608 > ragana.canonical.com.http: Flags [.], cksum 0xb561 (correct), seq 16007, ack 20867398, win 6835, options [nop,nop,TS val 1137497 ecr 1286593365], length 0
22:33:12.067548 IP (tos 0x0, ttl 64, id 13003, offset 0, flags [DF], proto TCP (6), length 52)
    40.0.0.4.40608 > ragana.canonical.com.http: Flags [.], cksum 0xaa6d (correct), seq 16007, ack 20870202, win 6835, options [nop,nop,TS val 1137497 ecr 1286593365], length 0
22:33:12.067799 IP (tos 0x0, ttl 64, id 13004, offset 0, flags [DF], proto TCP (6), length 52)

# tcpdump -i tap9c090153-08 -vv | tee logdump01.ext

22:33:06.010493 IP (tos 0x0, ttl 49, id 39783, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0x5fc1 (correct), seq 19841501:19842903, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.010612 IP (tos 0x0, ttl 49, id 39784, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0x3d2b (correct), seq 19842903:19844305, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.010734 IP (tos 0x0, ttl 49, id 39785, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0x6be5 (correct), seq 19844305:19845707, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.010756 IP (tos 0x0, ttl 63, id 12619, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.106.40608 > ragana.canonical.com.http: Flags [.], cksum 0xf743 (correct), seq 14014, ack 19833089, win 6835, options [nop,nop,TS val 1135984 ecr 1286591851], length 0
22:33:06.010862 IP (tos 0x0, ttl 49, id 39786, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0xbe54 (correct), seq 19845707:19847109, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.010984 IP (tos 0x0, ttl 49, id 39787, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0x1401 (correct), seq 19847109:19848511, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.011005 IP (tos 0x0, ttl 63, id 12620, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.106.40608 > ragana.canonical.com.http: Flags [.], cksum 0xec4f (correct), seq 14014, ack 19835893, win 6835, options [nop,nop,TS val 1135984 ecr 1286591851], length 0
22:33:06.011111 IP (tos 0x0, ttl 49, id 39788, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0x6967 (correct), seq 19848511:19849913, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.011234 IP (tos 0x0, ttl 49, id 39789, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0xa08b (correct), seq 19849913:19851315, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.011256 IP (tos 0x0, ttl 63, id 12621, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.106.40608 > ragana.canonical.com.http: Flags [.], cksum 0xe15b (correct), seq 14014, ack 19838697, win 6835, options [nop,nop,TS val 1135984 ecr 1286591851], length 0
22:33:06.011360 IP (tos 0x0, ttl 49, id 39790, offset 0, flags [DF], proto TCP (6), length 1454)
    ragana.canonical.com.http > 192.168.1.106.40608: Flags [.], cksum 0x651d (correct), seq 19851315:19852717, ack 14014, win 392, options [nop,nop,TS val 1286591851 ecr 1135946], length 1402
22:33:06.011483 IP (tos 0x0, ttl 49, id 39791, offset 0, flags [DF], proto TCP (6), length 1454)