A Fast Analysis of TCP Fast Open

A Fast Analysis of TCP Fast Open | qa | cafe

Part 3 of our series exploring TCP examines the TCP “Fast Open” option and what to look for when troubleshooting

TCP Fast Open (TFO) is an optional mechanism within TCP that lets endpoints that have established a full TCP connection in the past eliminate a round-trip of the handshake and send data right away. This speeds things up for endpoints that are going to keep talking to each other in the future and is especially beneficial on high-latency networks where time-to-first-byte is critical. 

Fast Open is defined in RFC 7413 which explains:

TFO allows data to be carried in the SYN and SYN-ACK packets and consumed by the receiving end during the initial connection handshake, and saves up to one full round-trip time (RTT) compared to the standard TCP, which requires a three-way handshake (3WHS) to complete before data can be exchanged.

Follow along with these sample captures

As always, we've gathered up the captures mentioned in this article into this collection over on CloudShark

You know you can make collections just like this with CloudShark! Learn more...

How does it work?

TCP Fast Open works by allowing the server to set a special “TFO” cookie to use for your later connections. The next time you connect, you can include data in the SYN packet, and the server can include its response data in the first ACK. Normally, the client and server would need to do a full TCP handshake every time before sending application data. The RFC goes into detail on the protocol sequence in section 3.

Interestingly, TLS 1.3 has a very similar feature for saving a round trip, called “0-RTT” or zero round trip, which we’ll go into more detail about in our series on TLS.

What does it look like?

Here’s an initial SYN packet from this example capture. Notice that the TCP Length (the Len: field) is zero, since there’s no data in the initial handshake:

CS Enterprise on cloudshark.org

 

If you expand the TCP Options you’ll see that the TCP Fast Open option is included (ALT or OPTION-Click to expand everything). Here the client sends the Fast Open Cookie Request option asking the server give it a TFO cookie for next time.

Here’s the resulting SYN+ACK from the server:

CS Enterprise on cloudshark.org

 

If you expand the TCP Options you’ll again see the TCP Fast Open option, this time including the cookie value of 037110cc0de73575

The TCP stream then continues as normal after that.

Second connection using the TFO cookie

So what happens when the same client makes a new connection to the same server using TCP Fast Open? Take a look at the first packet in this capture:

CS Enterprise on cloudshark.org

 

Even without expanding any fields, you immediately see two things:

• The TCP Length field (Len:) is bigger than zero.
• There’s HTTP data in the packet!

The server recognizes the valid TFO cookie, accepts the payload data, and replies with its SYN+ACK. In this case, the Ack: field value is set to the total length of the TCP payload (76 in this case).

CS Enterprise on cloudshark.org

 

Why use TCP Fast Open?

The biggest reason to use TFO is to get that first chunk of data faster. Our captures were on our local network so the results aren’t quite as dramatic as they would be over say a mobile network, but still significant. If you look at the timestamps of the packets in the first and second captures, the first capture doesn’t get an HTTP response back from the server until 0.00447 seconds have passed. In the second capture, it only takes 0.00184 seconds. That’s more than twice (2.42 times) as fast!

Trouble with TCP Fast Open

What happens when the TFO cookie is invalid or not supported? Section 4.2.2 of RFC 7413 gets really deep into the details of implementation and handling of error cases. Here are some scenarios at the packet layer of what you might see.

Server Reboot

If the server ends up rebooting or clears its TFO cookies for any reason, the client will still try to use a cookie that is now invalid. Take a look at the SYN from the client in this capture:

CS Enterprise on cloudshark.org

 

The client still tries to include data with the initial SYN, and uses the old cookie value. However, the SYN+ACK from the server is now different:

CS Enterprise on cloudshark.org

 

You can see that the value of Ack: is 1 (not 76) indicating that the additional payload wasn’t accepted. However, the server includes a new TFO cookie in the options with a value of 0290a10f721b8d38.

Take a look at the next packet after the SYN+ACK in that capture:

CS Enterprise on cloudshark.org

 

Here we see that the client sent the same data in the initial SYN, but now again in the third packet of the normal 3-way handshake. CloudShark marks this as a retransmission as well because the data had already been sent in the first packet.

This might look strange, as in the normal flow of TCP, it’s usually:

SYN–>SYN+ACK–>ACK–>Data Packet

You can clearly see the difference in this ladder diagram of the conversation.

 

Retransmitting the data Section 4.2.2 of RFC 7413 states "The client SHOULD retransmit any unacknowledged data in the first ACK packet in step 2. The data exchange will start after the handshake like a regular TCP Connection. 

Server stops supporting TFO

What happens if the server decides to stop accepting TFO? Take a look at the ladder diagram of this conversation.

This example is similar to the one above but the server does not respond with a new TFO cookie. In this case, since the client can still connect, it continues to try using the cookie. We aren’t getting any benefits from TFO, but this isn’t causing any problems either.

Can’t connect using TFO

What if a device along the path doesn’t understand the TFO option and starts blocking the client’s connection attempts? What if the client’s traffic is taking a new route to the server, or an older piece of equipment was pulled from the “back closet” and added to the network? Sending SYN packets with a TCP length > 0 is unusual, and could cause issues with older devices, including dropping the packet!

Take a look at the 11th frame of this capture:

CS Enterprise on cloudshark.org

 

Here the client tries to connect and includes a cookie, but this packet is never gets a response. One second later it is retransmitted:

CS Enterprise on cloudshark.org

Unlike before, the retransmission does not contain the TFO option or additional data, just in case that’s what’s causing the packet to get dropped before the server receives it.

How do I find TFO in my own traffic?

Have you seen TCP Fast Open traffic in your captures? Here are some display filters to help you find samples of this in your own files or when diagnosing connection problems:

• Filter for option: tcp.option_kind == 34
• Filter for request: tcp.options.tfo.request
• Filter for tcp server: tcp.options.tfo.cookie
• Filter for tcp syn packet using a cookie: tcp.flags.syn && tcp.payload && tcp.options.tfo.cookie

Remember in CloudShark you can also use the DeepSearch feature to scan your capture list for the presence of TFO. Captures that have packets matching these filters will be highlighted for you.

Conclusions

Did we say this was going to be a quick and easy post about TCP Fast Open? It turns out even this feature can get pretty complicated, but like window scaling, TFO demonstrates the flexibility of TCP to evolve to the networking world’s changing needs.

Do you have a scenario with TCP Fast Open that has looked strange? Feel free to send it our way or use CloudShark collections to help tell the whole story.

We’ll continue our TCP series with other topics, but if you have something you’d love to learn about or see outlined using live captures, let us know!