TCPCopy - A TCP Stream Replay Tool

TCPCopy - A TCP Stream Replay Tool

TCPCopy is a TCP stream replay tool to support real testing of Internet server applications.

An online request replication tool, also a tcp stream replay tool, fit for real testing, performance testing, stability testing, stress testing, load testing, smoke testing.

TCPCopy 是一种请求复制（所有基于TCP的packets）工具，可以把在线流量导入到测试系统中去。

TCPCopy 是一个分布式在线压力测试工具，可以将线上流量拷贝到测试机器，实时地模拟线上环境，达到在程序不上线的情况下实时承担线上流量的效果，尽早发现 bug，增加上线信心。

tcpcopy 的优势在于其实时性及真实性，除了少量的丢包，完全拷贝线上流量到测试机器，真实的模拟线上流量的变化规律。

Although the real live flow is important for the test of Internet server applications, it is hard to simulate it as online environments are too complex.

To support more realistic testing of Internet server applications, we develop a live flow reproduction tool - TCPCopy, which could generate the test workload that is similar to the production workload.

Currently, TCPCopy has been widely used by companies in China. TCPCopy has little influence on the production system except occupying additional CPU, memory and bandwidth.

Moreover, the reproduced workload is similar to the production workload in request diversity, network latency and resource occupation.

虽然真实的实时流对于Internet服务器应用程序的测试非常重要，但由于在线环境过于复杂，因此很难对其进行模拟。为了支持对Internet服务器应用程序进行更真实的测试，开发了一个实时流复制工具-TCPCopy，它可以生成与生产工作负载类似的测试工作负载。TCPCopy除了占用额外的CPU、内存和带宽外，对生产系统的影响很小。此外，再现的工作负载在请求多样性、网络延迟和资源占用方面与生产工作负载相似。

Scenarios-TCPCopy应用场景

1.Distributed stress testing-分布式压力测试

Use tcpcopy to copy real-world data to stress test your server software. Bugs that only can be produced in high-stress situations can be found

使用tcpcopy复制真实数据以对服务器软件进行压力测试，发现只有在高压场景才会产生的Bugs

2.Live testing-实况测试

Prove the new system is stable and find bugs that only occur in the real world

证明新上线的系统是稳定的，并发现只会在生产环境才会出现的Bugs

3.Regression testing-回归测试

4.Performance comparison-性能比较

同样请求，针对不同或不同版本程序，可以做性能对比等试验

5.流量放大功能，可以利用多种手段构造无限在线压力，满足中小网站压力测试要求

6.利用TCPCopy转发传统压力测试工具发出的请求，可以增加网络延迟，使其压力测试更加真实

7.热备份

Architecture-TCPCopy架构

As shown in Figure 1, TCPCopy consists of two parts: tcpcopy and intercept. While tcpcopy runs on the online server and captures the online requests, intercept runs on the assistant server and does some assistant work, such as passing response info to tcpcopy. It should be noted that the test application runs on the target server.
tcpcopy utilizes raw socket input technique by default to capture the online packets at the network layer and does the necessary processing (including TCP interaction simulation, network latency control, and common upper-layer interaction simulation), and uses raw socket output technique by default to send packets to the target server (shown by pink arrows in the figure).
The only operation needed on the target server for TCPCopy is setting appropriate route commands to route response packets (shown by green arrows in the figure) to the assistant server.
intercept is responsible for passing the response header(by default) to tcpcopy. By capturing the reponse packets, intercept will extract response header information and send the response header to tcpcopy using a special channel(shown by purple arrows in the figure). When tcpcopy receives the response header, it utilizes the header information to modify the attributes of online packets and continues to send another packet. It should be noticed that the responses from the target server are routed to the assistant server which should act as a black hole.

TCPCopy 由两部分组成：tcpcopy 和 intercept。tcpcopy 在 online server 上运行并捕获 online requests，而 intercept 在 assistant server 上运行并执行一些 assistant work，例如将 response-响应信息传递给 tcpcopy。应该注意，测试应用程序在 target server 上运行。

默认情况下，tcpcopy使用 raw socket input technique 在网络层捕获 online packets，并进行必要的处理（包括TCP交互模拟、网络延迟控制和公共上层交互模拟）；使用 raw socket output technique 将 packets-数据包发送到 target server（图中的粉红色箭头所示）

TCPCopy 在 target server 上所需的唯一操作是设置适当的 route 命令，以将 response packets（图中绿色箭头所示）路由传递到 TCPCopy 在目标服务器上所需的唯一操作是设置适当的路由命令，以将响应数据包（图中绿色箭头所示）路由到辅助服务器。

intercept 负责将 response header（默认情况下）传递给 tcpcopy。通过捕获 reponse packets，intercept 将提取 response header information，并使用 special channel（图中紫色箭头所示）将 response header 发送给 tcpcopy。当 tcpcopy 接收到 response header 时，它利用header information修改 online packets 的属性，并继续发送另一个 packet。应该注意的是，来自 target server 的 responses 被路由到 assistant server（which should act as a black hole）。

TCPCopy拷贝一次流量访问的步骤：
1）一个访问请求到达线上内核后端机；
2）socket 包在 IP 层被拷贝了一份传给tcpcopy 进程；
3）tcpcopy 修改包的目的及源地址，发给测试内核后端机；
4）拷贝的包到达测试内核后端机；
5）测试内核后端机的推荐内核处理访问，并返回结果；
6）返回结果在 IP 层被截获、丢弃，由 intercept 拷贝返回结果的 IP header 返回；
7）IP header 被发送给线上内核后端机的 tcpcopy 进程。

TCPCopy搭建

1）Quick start
Two quick start options are available for intercept:
Download the latest intercept release.
Clone the repo: git clone git://github.com/session-replay-tools/intercept.git.

Two quick start options are available for tcpcopy:
Download the latest tcpcopy release.
Clone the repo: git clone git://github.com/session-replay-tools/tcpcopy.git.

199（online-server）：/home/appuser/tcpcopy-online
200（target-server）：/home/appuser/tcpcopy-target
201（assistant-server）：/home/appuser/tcpcopy-assistant

2）Getting intercept installed on the assistant server
安装 intercept 之前先安装 libpcap（libpcap是一个网络数据包捕获函数库），libpcap下载地址，下载下来之后

tar -zxvf libpcap-1.10.1.tar.gz
mv libpcap-1.10.1.tar.gz libpcap
cd libpcap
./configure （如果出现报错：configure: error: Neither flex nor lex was found.，则执行yum install flex bison）
make 
（make 命令如果出现报错：make:*** No targets specified and no makefile found.Stop.，则执行
yum install gcc gcc-c++ autoconf automake
yum -y install zlib zlib-devel openssl openssl-devel pcre pcre-devel
yum update
yum install gcc build-essential
）
make install

WinSCP transfer to 201:/tmp/ path
cd /home/appuser/tcpcopy-assistant
pwd
mv /tmp/intercept.zip .
unzip intercept.zip
./configure
make
make install

参考文章：
TCPCopy
intercept
大神教你玩转tcpcopy
在线引流工具Tcpcopy原理、使用、采坑
TCPCOPY使用