一、启动一个tcp server,把送来的数据返回。
$ more echoserver.py
#!/usr/bin/env python
"""Simple server that listens on port 16000 and echos back every input to the client.
Connect to it with:
telnet localhost 16000
Terminate the connection by terminating telnet (typically Ctrl-] and then 'quit').
"""
from __future__ import print_function
from gevent.server import StreamServer
# this handler will be run for each incoming connection in a dedicated greenlet
def echo(socket, address):
print('New connection from %s:%s' % address)
socket.sendall(b'Welcome to the echo server! Type quit to exit.\r\n')
# using a makefile because we want to use readline()
rfileobj = socket.makefile(mode='rb')
while True:
line = rfileobj.readline()
if not line:
print("client disconnected")
break
if line.strip().lower() == b'quit':
print("client quit")
break
socket.sendall(line)
print(line)
rfileobj.close()
if __name__ == '__main__':
# to make the server use SSL, pass certfile and keyfile arguments to the constructor
server = StreamServer(('0.0.0.0', 16000), echo)
# to start the server asynchronously, use its start() method;
# we use blocking serve_forever() here because we have no other jobs
print('Starting echo server on port 16000')
server.serve_forever()
二、连接server,发送单个字符,并记录,100个做一次统计。
$ more tcptest.py
import subprocess, time, sys, errno
import socket
import numpy as np
def tcptest(host, port, interval=.1, count=None):
print host, port
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((host, port))
i = 0
err = 0
t = []
while True:
try:
i += 1
t1 = time.time()
s.sendall(str(i)+'\n')
data = s.recv(1024)
t2 = time.time()
t.append(t2-t1)
if i % 100 == 0:
summary(t)
t = []
if count and i >= count:
break
time.sleep(interval)
except IOError as e:
print e
if e.errno == errno.EPIPE:
break
except Exception, e:
print e
err += 1
if err > 100:
break
summary(t)
s.close()
def summary(t):
if len(t) > 0:
print '%s %d packets(min/mean/median/max/std):%f/%f/%f/%f/%f' % (time.asctime(), len(t), np.min(t), np.mean(t),np.median(t)
, np.max(t), np.std(t))
sys.stdout.flush()
if __name__ == '__main__':
host, port = sys.argv[1], int(sys.argv[2])
tcptest(host, port)
------
日志类似于:
xxx.xxx.xxx.xx 16000
Fri Sep 2 16:59:26 2016 100 packets(min/mean/median/max/std):0.000338/0.000647/0.000624/0.002038/0.000269
Fri Sep 2 16:59:36 2016 100 packets(min/mean/median/max/std):0.000329/0.001173/0.000483/0.063784/0.006295
Fri Sep 2 16:59:46 2016 100 packets(min/mean/median/max/std):0.000332/0.000498/0.000477/0.000846/0.000102
Fri Sep 2 16:59:56 2016 100 packets(min/mean/median/max/std):0.000325/0.000546/0.000487/0.001736/0.000201
Fri Sep 2 17:00:06 2016 100 packets(min/mean/median/max/std):0.000371/0.000633/0.000576/0.005159/0.000474
Fri Sep 2 17:00:16 2016 100 packets(min/mean/median/max/std):0.000307/0.000663/0.000547/0.008853/0.000851
并在多个服务器启动这个进程。
三、从各个服务器获取日志
服务器记录在
$ more loghost.txt
msg0=172.41.17.127
msg1=172.41.17.128
msg2=172.41.17.129
msg3=172.41.17.130
msg4=172.41.17.131
msg5=172.41.17.132
msg6=172.41.17.135
msg7=172.41.17.136
循环从服务器获取日志:
$ more fetchlog.sh
#!/bin/bash
basedir=/tmp/ana
remotelogdir=/tmp
loghostfile=$basedir/loghost.txt
cat $loghostfile|while read line;do
dirname=`echo $line|awk -F = '{print $1}'`
ip=`echo $line|awk -F = '{print $2}'`
echo mkdir -p $basedir/$dirname;
mkdir -p $basedir/$dirname;
echo scp $ip:$remotelogdir/tcptest.log $basedir/$dirname;
scp $ip:$remotelogdir/tcptest.log $basedir/$dirname;
done
四、对日志文件做summary
$ more summary.sh
#!/bin/bash
basedir=/tmp/ana
datadir=$1
if [ "$datadir" == "" ];then
echo input datadir,pls.
exit 2;
esle
echo do summary datadir=$datadir
fi
cat /dev/null > $datadir/summary.log
#find dataline and date string
grep "min\/mean\/median\/max\/std" $datadir/tcptest.log |cut -c 1-10 |uniq|while read date;do
echo date="$date" ;
day=`echo $date|awk '{print $3}'`;
month=`echo $date|awk '{print $2}'`;
datefile=$month$day.log
echo split into datefile=$datefile;
echo "min/mean/median/max/std" > $datadir/$datefile ;
grep "$date" $datadir/tcptest.log |awk -F : '{print $4}' >> $datadir/$datefile;
echo $date >> $datadir/summary.log
python $basedir/summary.py $datadir/$datefile / >> $datadir/summary.log
done
--------
$ more summary.py
import sys, pandas;
if __name__ == '__main__':
file = sys.argv[1];
separator = sys.argv[2];
data = pandas.read_csv(file,sep = separator);
summary = data.describe();
print(summary.head(10));
print("end");
五、获取日志和分析全部过程串联在一起
]$ more doall.sh
#!/bin/bash
basedir=/tmp/ana
loghostfile=$basedir/loghost.txt
$basedir/fetchlog.sh
cat $loghostfile|while read line;do
dirname=`echo $line|awk -F = '{print $1}'`
$basedir/summary.sh $dirname
done