流式数据处理

1、 直接登陆服务器：ssh 2014210***@thumedia.org -p 6349

<span style="font-size:14px;">#! /usr/bin/python
import logging
import math
import time
pg2count={}
t=1
while 1:
fp=open('/tmp/hw3.log','r')
for line in fp:
line = line.strip()
times, page, count = line.split()[0],line.split()[1],line.split()[2]
if count.isdigit() & page.startswith('Page-'):
try:
pg2count[page] = [pg2count[page][0] + int(count),t]
except:
pg2count[page] = [int(count),t]
fp.close()
a=sorted(pg2count.items(), key=lambda page:page[1][0], reverse = True)
print '%s%s%s' % ('the page rank at current time ',times,' is:')
for i in range(0,10):
print '%s\t%d' % (a[i][0],a[i][1][0])
logger = logging.getLogger()
#set loghandler
file = logging.FileHandler("output.log")
#set formater
formatter = logging.Formatter("%(asctime)s %(levelname)s %(message)s")
file.setFormatter(formatter)
#set log level
logger.setLevel(logging.NOTSET)
logger.info('%s%s%s' % ('the page rank at current time ',times,' is:'))
for i in range(0,10):
logger.info('%s\t%d' % (a[i][0],a[i][1][0]))
time.sleep(60)</span>

2、 写好代码之后测试运行：python streaming.py输出如下：

nohup: ignoring input and appending output to nohup.out'，则表示后台运行成功，输出显示会保存到nohup.out中，

[1] 8994

2014210***@cluster-3-1:~$nohup: ignoring input and appending output to nohup.out' 一天之后，我们再次查看结果： 可以看到，累计的结果已经和第一次不太一样 3、 杀掉进程：ps -ef|grep 1020得到如下输出： 2014210***@cluster-3-1:~$ ps -ef|grep 1020

1020      7512  7471  0 Jan10 ?        00:00:00 sshd: 2014210***@pts/30

1020      7513  7512  0 Jan10 pts/30   00:00:00 -bash

1020      7574  7508  0 20:55 ?        00:00:00 sshd: 2014210***@pts/52

1020      7575  7574  0 20:55 pts/52   00:00:00 -bash

1020      8282  7575  0 21:04 pts/52   00:00:00 ps -ef

1020      8283  7575  0 21:04 pts/52   00:00:00 grep --color=auto 1020

1020      8994     1  0 13:20 ?        00:01:46 python streaming.py

1020     12260 12232  0 Jan10 ?        00:00:00 sshd: 2014210***@pts/35

1020     12261 12260  0 Jan10 pts/35   00:00:01 –bash

2014210***@cluster-3-1:~$kill 8994 2014210***@cluster-3-1:~$ ps -ef|grep 1020

1020      7512  7471  0 Jan10 ?        00:00:00 sshd: 2014210***@pts/30

1020      7513  7512  0 Jan10 pts/30   00:00:00 -bash

1020      7574  7508  0 20:55 ?        00:00:00 sshd: 2014210***@pts/52

1020      7575  7574  0 20:55 pts/52   00:00:00 -bash

1020      8335  7575  0 21:05 pts/52   00:00:00 ps -ef

1020      8336  7575  0 21:05 pts/52   00:00:00 grep --color=auto 1020

1020     12260 12232  0 Jan10 ?        00:00:00 sshd: 2014210***@pts/35

1020     12261 12260  0 Jan10 pts/35   00:00:01 –bash

Question

How can your design scale when the streaming is large and the calculation is complicated?