2013年12月_robert_zhai

原创 python 启动多个php进程处理数据

#-*- coding: utf-8 -*-'''Created on 2013-12-29@author: Administrator'''from multiprocessing import Processimport sys, osdef parse_task(arg): os.system("nohup php parse_meis

2013-12-29 22:40:32 770

原创 python connection pool

import MySQLdbfrom DBUtils.PooledDB import PooledDBpool = PooledDB(MySQLdb,15,host='localhost',user='test',passwd='test',db='crawl',port=3306) #15为连接池里的最少连接数 conn = pool.connection() cu

2013-12-26 23:22:48 2889

原创曾国藩终于看完了

曾国藩终于看完了，感概良多。

2013-12-22 16:26:20 576

原创 python smtp send mail

import smtplibimport base64from email.mime.text import MIMETextsender = 'wwwww6@163.com'mailto = '3331@qq.com'#邮件信息msg =MIMEText("It's a text email!")msg['Subject'] = 'Hello world'msg['to'

2013-12-21 20:01:21 3664

原创 python yellow page thread crawler

把之前的爬虫用python改写了，多线程和队列来抓取，典型的生产者消费者模式 #-*- coding: utf-8 -*-'''spider.pyCreated on 2013-12-21http://www.cn360cn.com/news.aspx?pageno=2@author: Administrator'''import Pagerimport Queuei

2013-12-21 17:44:54 833

转载阿里数据平台

http://www.alidata.org/archives

2013-12-20 20:09:43 640

原创 solve hadoop streaming memory exceed

出现内存不够，指定reducer数大点，就能避免这个问题，如指定 -numReduceTasks 500#!/bin/bash/usr/bin/hadoop fs -rmr /user/hmthdfs/upload/out/usr/bin/hadoop fs -lsr /user/hmthdfs/upload/usr/bin/hadoop jar /usr/lib/hadoo

2013-12-20 19:30:12 630

原创开始深入学习python

今天和全威聊了会，他强调不会python就会落伍，哥从现在起开始用python写小项目，之前把官方的手册看了一遍，没怎么写代码。

2013-12-18 21:26:52 789

原创 array left shift

#include void print(int arr[], int n) { for(int i=0; i < n; i++){ printf("%d ", arr[i]); } printf("\n");}void reverse(int *arr ,int start, int end) { while(start < end) { int temp

2013-12-14 11:16:39 598

转载 http://refactoring.com/catalog/index.html

http://refactoring.com/catalog/index.html

2013-12-08 15:03:56 672

原创你妹，又感冒了，要死的节奏

感冒了，要死的节奏，以后要加强运动。。。

2013-12-08 14:00:55 570

原创 python threading server

'''Created on 2013-12-8@author: Administrator'''import socket, traceback,os,sysimport threadingdef handlechild(clientsock): print "got connection from ", clientsock.getpeername() while

2013-12-08 11:30:17 737

原创 python deadlock

'''Created on 2013-12-8@author: Administrator'''import threading,timea=1alock = threading.Lock()b=2block = threading.Lock()def calc1(): print "calc1 acquiring lo

2013-12-08 10:50:06 669

原创 python semaphore

'''Created on 2013-12-8@author: Administrator'''import threading,time,randomdef numbergen(sem, queue, qlock): while 1: time.sleep(2) if random.randin

2013-12-08 10:40:02 2181

原创 python threading.Lock()

'''Created on 2013-12-7@author: Administrator'''import threading,timeb=50l=threading.Lock()def threadcode(): global b print "thread %s invoked " % threading.current

2013-12-08 09:35:21 2508

转载淘宝网络框架tbnet源码分析

http://blog.yunnotes.net/index.php/read_tbnet_code_again/

2013-12-06 19:30:53 2369

转载 TCP,IP,HTTP,SOCKET区别和联系

TCP,IP,HTTP,SOCKET区别和联系网络由下往上分为: 对应物理层-- 数据链路层--网络层-- IP协议传输层-- TCP协议会话层--表示层和应用层-- HTTP协议socket则是对TCP

2013-12-05 08:49:59 595

原创一套房70万买210万转手卖出

下班在地铁上听俩人说某某一套房70万买210万转手卖出，。。。。

2013-12-03 22:52:02 576

原创 http://www.cn360cn.com/ 抓取分类爬虫

include "../../config/common.php";define( "PREG_HREF_CAT", "/(.*?)/is" );$proxy = new Proxy();for($page=1; $page$base = 'http://www.cn360cn.com/';$url = "http://www.c

2013-12-03 14:40:03 2612

原创 python HTTPServer

'''Created on 2013-12-3@author: Administrator'''from BaseHTTPServer import HTTPServer, \ BaseHTTPRequestHandlerclass RequestHandler(BaseHTTPRequestHandler): def _writeheade

2013-12-03 09:09:52 2250

转载 HTMLParser python usage

'''Created on 2013-12-2@author: Administrator'''from HTMLParser import HTMLParserimport urllibimport sysclass ParseLinks(HTMLParser): def __init__(self):

2013-12-02 22:07:31 564

robert198837的专栏