爬虫：Python2

1，python2简单爬虫解析

URLopen参数解析：

url, 统一资源定位符
data=None,如果为None get/否则是post
timeout=超时

request参数解析：

Accept: text/plain, */*; q=0.01  # 接收类型

Accept-Encoding: gzip, deflate, br # 接收的压缩类型， 做爬虫的时候一定不加

Accept-Language: zh,en;q=0.9,zh-CN;q=0.8 # 支持的语言

Connection: keep-alive # 保持连接

Cookie: BIDUPSID=BCD315DCE8F7FF6947455A0DE12663EB; PSTM=1501484586; __cfduid=d18bd72e2cea6d9181f2e7f2d3e861c161501509299; BAIDUID=ECCE54C3614930157DE746EB6005C67D:FG=1; BD_UPN=12314753; MCITY=-%3A; ispeed_lsm=0; BDUSS=JnZjdDYjlOcGZOSW4tNzJzNFpzTndadDBQeVhKZkM2Zi1TaHpxSzhBTkhhejFiQUFBQUFBJCQAAAAAAAAAAAEAAAAOGvGFc2t5YXNkZmdkZmdoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEfeFVtH3hVbeU; BD_HOME=1; H_PS_PSSID=1468_21099_26430_26577_20929
#本地会话

Host: www.baidu.com # 主机名

Referer: https://www.baidu.com/ # 我来自哪里

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
#用户代理

X-Requested-With: XMLHttpRequest # json异步的

简单爬虫代码：

import urllib2
url = "http://www.baidu.com"
response = urllib2.urlopen(url)
print response  # _fileobject 文件对象
print response.read() # read() 返回一个字符串
print response.readline() # 读取一行
print response.readlines()  # 返回一个列表 for

伪装服务器，获取请求的相关信息：

import urllib2

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"}
url = "http://www.baidu.com"
request = urllib2.Request(url, headers=headers)   # 伪装成浏览器

print request.get_full_url()  # 完整的url
print request.get_data()  # 请求时间
print request.get_host()  # 主机名
print request.get_header("User-agent")  # 首字母大写，其余必须小写
print request.get_method()  # 请求方式 GET、POST
print request.get_type()  # 请求类型http/https

request.add_header("Connection", "keep-alive") # 动态添加请求头
print request.get_header("Connection")

response = urllib2.urlopen(request)
print response.info
print response.code # 状态码 200 ，301，403,404，500
print response.read()

模拟百度应用

import urllib2
def baiduAPI(kw):
    '''
    模拟百度搜索
    :param kw: 搜索关键字
    :return: 响应
    '''
    
    header = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"}
    url = "https://www.baidu.com/s?" + kw
    requset = urllib2.Request(url, headers=header)
    response = urllib2.urlopen(requset)
    return response.read()

if __name__ == '__main__':
    kw = raw_input("请输入你想查找的内容：")
    wd = {"wd": kw, "ie": 'utf-8'}
    wd = urllib.urlencode(wd)  # url编码