Python网络爬虫(3)——以爬虫的角度看待网络内容

修改User-Agent字段
其实这是一种模拟浏览器的方法

        kv = {'user-agent':'Mozilla/5.0'}
        r = requests.get(url,headers=kv)

根据关键词查询

import requests
def GetKwyword(keyword):
    try:
        kv = {'kw':keyword}
        r = requests.get("http://www.baidu.com/s",params=kv)
        r.raise_for_status()
        return r
    except:
        return "Error!"
if __name__ == '__main__':
    keyword = "NJUPT"
    print(len(GetKwyword(keyword).text))

IP地址查询示例

import requests
url = "http://www.ip138.com/ips138.asp?ip="
r = requests.get(url+'120.199.34.101')
r.encoding = r.apparent_encoding
print(r.text)