Python爬虫贴吧无数据异常

强推一波新搭建的blog

今天高高兴兴按照视频上敲了一下使用urilib2 爬去贴吧的Get请求，很开心啊，就像写Hello World 一样，
然后还真一样，结果出了一个摸不到头绪的错误

emmm 就是爬了贴吧的页面之后没有显示数据只有一个干巴巴的页面，我尝试过其他页面，是正常的，贴吧不好使，我怀疑这玩意肯定是对Hearder 进行了限制

放一下我的代码。代码很丑，勿要介意

# !/usr/bin/python
# -*- coding: UTF-8 -*-
import urllib2
import urllib


def write_txt(name, content):
    with open("%s.html" % name, "w") as f:
        f.write(content)


def main():
    html, value = get_html()
    write_txt(value["kw"], html)


def get_html():
    header = {
        "User-Agent": "Mozilla/5.0(Windows NT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/61.0.3163.100Safari/537.36 "
    }
    value = {"kw": "python"}
    title_name = urllib.urlencode(value)
    request = urllib2.Request("https://tieba.baidu.com/f?%s&ie=utf-8&pn=50" % title_name, headers=header)
    response = urllib2.urlopen(request)
    html = response.read()
    print request.get_full_url()
    return html, value


if __name__ == '__main__':
    main()

看官尝试跑一下，你就会发现确实没有数据

我这个纠结啊，尝试了好多写法，最后我战胜了error ,

其实就是把Hearder 的 Chrome/61.0.3163.100Safari/537.36 去掉就OK了

# !/usr/bin/python
# -*- coding: UTF-8 -*-
import urllib2
import urllib


def write_txt(name, content):
    with open("%s.html" % name, "w") as f:
        f.write(content)


def main():
    html, value = get_html()
    write_txt(value["kw"], html)


def get_html():
    header = {
        "User-Agent": "Mozilla/5.0(Windows NT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)"
    }
    value = {"kw": "python"}
    title_name = urllib.urlencode(value)
    request = urllib2.Request("https://tieba.baidu.com/f?%s&ie=utf-8&pn=50" % title_name, headers=header)
    response = urllib2.urlopen(request)
    html = response.read()
    print request.get_full_url()
    return html, value


if __name__ == '__main__':
    main()