爬取人民日报_python爬取新浪微博人民日报信息

首先构造请求头

base_url ='https://m.weibo.cn/api/container/getIndex?'

headers = {

'Host':'m.weibo.cn',

'Referer':'https://m.weibo.cn/u/2803301701',

'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36',

'X-Requested-With':'XMLHttpRequest'

}

之后根据人民日报的Cookie数值(在审查元素的Network(getindex中uid以及containderid)

def get_page(page):

params = {

'type':'uid',

'value':'2803301701',

'containerid':'1076032803301701',

'page': page

}

构造网页

url = base_url + urlencode(params)

判断请求是否成功(200成功，404丢失，500服务器问题)

try:

response = requests.get(url, headers=