1、一个简单的例子
1)获取网页内容
url = ""
response = requests.get(url).content.decode('utf-8')
- requests:get & 指定header内容
url = ""
# 指定浏览器代理,可以通过浏览器查看;也可以指定其他信息,
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
}
cookie = {"Cookie": 'BAIDUID=FE0F97F1FC37C47792091A2523CD945F:FG=1; HMACCOUNT=CC6D0E280C842123'}
try:
response = requests.get(url, headers=headers, cookie=cookie).content.decode('utf-8')
json_dict = json.loads(response)
except:
v_url = ''
2)解析网页内容