爬虫入门

1、一个简单的例子

1）获取网页内容

• requests：get

url = ""
response = requests.get(url).content.decode('utf-8')

• requests：get & 指定header内容

url = ""
# 指定浏览器代理，可以通过浏览器查看；也可以指定其他信息，
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36'
}
cookie = {"Cookie": 'BAIDUID=FE0F97F1FC37C47792091A2523CD945F:FG=1; HMACCOUNT=CC6D0E280C842123'}
try:
    response = requests.get(url, headers=headers, cookie=cookie).content.decode('utf-8')
    json_dict = json.loads(response)
except:
    v_url = ''

2）解析网页内容

• soup