Python爬虫基础

Python学习记录

Python基础爬虫：

代码：

import urllib.request
import random
import chardet


#请求头列表
us=[
    "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
    "Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11"
    "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36",
]
#伪造头部请求
header = {"User-Agent": random.choice(us)}
url = "https://www.sina.com.cn"
#创建一个请求对象
req = urllib.request.Request(url=url, headers=header)
res = urllib.request.urlopen(req)#将请求对象放入打开链接
data = res.read()
res = chardet.detect(data)
print(res)
html = data.decode("ascii", errors="ignore")
with open("bz.html", "wb") as f:
    f.write(data)
    f.close()

运行结果：

示例代码爬取内容的能力是有限的，若网站有反爬机制则无法爬取。