python第一个爬虫小程序以及遇到问题解决（中文乱码)+批量爬取网页并保存至本地

今天自己看了一下python试着写了一个爬虫小程序

原始代码：

from urllib import request
request.encoding = "utf-8"
response = request.urlopen("http://www.baidu.com")  # 打开网站
html =str(response.read(),'utf-8')
f=open('C:/Users/lenovo/Desktop/11.html','w+')
page = f.write(html)
f.close()

起初在将爬取得网页保存到本地的时候出现错误：

：UnicodeEncodeError: 'gbk' codec can't encode character '\xbb' in position 0: illegal multibyte seque

应该f的编码格式是GBK的，但是其它的是UTF-8的。所以指定一下编码格式即可。

f=open('C:/Users/lenovo/Desktop/11.html','w+',encoding='utf-8')

修改后代码:

from urllib import request
request.encoding = "utf-8"
response = request.urlopen("http://www.baidu.com")  # 打开网站
html =str(response.read(),'utf-8')
f=open('C:/Users/lenovo/Desktop/11.html','w+',encoding='utf-8')
page = f.write(html)
f.close()

批量爬取网页并保存至本地

from urllib import request
request.encoding = "utf-8"
fr = open("C:/Users/lenovo/Desktop/url.txt", "r").readlines()
count = 0
print(fr)
for line0 in fr:
    line = line0.strip('\n')
    line = line.strip('\'')
    print(line+"===========================")
    response = request.urlopen(line)
    html = str(response.read(), 'utf-8')

    fw = open("C:/Users/lenovo/Desktop/%d.html" % count, "w", encoding='utf-8')
    count+=1
    page = fw.write(html)
    fw.close()