近日学习了python语言,简单实现了一个爬虫,爬取了慕课网课程简介上的图片,并保存到本地。以下是实验代码:
# -*- coding: utf-8 -*-
"""
Spyder Editor
"""
import re
import os
import urllib.request #在python3.6环境中实现
f_soure = urllib.request.urlopen('http://www.imooc.com/course/list')
#爬取目标地址
mybytes = f_soure.read()
mystr = mybytes.decode('utf8')
result = re.findall(r'http:.+\.jpg',mystr)
#打印输出并对正则结果进行字符串切割
print(len(result[0]))
print(result[0].index('.jpg'))
print(result[0][:60])
l = []
for i in result:
l.append(i[:60])
print (l)
#重新生成图片url地址,读出并保存到本地
k=0
for url in l:
f = open('F:\\python_test\\%d.jpg'%(k),'wb+')
rep = urllib.request.urlopen(url)
f.write(rep.read())
f.close()
k+=1
print ('success')
在实验过程中,发现使用python自带的os模块,当urllib.request.urlopen(url)方法返回的类file对象时,使用此对象的read方法会出现读取不完全现象。代码如下:
# -*- coding: utf-8 -*-
"""
Spyder Editor
"""
import re
import urllib.request
f_soure = urllib.request.urlopen('http://www.imooc.com/course/list')
mybytes = f_soure.read()
mystr = mybytes.decode('utf8')
result = re.findall(r'http:.+\.jpg',mystr)
print(len(result[0]))
print(result[0].index('.jpg'))
print(result[0][:60])
l = []
for i in result:
l.append(i[:60])
print (l)
k=0
#使用os模块进行写入
for url in l:
f = os.open('F:\\python_test\\%d.jpg'%(k),os.O_CREAT|os.O_RDWR)
rep = urllib.request.urlopen(url)
iter_f = iter(rep)
for line in iter_f:
os.write(f,line)
os.close(f)
k+=1
print ('success')
如果有大神知道为什么这样,还请解惑~~