taks4 简单的爬虫实现自动化办公
1、requests库
Requests这个http请求库,能非常方便的对网页数据进行爬取。
2、请求响应 ,用到了re.status_code
代码:
import requests
re = requests.get('http://www.baidu.com')
print(re.status_code)
运行结果:
3、爬取文章的内容,存到txt文件中
注意打开文件的类型为'w',然后可以自动创建此文件
re.text响应内容的字符串形式,用于文本内容的获取与下载
代码:
re = requests.get('https://apiv3.shanbay.com/codetime/articles/mnvdu')
with open('1.txt','w') as f:
f.write(re.text)
运行结果:
4、re.content用于爬取图片,视频,音频(以图片为例)
代码:
re = requests.get('https://pic1.zhimg.com/v2-0703954b925845892ef559c423f598a0_r.jpg')
with open('你的名字.png','wb') as f:
f.write(re.content)
运行结果:
5、查看bilibili界面的html代码
当向浏览器输入某个网址时,浏览器向服务器发起请求,然后服务器向浏览器返回html代码,最后浏览器将html代码解析成我们看到的界面。
代码:
re = requests.get('http://www.bilibili.com')
re.encoding = 'utf-8'
print(re.text)
运行结果:
6、项目实战1 爬取数据并存入csv文件中
一些知识点:
headers表示请求网页的头,如果没有这个头,浏览器会认为我们在爬虫
接下来为 BeautifulSoup的简单使用
Beautifulsoup对象的作用:通过re.text产生的是字符串,但是字符串有很多局限性,将字符串解析成Beautifulsoup对象:也就是解释器,解释器对象里面有许许多多的内置函数方便我们对HTML代码进行相关操作
运用Beautifulsoup对象中的find()方法和find_all()方法返回对应的html语句
find方法还有很多其他的参数组合,可以组合起来使用
代码:
import requests
from bs4 import BeautifulSoup
import random
import time
import csv
#这里增加了很多user_agent
#能一定程度能保护爬虫
user_agent = [
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
"Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)"]
def get_info():
csvheader=['名称','面积','朝向','户型','位置','楼层','是否有电梯','建成时间',' 门锁','绿化']
with open('data.csv', 'a+', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow(csvheader)
for i in range(1,2): #总共有50页
print('正在爬取自如第%s页'%i)
timelist=[1,2,3]
print('有点累了,需要休息一下啦(¬㉨¬)')
time.sleep(random.choice(timelist)) #休息1-3秒,防止给对方服务器过大的压力!!!
url='https://wh.ziroom.com/z/p%s/'%i
headers = {'User-Agent': random.choice(user_agent)}
r = requests.get(url, headers=headers)
r.encoding = r.apparent_encoding
soup = BeautifulSoup(r.text, 'lxml')
all_info = soup.find_all('div', class_='info-box')
print('开始干活咯(๑><๑)')
for info in all_info:
href = info.find('a')
if href !=None:
href='https:'+href['href']#这个href是网址的意思
try:
print('正在爬取%s'%href)#正在爬取某个网址
house_info=get_house_info(href)
writer.writerow(house_info)
except:
print('出错啦,%s进不去啦( •̥́ ˍ •̀ू )'%href)
def get_house_info(href):
#得到房屋的信息
time.sleep(1)
headers = {'User-Agent': random.choice(user_agent)}
response = requests.get(url=href, headers=headers)
response=response.content.decode('utf-8', 'ignore')#把得到的东西变成一种。。unicode编码(解码)
soup = BeautifulSoup(response, 'lxml')#现在要去解析
name = soup.find('h1', class_='Z_name').text
sinfo=soup.find('div', class_='Z_home_b clearfix').find_all('dd')
area=sinfo[0].text
orien=sinfo[1].text
area_type=sinfo[2].text
dinfo=soup.find('ul',class_='Z_home_o').find_all('li')
location=dinfo[0].find('span',class_='va').text
loucen=dinfo[1].find('span',class_='va').text
dianti=dinfo[2].find('span',class_='va').text
niandai=dinfo[3].find('span',class_='va').text
mensuo=dinfo[4].find('span',class_='va').text
lvhua=dinfo[5].find('span',class_='va').text
['名称','面积','朝向','户型','位置','楼层','是否有电梯','建成时间',' 门锁','绿化']
room_info=[name,area,orien,area_type,location,loucen,dianti,niandai,mensuo,lvhua]
return room_info
if __name__ == '__main__':
get_info()
运行结果:
![在这里插入图片描述](https://img-blog.csdnimg.cn/4bec5c63bd154e9fb609a59678ee47f2.png)
6、项目实战2 爬取数据并通过邮件发送
代码:
因隐私原因,只分享模板部分:
import requests
import random
from bs4 import BeautifulSoup
import smtplib # 发送邮件模块
from email.mime.text import MIMEText # 定义邮件内容
from email.header import Header # 定义邮件标题
smtpserver = 'smtp.qq.com'
# 发送邮箱用户名密码
user = ''
password = '' #这里填授权码
# 发送和接收邮箱
sender = ''
receive = ''
user_agent = [
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; InfoPath.3; rv:11.0) like Gecko",
"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)",
"Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
"Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.8.131 Version/11.11",
"Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Maxthon 2.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; TencentTraveler 4.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; The World)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; SE 2.X MetaSr 1.0; SE 2.X MetaSr 1.0; .NET CLR 2.0.50727; SE 2.X MetaSr 1.0)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Avant Browser)"]
def main():
print('正在爬取数据')
url = 'https://36kr.com/newsflashes'
headers = {'User-Agent': random.choice(user_agent)}
response = requests.get(url, headers=headers)
response=response.content.decode('utf-8', 'ignore')
soup = BeautifulSoup(response, 'lxml')
news = soup.find_all('a', class_='item-title')
news_list=[]
for i in news:
title=i.get_text()
href='https://36kr.com'+i['href']
news_list.append(title+'<br>'+href)
info='<br></br>'.join(news_list)
print('正在发送信息')
send_email(info)
def send_email(content):
# 通过QQ邮箱发送
title='36kr快讯'
subject = title
msg = MIMEText(content, 'html', 'utf-8')
msg['Subject'] = Header(subject, 'utf-8')
msg['From'] = sender
msg['To'] = receive
# SSL协议端口号要使用465
smtp = smtplib.SMTP_SSL(smtpserver, 465) # 这里是服务器端口!
# HELO 向服务器标识用户身份
smtp.helo(smtpserver)
# 服务器返回结果确认
smtp.ehlo(smtpserver)
# 登录邮箱服务器用户名和密码
smtp.login(user, password)
smtp.sendmail(sender, receive, msg.as_string())
smtp.quit()
if __name__ == '__main__':
main()
运行结果:
给对应的邮箱发送了相应的数据
7、后记
1、通过本次的学习,我知道了爬虫的基本原理,并应用到基本的办公上。
借助requests http请求库对相应URL链接下自己需要的数据进行捕获。基本的过程为在浏览器中输入URL链接,浏览器向服务器发送请求,服务器向浏览器返回HTML代码,浏览器对HTML代码进行解析。在解析的过程中有用到Beautifulsoup 对象,这种对象有很多内置函数可以解析HTML网页并最终获取自己所需要的数据。
2、在学习中遇到的最大的困难就是对检索信息的不理解,因为自己本身对HTML标签的不熟悉导致在调用find函数时一些参数实在无法理解,这需要我下去自学一下HTML的相关知识。
3、多多记忆与总结吧。