python实战
->网课
一.基础
robots.txt
如https://www.taobao.com/robots.txt
查看淘宝能爬取的内容
常用请求头信息
User-Agent:：请求载体的身份标识
Connection：请求完毕后，是断开连接还是保持连接
常用响应头信息
Content-Type:服务器响应回客户端的数据类型
https协议
—安全的http协议（超文本传输协议）
加密方式
对称秘钥加密
客户端常用！有风险！
非对称秘钥加密
对称秘钥加密的升级
证书秘钥加密------https采用的
阿贾克斯是什么
二.ruquests模块
urllib模块复杂，ruquests模块:方便！
1.作用：
模拟浏览器发请求
2.如何使用：
1.指定url
URL就是你的IE地址栏上的那串字符
2.-发起请求
3.获取响应数据
4.持久化存储
3.环境安装
pycharm
pip install requests
anacnda不用安装
4.实战
爬取搜狗首页的数据
coding=utf-8的作用
爬取网页源代码
import requests
if __name__ == "__main__":
# 指定url
url = 'https://hao.360.com/'
# 发起请求
# get方法会返回一个响应对象
response = requests.get(url=url)
# 获取相应数据,text返回 的是字符串形式的响应数据
page_text = response.text
print(page_text)
# 持久化储存
with open('./sougou.html','w',encoding='utf-8') as fp:
fp.write(page_text)
print('爬取数据结束！！！')
爬取sogou词条
UA检测
UA伪装
将对应的User-Agent封装到一个字典中
import requests
if __name__ == "__main__":
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
url = 'https://www.sogou.com/web'
#将参数封装的字典中
kw = input('enter a word')
param = {
'query':kw
}
#对指定url发起请求是携带参数的
response = requests.get(url=url,params=param,headers=headers)
page_text = response.text
fileName = kw+'.html'
with open(fileName,'w',encoding='utf-8') as fp:
fp.write(page_text)
print(fileName,'保存成功！！！')
破解百度翻译
交互数据包—post请求—携带了参数
响应数据是一组json数据
爬取sogou词条
import requests
if __name__ == "__main__":
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
url = 'https://www.sogou.com/web'
#将参数封装的字典中
kw = input('enter a word')
param = {
'query':kw #默认是字符串
}
#对指定url发起请求是携带参数的
response = requests.get(url=url,params=param,headers=headers)
page_text = response.text
fileName = kw+'.html'
with open(fileName,'w',encoding='utf-8') as fp:
fp.write(page_text)
print(fileName,'保存成功！！！')
爬取豆瓣
import requests
import json
if __name__ == "__main__":
url = 'https://movie.douban.com/j/chart/top_list'
param = {
'type' : '24',
'interval_id' : '100:90',
'action' : '',
'start' : '0',#从库中的第几部电影去取
'limit' : '20',#一次取出的个数
}
headers = {
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
response = requests.get(url=url,params=param,headers=headers)
list_date = response.json()
fp = open('./douban.json','w',encoding='utf-8')
json.dump(list_date,fp=fp,ensure_ascii=False)
print('over!!!')
爬取KFC位置
import requests
url = 'http://www.kfc.com.cn/kfccda/ashx/GetStoreList.ashx?op=keyword'
key = input('请输入名称：')
date = {
'MIME类型': 'application/x-www-form-urlencoded; charset=UTF-8',
'cname':'',
'pid':'',
'keyword':key,#默认是字符串
'pageIndex':'1',
'pageSize':'10'
}
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
response = requests.post(url=url,data=date,headers=headers)
page_text = response.text
with open(key+'.text',mode='w',encoding='utf-8') as fp:
fp.write(page_text)
print('保存成功！！！')
如果结果是-1000，则说明是url错误
爬取化妆品生产许可平台
检测各公司是否是由ajax对应的请求
1.代码
import requests
if __name__ == "__main__":
url = 'http://scxk.nmpa.gov.cn:81/xk/'
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
page_text = requests.get(url=url,headers=headers).text
with open('./化妆品.html','w',encoding='utf-8') as fp:
fp.write(page_text)
html打开如下
2.抓包工具
按ctrl f 搜索
0 of 0 表示没搜到
证明猜想成立
数据解析
聚焦爬虫
->爬取页面中指定的页面内容.
通用->聚焦
数据解析分类
-正则
爬取囧图
# -*- coding: utf-8 -*-
"""
Created on Mon Mar 29 19:35:56 2021
@author: ASUS
"""
import requests
import re
import os
if __name__ == "__main__":
if not os.path.exists('./囧图图片'):
os.mkdir('./囧图图片')
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
#创建一个文件夹，保存所有的图片
#ex = '<img src="(.*?)" alt="(.*?)" width=.*? height=.*?>'错误
#通用模板
url = 'https://www.qiushibaike.com/imgrank/page/%d/'
for pageNum in range(1,3):#页码控制
new_url = format(url%pageNum)
page_text = requests.get(url=new_url,headers=headers).text
ex = '<div class="thumb">.*?<img src="(.*?)" alt.*?</div>'
img_src_list = re.findall(ex,page_text,re.S)#re.S单行匹配#.M多行匹配
#print(img_src_list)
for src in img_src_list:
#拼接出一个完成的图片url
src = 'https:'+src
#请求到了图片的二进制数据
img_data = requests.get(url=src,headers=headers).content
#生成图片名称
img_name = src.split('/')[-1]
#图片储存的路径
imgPath = './囧事图片'+img_name
with open(imgPath,'wb') as fp:
fp.write(img_data)
print(img_name,'下载成功！！！')
bs4
讲解代码
from bs4 import BeautifulSoup
if __name__ == "__main__":
#将本地的html文档中的数据加载到该对象中
fp = open('./text.html','r',encoding='utf-8')
soup = BeautifulSoup(fp,'lxml')
# print(soup)
# tagName标签名
# print(soup.a)# soup.tagName返回的是html中第一次返回的a标签
# print(soup.div)
# find('tagName')等同于soup.div
# print(soup.find('div'))#print(soup.div)
# print(soup.find('div',class = 'song'))
# print(soup.find_all('a'))
print(soup.select('.left')[0].string)
案例代码：爬三国演义
import requests
from bs4 import BeautifulSoup
#需求：爬取三国演义的章节标题
if __name__ == "__main__":
#对首页的页面数据进行爬取
url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
page_text = requests.get(url=url,headers=headers).text
# 在首页中解析出章节的标题和详情也的url
# 1.实例化一个对象
soup = BeautifulSoup(page_text,'lxml')
# 解析章节标题和详情页的url
li_list = soup.select('.book-mulu > ul > li')
fp = open('./三国.txt','w',encoding='utf-8')
for li in li_list:
title = li.a.string
detail_url = 'https://www.shicimingju.com'+ li.a['href']
#对详情也发起请求，解析出出章节内容
detail_page_text = requests.get(url=detail_url,headers=headers).text
# 解析出详情页中相关的章节内容
detail_soup = BeautifulSoup(detail_page_text,'lxml')
detail_soup.find('div',class_='chapter_content')
# 解析到了章节的内容
content = div_tag.text
fp.write(title+':'+content+'\n')
print(title,'爬取成功！！！')
数据解析的原理
** 1.标签定位**
** 2.提取标签、标签属性中的储存的数据值**
bs4数据解析的原理：
-1.实例化一个BeautSoup对象，并且将页面源码数据加载到该对象中
-2.通过调用BeautSoup对象中相关的属性或者方法进行标签定位和数据提取
环境安装
- pip install bs4
- pip install lxml
如何实例化BeautifulSoup对象：
- 1
from bs4 import BeautifulSoup - 2.对象的实例化：
- 1.将本地的html文档中的数据加载到该对象中
fp = open('./text.html','r',encoding='utf-8') soup = BeautifulSoup(fp,'lxml') - 2.将互联网上获取的页面源代码加载到该对象中
page_text = response.text soup = BeautifulSoup(page_text,'lxml')
- 1.将本地的html文档中的数据加载到该对象中
- 3.提供的用于数据解析的方法和属性
- soup.tagName：返回的是文档中第一次出现的a标签
- soup.find(‘tagName’)等同于soup.div
- 属性定位
- soup.find(‘div’,class/id/attr = ‘song’)
- 4.soup.find_all() :返回符合要求的所有标签（列表）（也可属性定位）
- 5.select：
- select（‘某种选择器（id,class,标签…选择器）’）返回的是一个列表
- soup.select(’.tang > ul >li >a’)[0]:>表示的是一个层级
- soup.select(’.tang > ul a’)[0]:空格表示的是多个层级
- select（‘某种选择器（id,class,标签…选择器）’）返回的是一个列表
- 获取标签之间文本数据
- soup.a.text/string/get_text()
- text/get_text()：可以获取某个标签中所有的文本内容
- string:只能获取该标签下面直系的文本内容
get_text()是个方法
- 获取标签中属性值
- soup.a[‘href’]
xpath解析：通用性强
最常用且最便捷高效的一种解析方式
讲解代码
from lxml import etree
if __name__ == "__main__":
# 实例化
# tree = etree.parse('text.html',etree.HTMLParser())
tree = etree.parse('text.html',etree.HTMLParser())
# r = tree.xpath('/html/body/div')
# r = tree.xpath('html//div')
# r = tree.xpath('//div')
# r = tree.xpath('//div[@class="left"]/a/text()')得到的是字符串---【“ ”】形式的，
# r = tree.xpath('//div[@class="left"]/a/text()')[0]----得到文本内容
r = tree.xpath('//div[@class="right"]//img/@src')
print(r)
- xpath解析原理：
- 1.化一个etree的对象，且需要将被解析的页面源代码数据加载到该对象中
- 2.调用etree对象中的xpath方法结合着xpath表达式实现标签的定位和内容的捕获
- 3.环境安装：pip install lxml
- 如何实例化一个etree对象
- 1.将本地的html文档中的源码数据加载到etree对象中：
etree.parse(filePath) - 2.可以将从互联网上获取的源代码加载到该对象中
etree.HTML('page_text') - 3.xpath('xpath表达式‘）
- /：表示的是从根节点开始定位。表示的是一个层级。
- //：表示的是多个层级，可以从任意位置开始定位
- 属性定位：r = tree.xpath(’//div[@class=“left”]’)----tag[@attrName=“attrValue”]
- 索引定位：r = tree.xpath(’//div[@class=“left”]/p[3] 索引是从1开始的。
- 取文本：r = tree.xpath(’//div[@class=“left”]/a/text()’)[0]
- /text() 获取的是标签中直系的文本内容
- //text() 标签中非直系的文本内容（所有的文本内容)
- 取属性：/@attrName ==>img/src
- 1.将本地的html文档中的源码数据加载到etree对象中：
爬取美女图片
import requests
from lxml import etree
import os
if __name__ == "__main__":
url = 'https://pic.netbian.com/4kmeinv/'
headers = {
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
response = requests.get(url=url,headers=headers)
# 手动设置响应数据的编码格式
# response.encoding = 'utf-8'
page_text = response.text
tree = etree.HTML(page_text)
li_list = tree.xpath('//div[@class="slist"]/ul/li')
#创建一个文件夹
if not os.path.exists('./picLibs'):
os.mkdir('./picLibs')
for li in li_list:
img_src = 'https://pic.netbian.com/' + li.xpath('./a/img/@src')[0]
img_name = li.xpath('./a/img/@alt')[0]+'.jpg'
#通用解决中文乱码的解决方案
img_name = img_name.encode('iso-8859-1').decode('gbk')
# print(img_name,img_src)
# 请求图片进行持久化储存
img_date = requests.get(url=img_src,headers=headers).content# 二进制
img_path = 'picLibs/'+img_name
with open(img_path,'wb') as fp:
fp.write(img_date)
print(img_name,'下载成功！！！')
爬取全国城市名
# import requests
# from lxml import etree
# if __name__ == "__main__":
# headers = {
# 'user-agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
# }
# url = 'https://www.aqistudy.cn/historydata/'
#page_text = requests.get(url=url,headers=headers).text
#tree = etree.HTML(page_text)
#hot_li_list = tree.xpath('//div[@class="bottom"]/ul/li')
#all_city_names = []
#解析到热门城市
#for li in hot_li_list:
# hot_city_name = li.xpath('./a/text()')[0]
# all_city_names.append(hot_city_name)
#city_names_list = tree.xpath('//div[@class="bottom"]/ul/div[2]/li')
#for li in city_names_list:
# city_name = li.xpath('./a/text()')[0]
# all_city_names.append(city_name)
#print(all_city_names,len(all_city_names))
import requests
from lxml import etree
if __name__ == "__main__":
headers = {
'user-agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36'
}
url = 'https://www.aqistudy.cn/historydata/'
page_text = requests.get(url=url,headers=headers).text
tree = etree.HTML(page_text)
#('//div[@class="bottom"]/ul/li')
#('//div[@class="bottom"]/ul/div[2]/li/a')
a_list = tree.xpath('//div[@class="bottom"]/ul/li/a | //div[@class="bottom"]/ul/div[2]/li/a')
all_city_names = []
for a in a_list:
city_name = a.xpath('./text()')[0]
all_city_names.append(city_name)
print(all_city_names,len(all_city_names))
我用的 html 文件
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>这是一个低仿的QQ注册界面</title>
<style>
body,div,dl,dt,dd,ul,ol,li,h1,h2,h3,h4,h5,h6,pre,code,form,fieldset,legend,input,textarea,p,blockquote,th,td
{
margin:0;padding:0
}
html{
font-family: -apple-system,"Helvetica Neue",Arial,"PingFang SC","Hiragino Sans GB",STHeiti,"Microsoft YaHei","Microsoft JhengHei","Source Han Sans SC","Noto Sans CJK SC","Source Han Sans CN","Noto Sans SC","Source Han Sans TC","Noto Sans CJK TC","WenQuanYi Micro Hei",SimSun,sans-serif;
}
.left{
width: 480px;
height: 1000px;
float: left;
box-sizing: border-box;
background-image: url("https://cdn-go.cn/qq-web/zc.qq.com/latest/v3/img/01-1.jpg");
position: fixed;
}
.right{
width: 1039px;
height: 1000px;
/*background-color: skyblue;*/
float: right;
box-sizing: border-box;
}
.lianghao img{
float: left;
margin-bottom: 2px;
}
.right .top .lianghao div{
float:left;
border-radius: 6px;
width: 80px;
height: 25px;
line-height: 25x;
font-size: 16px;
/* margin: 0 10px 0 24px;
padding: 0 10px 0 24px;*/
margin: 10px 2px 0 0;
padding: 0 5px 0 0;
color: #fff;
font-weight: 300;
background: linear-gradient(126deg,#fc4754,#ff794c) #f54444;;
text-align: right;
box-sizing: border-box;
}
.top {
/*background-color: mediumslateblue;*/
width: 450px;
height: 100px;
float: right;
box-sizing: border-box;
}
.lianghao{
width: 150px;
height: 50px;
/*background-color: gold;*/
float: left;
margin-right: 40px;
margin-top: 12px;
}
.language{
width: 100px;
height: 50px;
/*background-color: blue;*/
float: left;
box-sizing: border-box;
margin-top: 21px;
}
.feedback{
width: 80px;
height: 40px;
float: right;
/*background-color:greenyellow;*/
box-sizing: border-box;
margin-top: 12px;
margin-right: 30px;
line-height: 40px;
}
/* .language select{
border: transparent;
height: 40px;
color: #000000;
font-size: 16px;
}*/
.feedback a{
color: #cccccc;
font-size: 16px;
text-decoration: none;
}
.bottom{
width: 520px;
height: 600px;
position: absolute;
left: 50%;
top: 35%;
}
.bottom .logo{
width: 520px;
height: 120px;
/*background-color: orange;*/
font-weight: 300px;
}
.bottom .logo .big{
font-size: 40px;
}
.bottom .logo .small{
font-size: 25px;
}
.logo a{
font-size: 25px;
text-decoration: none;
color: #3b78dd;
float: right;
}
.line{
width: 520px;
height: 250px;
margin-top: 50px;
}
.line .name{
border: 1px solid skyblue;
width: 520px;
height: 50px;
border-radius: 6px;
}
.line .psd{
border: 1px solid skyblue;
width: 520px;
height: 50px;
border-radius: 6px;
}
.line .phone{
border: 1px solid skyblue;
width: 340px;
height: 50px;
float: right;
border-radius: 6px;
}
.line select{
border: 1px solid skyblue;
width:150px;
height: 52px;
border-radius: 6px;
}
.re_button{
width: 520px;
height: 60px;
background-color: #3b78dd;
font-size: 22px;
color: white;
border-radius: 6px;
}
.button{
width: 520px;
height: 100px;
margin-top: 30px;
}
#xieyi{
color: #adadad;
font-size: 15px;
}
.footer{
width: 400px;
height: 50px;
box-sizing: border-box;
margin-top: 900px;
margin-left: 380px;
}
.line input{
font-size: 20px;
color: #adadad;
}
.left a{
background: url("https://qq-web.cdn-go.cn/zc.qq.com/1c037ba1/v3/img/logo.png") no-repeat;
background-size: 36px 43px;
height: 43px;
line-height: 43px;
padding-left: 46px;
margin-left: 30px;
margin-top: 18px;
font-size: 36px;
display: block;
color: #000000;
text-decoration: none;
}
.nav {
width: 100px;
height: 30px;
float: left;
box-sizing: border-box;
margin-top: 21px;
}
.nav ul li {
float: left;
list-style-type: none;
width: 80px;
/*margin-bottom: 15px;*/
}
.nav ul li a {
color: #cccccc;
text-decoration: none;
display: block;
}
.nav ul li a:link{
color: #cccccc;
opacity: 5;
}
.nav ul li a:hover {
color: #000000;
}
.nav li {
line-height: 30px;
text-align: center;
}
.nav ul li span{
display: none;
text-decoration: none;
position: absolute;
width: 80px;
}
.nav span ul li a{
color: #cccccc;
}
.nav ul li:hover span {
display: block;
text-decoration: none;
}
.feedback a:hover{
color: #000000;
}
</style>
</head>
<body>
<div class="left">
<a href="https://ssl.zc.qq.com/v3/index-chs.html?type=0" class="leftlogo">QQ</a>
</div>
<div class="right">
<div class="top">
<div class="lianghao">
<a href="https://haoma.qq.com/shop.html#from=zc" target="_blank">
<img src="images/qqlogo1.webp" srcset="images/qqlogo1.webp 2x"><div>QQ靓号</div>
</a>
</div>
<div class="nav">
<div id="sunav">
<ul>
<li class="sunav"><a href="#">简体中文</a>
<span>
<ul>
<li><a href="#">繁体中文</a></li>
<li><a href="#">English</a></li>
</ul>
</span>
</li>
</ul>
</div>
<img src="https://qq-web.cdn-go.cn/zc.qq.com/1c037ba1/v3/img/down.png" alt="">
</div>
<div class="feedback">
<a href="https://support.qq.com/products/14802">意见反馈</a>
</div>
</div>
<div class="bottom">
<div class="logo">
<span class="big">欢迎注册QQ</span> <br><br><br>
<span class="small">每一天，乐享沟通。</span>
<a href="https://ssl.zc.qq.com/v3/index-chs.html?type=3" target="_self">免费靓号</a>
</div>
<div class="line">
<input type="text" placeholder="昵称" class="name" ><br><br><br>
<input type="password" placeholder="密码" class="psd"><br><br><br>
<select name="" id="" >
<option value="">中国 +86</option>
<option value="">中国香港特别行政区 +852</option>
<option value="">中国澳门特别行政区 +853</option>
<option value="">中国台湾地区 +886</option>
<option value="">阿尔巴尼亚 +355</option>
</select>
<input type="text" placeholder="手机号码" class="phone">
</div>
<div class="button">
<input type="button" value="立即注册" class="re_button"><br><br>
<input type="checkbox" > <span id="xieyi">我已阅读并同意相关服务条款和隐私政策</span> <img src="https://qq-web.cdn-go.cn/zc.qq.com/1c037ba1/v3/img/down.png" alt="">
</div>
</div>
<div class="footer">
Copyright © 1998-2021Tencent All Rights Reserved
</div>
</div>
</body>
</html>
885
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
