今日份作业,今天爬取的是有关于菜价的数据。今天遇到的问题是自己爬取下来的页面与页面检查中Elements不一致,这个是有关于网页的一些知识,还得补这些知识😢😢😢,越学越觉得自己菜。
"""
逻辑
1.如何提取单个页面的数据
2.上线程池,多个页面同时抓取
"""
import requests
import json
import csv
from concurrent.futures import ThreadPoolExecutor
"""
import requests
import json
def download_one_page(url):
resp=requests.get(url)
html = resp.text
print(html)
print(type(html))
dict = json.loads(html)
print(dict)
list = dict['list']
for i in range(len(list)):
a = list[i]
name = a['prodName']
lowprice = a['lowPrice']
highprice = a['highPrice']
avgprice = a['avgPrice']
place = a['place']
unitInfo = a['unitInfo']
pubDate = a['pubDate']
b = [name,lowprice,avgprice,highprice,place,unitInfo,pubDate]
print(b)
"""
f=open('菜价.csv',mode='a',encoding='utf-8' )
csvwriter=csv.writer(f)
def download_one_page(url):
resp=requests.get(url,data)
#print(resp.text)
#print(type(resp.text))
## json.loads()可以识别出字符串中的json格式:去掉引号,并变成通用的json,所有语言都识别
dict=json.loads(resp.text)
#print(type(dict))
#print(dict)
list=dict['list']
#print(list)
for i in range(len(list)):
a=list[i]
name=a['prodName']
lowPrice=a['lowPrice']
highPrice=a['highPrice']
avgPrice=a['avgPrice']
place=a['place']
pubDate=a['pubDate']
b=[name,lowPrice,highPrice,avgPrice,place,pubDate]
#print(b)
#print(type(b))
csvwriter.writerow(b)
print(url,"提取完毕")
#html=etree.HTML(resp.text)
"""table=html.xpath("/html/body/div[2]/div/div/div/div[4]/div[1]/div/table")[0]
print(table)
trs=table.xpath("./tr")#xpath的相对查找
print(len(trs))
"""
if __name__ == '__main__':
url = "http://www.xinfadi.com.cn/getPriceData.html"
with ThreadPoolExecutor(50)as t:
for i in range(1, 100):
data = {
'limit': '20',
'current': i
}
t.submit(download_one_page,url,data)
print("全部下载完毕!")
今日学到的知识是:
1.多线程怎么创建with ThreadPoolExecutor(50)as t:
,怎么应用t.submit(download_one_page,url,data)
2.选中一段代码后,
按 shift + tab 回退空一格
按 tab 前进空一格
3.数据提取之json与xpath的区别和应用,就我这个案例里边是json适用于爬取出字典后使用,xpath是对应HTML数据提取。
参考文献:
https://blog.csdn.net/qq_35709559/article/details/84928920