多线程爬虫

今日份作业，今天爬取的是有关于菜价的数据。今天遇到的问题是自己爬取下来的页面与页面检查中Elements不一致，这个是有关于网页的一些知识，还得补这些知识😢😢😢，越学越觉得自己菜。

"""
逻辑
1.如何提取单个页面的数据
2.上线程池，多个页面同时抓取
"""
import requests
import json
import csv
from concurrent.futures import ThreadPoolExecutor
"""
import requests
import json

def download_one_page(url):
    resp=requests.get(url)
    html = resp.text
    print(html)
    print(type(html))
    dict = json.loads(html)
    print(dict)
    list = dict['list']
    for i in range(len(list)):
        a = list[i]
        name = a['prodName']
        lowprice = a['lowPrice']
        highprice = a['highPrice']
        avgprice = a['avgPrice']
        place = a['place']
        unitInfo = a['unitInfo']
        pubDate = a['pubDate']
        b = [name,lowprice,avgprice,highprice,place,unitInfo,pubDate]
        print(b)
"""
f=open('菜价.csv',mode='a',encoding='utf-8' )
csvwriter=csv.writer(f)
def download_one_page(url):

    resp=requests.get(url,data)
    #print(resp.text)
    #print(type(resp.text))
    ## json.loads()可以识别出字符串中的json格式：去掉引号，并变成通用的json，所有语言都识别
    dict=json.loads(resp.text)
    #print(type(dict))
    #print(dict)
    list=dict['list']
    #print(list)
    for i in range(len(list)):
        a=list[i]
        name=a['prodName']
        lowPrice=a['lowPrice']
        highPrice=a['highPrice']
        avgPrice=a['avgPrice']
        place=a['place']
        pubDate=a['pubDate']
        b=[name,lowPrice,highPrice,avgPrice,place,pubDate]
        #print(b)
        #print(type(b))
        csvwriter.writerow(b)
    print(url,"提取完毕")

    #html=etree.HTML(resp.text)
    """table=html.xpath("/html/body/div[2]/div/div/div/div[4]/div[1]/div/table")[0]
    print(table)
    trs=table.xpath("./tr")#xpath的相对查找
    print(len(trs))
"""



if __name__ == '__main__':
    url = "http://www.xinfadi.com.cn/getPriceData.html"
    with ThreadPoolExecutor(50)as t:

        for i in range(1, 100):
            data = {
                'limit': '20',
                'current': i
            }
            t.submit(download_one_page,url,data)




    print("全部下载完毕！")

今日学到的知识是：
1.多线程怎么创建with ThreadPoolExecutor(50)as t:，怎么应用t.submit(download_one_page,url,data)
2.选中一段代码后，
按 shift + tab 回退空一格
按 tab 前进空一格
3.数据提取之json与xpath的区别和应用，就我这个案例里边是json适用于爬取出字典后使用，xpath是对应HTML数据提取。

参考文献：

https://blog.csdn.net/qq_35709559/article/details/84928920