python爬取iphone11在售信息

爬取示例

一、首先分析京东网址规律

首先在京东搜索栏中输入“iphone11” 并点击查询，出来如下结果

然后依次复制前5页的网址，如下

# https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=1&s=1&click=0
# https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=3&s=56&click=0 #60
# https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=5&s=116&click=0 #60
# https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=7&s=176&click=0
# https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=9&s=236&click=0

观察发现，只有最后两个数值“&page=&s=”在变化，其他都相同，这样就方便我们定义url。
继续分析，目前只有第一页没有规律，从第二页开始，参数page递增2，参数s递增60，那么我们应该分别构建两个url语句如下

urls = ['https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=1&s=1&click=0']

 urls=['https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page={}&s={}&click=0'.format(str(i),str(j)) for i in range(3,11,2) for j in range(56,237,60)]

二、分析京东html页面容器

在京东网站点击右键，点击“检查”，如下图

此时打开调试窗口，如下图

然后寻找当前每家店铺所处的容器div，


进而找到想要爬取的字段所处的容器div

三、编写爬取页面路径和内容

    prices=soup.select('div.gl-i-wrap>div.p-price')
    ads=soup.select('div.gl-i-wrap>div.p-name')
    shops = soup.select('div.gl-i-wrap>div.p-shop')


    for price,ad,shop in zip(prices,ads,shops):
        i=i+1
        data={
            'index':i,
            'price':price.get_text().split()[0],
            'shop':shop.get_text().split(),
            'ad':ad.get_text().strip()
        }

四、完整代码

import requests
from bs4 import BeautifulSoup
import time
import sys

headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36'}

def get_info(url,i):
    wb_data=requests.get(url,headers=headers)
    soup=BeautifulSoup(wb_data.text,'lxml')

    prices=soup.select('div.gl-i-wrap>div.p-price')
    ads=soup.select('div.gl-i-wrap>div.p-name')
    shops = soup.select('div.gl-i-wrap>div.p-shop')


    for price,ad,shop in zip(prices,ads,shops):
        i=i+1
        data={
            'index':i,
            'price':price.get_text().split()[0],
            'shop':shop.get_text().split(),
            'ad':ad.get_text().strip()
        }
        print(data)
    return i
if __name__=='__main__':
    i = 0
    urls = ['https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page=1&s=1&click=0']
    for url in urls:
        i=get_info(url,i)

    urls=['https://search.jd.com/Search?keyword=iphone11&wq=iphone11&pvid=9c3dd3ff63fc41e2893763e521450969&page={}&s={}&click=0'.format(str(i),str(j)) for i in range(3,11,2) for j in range(56,237,60)]
    for url in urls:
        i=get_info(url,i)