requests html 下一页url,python-用BeautifulSoup和Requests抓取多个分页链接

Python初学者在这里.我正在尝试从one category on dabs.com抓取所有产品.我设法在给定页面上抓取了所有产品,但是在遍历所有分页的链接时遇到了麻烦.

现在,我尝试将所有的分页按钮都用span class =’page-list“隔离开,但即使这样也无法正常工作.理想情况下,我想让抓取工具继续单击下一步,直到将所有产品都刮到所有页面,我该怎么做？

非常感谢任何输入

from bs4 import BeautifulSoup

import requests

base_url = "http://www.dabs.com"

page_array = []

def get_pages():

html = requests.get(base_url)

soup = BeautifulSoup(html.content, "html.parser")

page_list = soup.findAll('span', class="page-list")

pages = page_list[0].findAll('a')

for page in pages:

page_array.append(page.get('href'))

def scrape_page(page):

html = requests.get(base_url)

soup = BeautifulSoup(html.content, "html.parser")

Product_table = soup.findAll("table")

Products = Product_table[0].findAll("tr")

if len(soup.findAll('tr')) > 0:

Products = Products[1:]

for row in Products:

cells = row.find_all('td')

data = {

'description' : cells[0].get_text(),

'price' : cells[1].get_text()

}

print data

get_pages()

[scrape_page(base_url + page) for page in page_array]

解决方法:

他们的下一页按钮标题为“下一页”,您可以执行以下操作：

import requests

from bs4 import BeautifulSoup as bs

url = 'www.dabs.com/category/computing/11001/'

base_url = 'http://www.dabs.com'

r = requests.get(url)

soup = bs(r.text)

elm = soup.find('a', {'title': 'Next'})

next_page_link = base_url + elm['href']

希望能有所帮助.

标签：for-loop,beautifulsoup,web-scraping,screen-scraping,python

来源： https://codeday.me/bug/20191028/1954008.html