Web_Crawler

最新推荐文章于 2022-04-27 12:21:16 发布

Lambert_Zhang

最新推荐文章于 2022-04-27 12:21:16 发布

阅读量405

点赞数

分类专栏： python

本文链接：https://blog.csdn.net/Lambert_Zhang/article/details/43925385

版权

python 专栏收录该内容

14 篇文章 0 订阅

订阅专栏

import requests
from bs4 import BeautifulSoup

def trade_spider(max_page):
    page = 1
    i = 0
    while page <= max_page:
        url = 'https://www.thenewboston.com/search.php?type=0&sort=reputation&page=' + str(page)
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text)
        for link in soup.findAll('a',{'class':'desc-title'}):
            href = 'https://www.thenewboston.com'+ link.get('href')
            title = link.string
            print(title)
            print(href)
            i += 1
            print('\n')
        page += 1
    print(i)


trade_spider(5)