创建爬虫----爬取大众点评数据并存入mongoDB数据库

最新推荐文章于 2021-08-09 21:39:01 发布

yyq675886993

最新推荐文章于 2021-08-09 21:39:01 发布

阅读量710

点赞数 1

分类专栏： python爬虫文章标签：爬虫大众点评数据多页数据数据库

本文链接：https://blog.csdn.net/yyq675886993/article/details/74073485

版权

python爬虫专栏收录该内容

5 篇文章 0 订阅

订阅专栏

#爬取大众点评多页数据
from urllib.request import urlopen
from bs4 import BeautifulSoup
import pymongo


client = pymongo.MongoClient('localhost',27017)
dzdp = client['dzdp']
url_list = dzdp['url_list']
item_info = dzdp['item_info']

def get_page_link(page_number):
     for each_number in range(1,page_number):
          full_url="http://www.dianping.com/shop/2341453/review_all_grade?pageno={}".format(each_number)
          html=urlopen(full_url)
          bsObj=BeautifulSoup(html)

          nameList=bsObj.findAll("div",{"class":"J_brief-cont"})
          for name in nameList:
              print(name.get_text())
              url_list.insert_one({'text':name.get_text() })



s=get_page_link(71)
print(s)