python爬虫：使用scrapy框架对链家租房深度爬取，并存入redis、mysql、mongodb数据库

最新推荐文章于 2023-06-25 11:07:57 发布

刘建鑫

最新推荐文章于 2023-06-25 11:07:57 发布

阅读量977

点赞数 1

分类专栏： python爬虫文章标签： python爬虫链家租房 scrapy

本文链接：https://blog.csdn.net/weixin_44367450/article/details/89971678

版权

本文介绍了如何使用Python的Scrapy框架，对链家租房信息进行深度爬取，详细讲解了items.py、lianjia.py、pipelines.py和main.py四个关键文件的实现，最后将爬取的数据存储到Redis、MySQL和MongoDB数据库中。

摘要由CSDN通过智能技术生成

1.items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class LianjiaItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()

    pic = scrapy.Field()
    title = scrapy.Field()
    detail_url = scrapy.Field()
    price = scrapy.Field()
    publish_info = scrapy.Field()
    pic_list = scrapy.Field()
    house_code = scrapy.Field()
    ucid = scrapy.Field()
    agent_name = scrapy.Field()
    agent_phone = scrapy.Field()

2.lianjia.py

# -*- coding: utf-8 -*-
import scrapy
from LianJia.items import LianjiaItem
import re
import json
import requests


class LianjiaSpider(scrapy.Spider):
    name = 'lianjia'
    allowed_domains = ['lianjia.com']
    start_urls = ['https://www.lianjia.com/city/']

    def parse(self, response):
        # 获取到的是新房的url
        city_url_list = response.xpath("//div[@class='city_province']//li/a/@href").extract()
        # print(city_url_list)
        city_name_list = response.xpath("//div[@class='city_province']//li/a/text()").extract()
        for index in range(len(city_url_list)):
            city_name = city_name_list[index]
            city_url = city_url_list[index]
            # print(city_url)
            # 城市首字母
            city_alp = re.findall(r"https://(\w*).", city_url)[0]
            # print(city_alp)
            # 拼接租房城市url
            city_url = "https://" + city_alp + ".lianjia.com/zufang/"
            # print("--------------------{}开始下载-------------------------------".format(city_name))
            yield scrapy.Request(url=city_url,