scrapy框架爬取安居客成都某区域二手房信息并保存至Mysql

本文链接：https://blog.csdn.net/pythoner111/article/details/100174910

本文档介绍如何利用Python的scrapy框架爬取安居客成都特定区域的二手房信息，并将数据存储到Mysql数据库。首先通过scrapy创建项目和spider，接着在settings中配置参数，items.py中定义字段，然后在spider的py文件中编写爬取逻辑，并使用pymsql库连接数据库。最后在pipelines.py中处理数据并执行入库操作。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

需求：利用scrapy框架爬取安居客成都某区的房屋信息，并保存到Mysql。

环境：pycharm,py3.6,scrapy ,bs4，mysql

第一步：创建scrapy项目，cmd执行命令 scrapy startpeoject anjuke_spider，创建了一个名为anjuke_spider的scrapy项目，然后再输入 scrapy crawl anjuke_house www.anjuke.com/ ，在项目中创建了一个名为anjuke_house 的spider。然后在项目中看到如下文件：
在这里插入图片描述
在settings 文件中配置框架相应的参数。注意没有被注释的语句就是相关配置。

# -*- coding: utf-8 -*-

# Scrapy settings for anjukespider project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'anjukespider'

SPIDER_MODULES = ['anjukespider.spiders']
NEWSPIDER_MODULE = 'anjukespider.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'anjukespider (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The dow