爬取淘宝数据

最新推荐文章于 2024-09-09 15:45:46 发布

bus_lupe

最新推荐文章于 2024-09-09 15:45:46 发布

阅读量1k

点赞数

分类专栏：爬虫

本文链接：https://blog.csdn.net/bus_lupe/article/details/101929084

版权

爬虫专栏收录该内容

19 篇文章 0 订阅

订阅专栏

创建数据库

mysql> create database taobao;

创建表

create table goods (
	id 
    title varchar(100),
    price varchar(100),
    link varchar(100),
    comment varchar(100)
)

创建爬虫项目

scrapy startproject taobao
cd taobao
# 创建爬虫文件
scrapy genspider -t basic app taobao.com

修改items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class TaobaoItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    title = scrapy.Field()
    link = scrapy.Field()
    price = scrapy.Field()
    comment = scrapy.Field()

修改settings.py

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'taobao.pipelines.TaobaoPipeline': 300,
}