scrapy中pipeline数据去重和更新

最新推荐文章于 2022-09-07 18:20:02 发布

風の住む街~

最新推荐文章于 2022-09-07 18:20:02 发布

阅读量870

点赞数

分类专栏： # Scrapy爬虫框架

本文链接：https://blog.csdn.net/weixin_38924500/article/details/91349424

版权

Scrapy爬虫框架专栏收录该内容

18 篇文章 3 订阅

订阅专栏

class NewsEducationPipeline(object):
    def __init__(self):
        self.connect = pymysql.connect(
            host=settings.MYSQL_HOST,
            db=settings.MYSQL_DBNAME,
            user=settings.MYSQL_USER,
            passwd=settings.MYSQL_PASSWD,
            charset='utf8',
            use_unicode=True)
        self.cursor = self.connect.cursor()
    def process_item(self, item, spider):
        item["crawled"] = datetime.now()
        if item.__class__ == NewsEducationItem:
            crawled = item["crawled"]
            new_id = item["new_id"]
            title  = item["title"]
            url = item["url"]
            intro = item["intro"]
            img = item["img"]
            kl = item["kl"]
            time = item["time"]
            media = item["media"]
            source = item["source"]
            try:
                self.cursor.execute("""select * from new_info where url = %s""", url)
                ret = self.cursor.fetchone()
                if ret:
                    self.cursor.execute(
                        """update new_info set new_id = %s,title = %s,intro = %s,img = %s,
                            url = %s,kl = %s,time = %s,crawled =%s,media =%s,source =%s
                            where url  = %s""",
                        (new_id, title,intro,img,url, kl,time,crawled,media,source,url,))
                else:
                    self.cursor.execute(
                        """insert into new_info(new_id,title,intro,img,url,kl,time,crawled, media,source)
                          value (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)""",
                        (new_id,title,intro,img,url,kl,time,crawled, media,source,))
                    self.connect.commit()
            except Exception as error:
                print("错误")
            return item