pyspider爬虫设置延时

pyspider爬虫设置延时,在scrawl中添加function函数,设置时延.

另外可以正则匹配网址方法也可以参见本条方案

    def index_page(self, response):
        print(response)
        for each in response.doc('a[href^="http://www.zhanqi.tv/games/"]').items():
            if re.match("http://www.zhanqi.tv/games/\w+", each.attr.href, re.U):
                self.crawl(each.attr.href, 
                fetch_type='js', # fetch_type参数                   
                js_script="""
                function() {
                    setTimeout(window.scrollTo(0,document.body.scrollHeight), 5000);         
                }
                """,callback=self.list_page)