java爬虫爬取豆瓣电影_小爬虫之爬取豆瓣电影排行榜

1.技术路线

python 3.6.0

scrapy 1.4.0

2.任务

爬取豆瓣电影排行榜电影相关信息

2.1查看豆瓣的robots

User-agent: *

Disallow: /subject_search

Disallow: /amazon_search

Disallow: /search

Disallow: /group/search

Disallow: /event/search

Disallow: /celebrities/search

Disallow: /location/drama/search

Disallow: /forum/

Disallow: /new_subject

Disallow: /service/iframe

Disallow: /j/

Disallow: /link2/

Disallow: /recommend/

Disallow: /trailer/

Disallow: /doubanapp/card

Sitemap: https://www.douban.com/sitemap_index.xml

Sitemap: https://www.douban.com/sitemap_updated_index.xml

# Crawl-delay: 5

User-agent: Wandoujia Spider

Disallow: /

按照规定设置好爬虫的相关设置，防止被禁。

关于如何防止被禁的策略网上有很多，这里只是简单实用一二。

1.设置 ：DOWNLOAD_DELAY = 5

2.禁止cookies ：COOKIES_ENABLED = False

3.使用User Agent池

关于设置user agent方法网上有很多总结帖

方法一

修改setting.py USER_AGENT。

import random

user_agent_list= [

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",

"Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",

"Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",

"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",

"Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",

"Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",

"Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",

"Mozilla/