python3爬虫爬取豆瓣电影并保存到sql serve数据库

最新推荐文章于 2021-11-15 11:34:21 发布

chj_orange

最新推荐文章于 2021-11-15 11:34:21 发布

阅读量7.8k

点赞数 2

分类专栏： python爬虫文章标签： python 爬虫豆瓣

本文链接：https://blog.csdn.net/chj_orange/article/details/65937230

版权

本人是比较喜欢电影的，最近自学了python爬虫，特来爬了下豆瓣来练习一下。

语言版本：Python 3.3

编辑器：eclipse + Pydiv

数据库：SQL Server 2008

首先，如果自己的爬虫不想被目标网站的杀虫剂灭掉的话，就要采取一些措施，了解一些反爬虫机制，对自己的爬虫加以伪装。这里也是用了一些普通的方法来伪装。在requests.get()请求加上headers{ }来伪装成浏览器访问。并且构建一个代理IP池。

下面上代码干货！

1，构建一个代理IP池

首先获取一定数量的代理IP，当然啦，这里是爬取代理IP的网站，并且将一定数量的IP保存到本地。

国内h高匿代理IP: http://www.xicidaili.com/nn

代码如下：

#coding=utf-8
'''
Created on 2017年3月23日
获取代理IP
@author: chj
'''

import time
import requests
from bs4 import BeautifulSoup

#num获取num页 国内高匿ip的网页中代理数据
def fetch_proxy(num):
    
    api = 'http://www.xicidaili.com/nn/'
    header = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 \
    (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'
    }
    #保存到本地文件
    fp = open('F:/host.txt', 'a+', encoding=('utf-8'))
    
    for i in range(num+1):
        api = api + str(i)
        respones = requests.get(url=api, headers=header)
        soup = BeautifulSoup(respones.text, 'html.parser')
        container = soup.find_all(name='tr',attrs={'class':'odd'})
        for tag in container:
            try:
                con_soup = BeautifulSoup(str(tag),'html.parser')
                td_list = con_soup.find_all('td')
                ip = str(td_list[1])[4:-5]
                port = str(td_list[2])[4:-5]
                IPport = ip + '\t' + port + '\n'
                fp.write(IPport)
            except Exception as e:
                print('No IP！')
        #这里要控制爬取频率，友好爬虫
        time.sleep(1)
    fp.close()
    
#执行函数    
fetch_proxy(10)

2，要设置请求头这里构建了一个完整的浏览器访问url的请求头包括：Accept，Accept-Encoding，Accept-Language，Connection，Cookie，Host，Referer，User-Agent；有些时候爬取一些网站只需要加上User-Agent就可以绕过反爬虫机制，但是构建一个完整的请求头会增大绕过反爬机制的几率，所以不要怕麻烦，给自己的爬虫一个坚固的盔甲。

代码如下：

1，不同浏览器的User-Agent

def getUA():
    UA_list = ["Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36","Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) App leWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53",
              "Mozilla/5.0 (Windows; U; Windows NT 5.2) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.27 Safari/525.13","Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ;  QIHU 360EE)",
              "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; Maxthon/3.0)",
              "'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
              "Mozilla/5.0 (Macintosh; U; IntelMac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1Safari/534.50",
              "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0"]
    return UA_list

2,构建一个Cookie池

def getCookie():
    cookie_list=['ll="118237"; bid=KvZaCPoQKJk; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1490332252%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _vwo_uuid_v2=912FADCFB9317239E37ED315ED9F488F|7ad5255e106b04510fc5d59ed2abc07b; _pk_id.100001.4cf6=490a4f915c676455.1490320279.2.1490333053.1490320313.; _pk_ses.100001.4cf6=*; __utma=30149280.90895996.1490320277.1490320277.1490332251.2; __utmb=30149280.1.10.1490332251; __utmc=30149280; __utmz=30149280.1490332251.2.2.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; __utma=223695111.2019742000.1490320279.1490320279.1490332252.2; __utmb=223695111.0.10.1490332252; __utmc=223695111; __utmz=223695111.1490332252.2.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/',
                 'll="118237"; bid=KvZaCPoQKJk; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1490332252%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _vwo_uuid_v2=912FADCFB9317239E37ED315ED9F488F|7ad5255e106b04510fc5d59ed2abc07b; _pk_id.100001.4cf6=490a4f915c676455.1490320279.2.1490334307.1490320313.; _pk_ses.100001.4cf6=*; __utma=30149280.90895996.1490320277.1490320277.1490332251.2; __utmb=30149280.1.10.1490332251; __utmc=30149280; __utmz=30149280.1490332251.2.2.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; __utma=223695111.2019742000.1490320279.1490320279.1490332252.2; __utmb=223695111.0.10.1490332252; __utmc=223695111; __utmz=223695111.1490332252.2.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/',
                 'll="118237"; bid=KvZaCPoQKJk; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1490332252%2C%22https%3A%2F%2Fwww.douban.com%2F%22%5D; _vwo_uuid_v2=912FADCFB9317239E37ED315ED9F488F|7ad5255e106b04510fc5d59ed2abc07b; _pk_id.100001.4cf6=490a4f915c676455.1490320279.2.1490334337.1490320313.; _pk_ses.100001.4cf6=*; __utma=30149280.90895996.1490320277.1490320277.1490332251.2; __utmb=30149280.1.10.1490332251; __utmc=30149280; __utmz=30149280.1490332251.2.2.utmcsr=baidu|utmccn=(organic)|utmcmd=organic; __utma=223695111.2019742000.1490320279.1490320279.1490332252.2; __utmb=223695111.0.10.1490332252; __utmc=223695111; __utmz=223695111.1490332252.2.2.utmcsr=douban.com|utmccn=(referral)|utmcmd=referral|utmcct=/',
                 'll="118237"; bid=KvZaCPoQKJk; _pk_ref.100001.4cf6=%5B%22%22%2C%22%22%2C1490332252