python request版 极客学院爬虫V1

1. 定向爬取极客学院视频，原本只有年费VIP只能下载，经过分析，只要找个免费体验VIP即可爬取所有视频
2. 涉及的基本技术：python xpath 正则 com
3. 通过python调用迅雷从组件，实现自动创建文件夹和自动添加批量下载任务，前提要成功安装迅雷和迅雷组件
4. 思路：path路径爬取所有标签-》搜索页面所有该课程分类-》课程页面获取课程明细-》正则分析视频地址
5. 极客学院的一直在改进，可能需要自己改进

import requests
from lxml import etree
import re
import sys, os, glob,time
import scrapy

reload(sys)
sys.setdefaultencoding(“utf-8”)

**# baesurl = “http://www.jikexueyuan.com/search/s/q_”

base_path = “f:/jike/”**

heanders Cookie需要自己抓取，否则只能抓取到免费课程

headers = {
“Host”: “www.jikexueyuan.com”,
“User-Agent”: “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0”,
“Accept”: “text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8”,
“Accept-Language”: “zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3”,
“Accept-Encoding”: “gzip, deflate”,
“Cookie”: “_ga=GA1.2.1700377703.1438173034; Hm_lvt_f3c68d41bda15331608595c98e9c3915=1438173034; MECHAT_LVTime=1438179151498; MECHAT_CKID=cookieVal=006600143817303272961295; stat_ssid=1438985023415; stat_uuid=1438173038588973692017; connect.sid=s%3AWt8IWWxkVZ6zlhop7HpbG-vtX_qtwIAs.QC1tYy4qV1bHOMDN0UTUfScLKFncl4NY5zAk1SS17Kw; QINGCLOUDELB=37e16e60f0cd051b754b0acf9bdfd4b5d562b81daa2a899c46d3a1e304c7eb2b|VbjfT|VbjfT; Hm_lpvt_f3c68d41bda15331608595c98e9c3915=1438179151; stat_isNew=0; stat_fromWebUrl=; _gat=1; uname=jike_76; uid=2992598; code=SMapFI; authcode=d572TzIvHFXNIVNXcNf4vI5lv1tQlyEknAG4m0mDQmvMRPa4VhDOtJXOSfO%2BeVFVPzra8M1sEkEzxqLX9qRgS6nWhd5VMobbDpeqvJ726i54TqMoDo81P4OlhQ”,
“Connection”: “keep-alive”
}

clas