爬虫爬取电影天堂电影链接

 

比较热爱python，最近在用eclipse写java web，那就使用eclipse+PyDv配置环境，小试一次爬虫吧~

看电影还要到处找资源，索性自己直接爬取电影链接，只要在迅雷上crtl+c/v就可以边播边下了~

仅以用来学习娱乐呦~~

进入正题：

　　网页打开电影天堂，发现是一个非常好解析的网页，它网页元素构成简单，容易理解，爬取就不用太多验证控制，首先下载网页源码，然后保存在txt文件中，然后用python读取，进行获取。（完整精简代码在文后）

　　

# -*- coding: utf-8 -*-
import pandas as pd
import numpy as np
import urllib
from urllib import request
from bs4 import BeautifulSoup
from pandas.tests.frame.test_validate import dataframe
import requests
import sys
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer,encoding='utf-8')


url =r'e:\tiantang.txt'
resp=open(url,encoding='gb18030',errors='ignore')
s = BeautifulSoup(resp,"lxml")

mingzitag=s.findAll('a')
#print(mingzitag)
href=[]
mingzis=[]
for tag in mingzitag:
    mingzis.append(str(tag.text))
    href.append(tag['href'])
#print(href)
view=pd.DataFrame(mingzis,columns=['NAME'])
view2=pd.DataFrame(href,columns=['URL'])
view3=view.join(view2)
print(view3)

结果：

                              NAME                                              URL

0                                                                       /index.html

1                             最新影片   http://www.ygdy8.net/html/gndy/dyzz/index.html

2                             经典影片        http://www.ygdy8.net/html/gndy/index.html

3                             国内电影  http://www.ygdy8.net/html/gndy/china/index.html

4                             欧美电影  

16                            本站首页                            http://www.dytt8.net/

17                              电影                            /html/gndy/index.html

18                            最新电影                       /html/gndy/dyzz/index.html

19                            日韩电影                      /html/gndy/rihan/index.html

20                            欧美电影                      /html/gndy/oumei/index.html

21                            国内电影                      /html/gndy/china/index.html

22                            综合电影                       /html/gndy/jddy/index.html

23                  2017年科幻动作《全球风暴              /html/gndy/dyzz/20180103/55959.html

24                  2017年高分恐怖《小丑回魂              /html/gndy/dyzz/20171230/55920.html

25                  2017年惊悚悬疑《雪人/雪              /html/gndy/jddy/20171224/55874.html

26                 2017年喜剧《性别之战》BD              /html/gndy/jddy/20171222/55861.html

73      2018年剧情动作《反贪风暴3/L风暴》HD国语中字              /html/gndy/dyzz/20181026/57680.html

74    2018年8.0分动画喜剧《超人总动员2》HD中英双字幕              /html/gndy/dyzz/20181025/57675.html

75                             [2]                                   list_23_2.html

76                             [3]                                   list_23_3.html

77                             [4]                                   list_23_4.html

85                           电影APP                   https://www.dytt8.net/app.html

86                            下载声明