python3学习之路 -- 9.1.2）- 获取豆瓣250top-CSDN博客

本文链接：https://blog.csdn.net/lurongrong520_/article/details/124794154

本文介绍了使用Python3进行网络爬虫实战，通过爬取豆瓣网站获取Top250书籍的信息，包括书名、作者、评分等关键数据，详细解析了爬虫代码的实现过程。

摘要由CSDN通过智能技术生成

# 拿到页面源代码  requests
# 通过re提取想要的信息  re
import requests
import re
import csv

url = "https://movie.douban.com/top250"
header = {
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
}
resp = requests.get(url, headers=header)
page_content = resp.text

'''
拿到的数据格式类似如下:
        <li>
            <div class="item">
                <div class="pic">
                    <em class="">1</em>
                    <a href="https://movie.douban.com/subject/1292052/">
                        <img width="100" alt="肖申克的救赎" src="https://img2.doubanio.com/view/photo/s_ratio_poster/public/p480747492.webp" class="">
                    </a>
                </div>
                <div class="info">
                    <div class="hd">
                        <a href="https://movie.douban.com/subject/1292052/" class="">
                            <span class="title">肖申克的救赎</span>
                                    <span class="title">&nbsp;/&nbsp;The Shawshank Redemption</span>
                                <span class="other">&nbsp;/&nbsp;月黑高飞(港)  /  刺激1995(台)</span>
                        </a>


                            <span class="playable">[可播放]</span>
                    </div>
                    <div class="bd">
                        <p class="">
                            导演: 弗兰克·德拉邦特 Frank Darabont&nbsp;&nbsp;&nbsp;主演: 蒂姆·罗宾斯 Tim Robbins /...<br>
                            1994&nbsp;/&nbsp;美国&nbsp;/&nbsp;犯罪 剧情
                        </p>

                        
                        <div class="star">
                                <span class="rating5-t"></span>
                                <span class="rating_num" property="v:average">9.7</span>
                                <span property="v:best" content="10.0"></span>
                                <span>2615051人评价</span>
                        </div>

                            <p class="quote">
                                <span class="inq">希望让人自由。</span>
                            </p>
                    </div>
                </div>
            </div>
        </li>
'''
# 解析数据
obj = re.compile(r'<li>.*?<div class="item">.*?<span class="title">(?P<title>.*?)</span>'
                 r'.*?''<p class="">.*?<br>(?P<year>.*?)&nbsp.*?<span class="rating_num" '
                 r'property="v:average">(?P<pingfen>.*?)</span>'
                 r'.*?<span>(?P<pingjia>.*?)人评价', re.S)
'''
# 开始匹配
result = obj.finditer(page_content)
for i in result:
    print(i.group("title"))
    print(i.group("year").strip())
    print(i.group("pingfen"))
    print(i.group("pingjia"))
print("over!")
'''
# 开始匹配，并将存到文件 格式类似： 肖申克的救赎,1994,9.7,2615095
result = obj.finditer(page_content)
#print(result) #<callable_iterator object at 0x000001E1CA83C340>,此时可以用迭代循环group输出内容
f = open("02_2_douban.csv", "w", encoding="UTF-8", newline="")
#基于文件对象构建csv写入对象
csvwriter = csv.writer(f)
csvwriter.writerow(['电影名字', '年份', '评分', '评价人数'])
for i in result:
    # 将数据放到字典
    dic = i.groupdict()
    #print(dic.values())
    dic['year'] = dic['year'].strip()
    csvwriter.writerow(dic.values())
f.close()
print("over!")