day2 作业

最新推荐文章于 2024-09-13 16:31:32 发布

whitewolf789

最新推荐文章于 2024-09-13 16:31:32 发布

阅读量180

点赞数

文章标签：爬虫

本文链接：https://blog.csdn.net/whitewolf789/article/details/124695686

版权

该代码使用Python的requests库爬取豆瓣Top250电影的详细信息，包括电影名、导演、主演、语言、上映时间和豆瓣评分，并将这些信息写入CSV文件中。通过正则表达式解析HTML内容，实现了数据的抓取和存储。

摘要由CSDN通过智能技术生成

import re
import requests
import random
import csv
import time
from tqdm import tqdm
f1 = open('豆瓣top250.csv', 'w', encoding='utf-8',newline='')
#将csv中的写方法赋给一个变量
writer = csv.writer(f1)
writer.writerow(['电影名', '导演', '主演', '语言', '上映时间', '豆瓣评分'])
for i in tqdm(range(0, 250, 25)):
url = f'https://movie.douban.com/top250?start={i}&filter='
headers = {
'User-Agent' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36'
}
resp = requests.get(url=url, headers=headers)

if resp.status_code == 200:
resp.encoding = 'utf-8'
# print(resp.text, type(resp.text))
# data = BeautifulSoup(resp.text, 'html.parser')
# print(data)
# print(resp.text)
else:
print(resp.status_code)
li_re =r' <div class="item">.*? <div class="pic">.*?<em class="">.+?</em>.*?<a href="(.*?)">'
url = re.findall(li_re, resp.text,
re.DOTALL)
# print(url, len(url))
for i in range(len(url)):
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36'
}
resp = requests.get(url[i], headers=headers)
resp.encoding = 'utf-8'
# print(resp.text)
name = re.findall(r'"@context": "http://schema.org",.*?"name": "(.*?)"', resp.text, re.DOTALL)
director = re.findall(r'rel="v:directedBy">(.*?)</a>', resp.text, re.DOTALL)
actor = re.findall(r'rel="v:starring">(.*?)</a>', resp.text, re.DOTALL)
language = re.findall(r'<span class="pl">语言:</span> (.*?)<br/>', resp.text, re.DOTALL)
date = re.findall(r'property="v:initialReleaseDate" content="(.*?)">', resp.text, re.DOTALL)
grades = re.findall(r'property="v:average">(.*?)</strong>', resp.text, re.DOTALL)
# print(name[0])
# print(director[0])
# print(actor[0])
# print(language[0])
# print(date[0])
# print(grades[0])
writer.writerow([[i1 for i1 in name], [i2 for i2 in director], [i3 for i3 in actor] if len(actor) else '无', [i4 for i4 in language], [i5 for i5 in date], grades[0]])
# time.sleep(2*random.random())
# time.sleep(2 * random.random())
f1.close()
print('写入完成')