菜鸟初学Python网页数据抓取01

最新推荐文章于 2024-08-10 21:26:27 发布

天天吃猪砂

最新推荐文章于 2024-08-10 21:26:27 发布

阅读量65

点赞数

文章标签：学习 python Powered by 金山文档

本文链接：https://blog.csdn.net/MyASCOT/article/details/129588704

版权

菜鸟学习python网页数据抓取，以Bing为用户代理，搜索“知网”，抓取到响应后数据界面。

初步学习后的代码如下：

import requests

# 1 explict Url
url = 'https://cn.bing.com/search'
# 2.1 ua pretending
headers = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111\
    .0.1661.41'
# 2.2 pakage params into dictionary
kw = input('please enter a word:')
param = {
    'query' : kw
}
# 2.3 excuting request for the resource in url
# 3 and get response data from servers
response = requests.get(url = url, params = param, headers =headers)
# 4 sustainably store data
page_text = response.text
fileName = kw + '.html'
with open(fileName,'w', encoding = 'utf-8') as fp:
    fp.write(page_text)
print(fileName,'sucessfully storing!!!')

运行后出现报错：

经了解后，发现'items'是 dic 的属性，而语句中的 ‘headers’是字符串形式，因此出现报错。

headers = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111\
    .0.1661.41'

尝试解决方式如下，将headers封装进 dic ：

ua = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36 Edg/111\
    .0.1661.41'
headers = {
    'headers':ua
}

问题解决，运行结果和抓取数据如下：