爬虫实战：制作简单的页面采集器

以百度为例制作简单的页面采集器

首先尝试利用百度搜做引擎搜索任意信息：

url如下：
观察可知 我们需要的URL如下：
https://www.baidu.com/s?wd=123

首先利用python脚本发送请求：

import requests

url = 'https://www.baidu.com/s'
kw = input('please input your keywords:')
param = {
    'wd':kw
}
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
response = requests.get(url=url,params=param,headers=headers)
response.encoding = 'utf-8'
print(response.text)

返回结果如下：

将爬取下来的内容保存到相应的html文件中：
完整代码：

import requests

url = 'https://www.baidu.com/s?'
kw = input('input your keyword:')
param = {
    'wd':kw
}
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
response = requests.get(url=url,params=param,headers=headers)
response.encoding = 'utf-8'
page_text = response.text
filename = kw+'.html'
with open(filename,'w',encoding='utf-8') as fp:
    fp.write(page_text)
print('保存成功！')

在响应文件夹中找到该html文件
用chrome打开如下：