一段简单的页面爬虫代码:爬取微信读书主页
import requests
url = "http://weread.qq.com"
try:
r = requests.get(url)
r.raise_for_status()
r.encoding = r.apparent_encoding
print(r.text[:1000])
except:
print("爬取失败")
效果展示:
爬取搜索引擎中的数据代码段:
import requests
keyword = "Python"
try:
kv = {'wd':keyword}
r = requests.get("http://www.baidu.com/s",params=kv)
print(r.request.url)
r.raise_for_status()
print(len(r.text))
except:
print("爬取失败")
效果展示:
爬取一个图片或一段视频:
import requests
import os
url = "blob:https://www.iqiyi.com/9eabe47d-2db2-43a5-b0da-702e73add90e"
root = "C:\\Users\\Administrator\\Desktop\\python_work\\vids\\"
path = root + url.split('/')[-1]
try:
if not os.path.exists(root):
os.mkdir(root)
if not os.path.exists(path):
r = requests.get(url)
with open(path, 'wb') as f:
f.write(r.content)
f.close()
print("文件成功保存")
else:
print("文件已存在")
except:
print("爬取失败")
引用BeautifulSoup库:
import requests
r = requests.get("http://python123.io/ws/demo.html")
demo = r.text
from bs4 import BeautifulSoup
soup = BeautifulSoup(demo, "html.parser")
print(soup.prettify())
查找网页内URL链接:
>>> import requests
>>> r = requests.get("http://python123.io/ws/demo.html")
>>> demo = r.text;
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(demo, "html.parser")
>>> for link in soup.find_all('a')
>>> for link in soup.find_all('a'):
print(link.get('href'))
效果展示: