2021-05爬虫模板

一段简单的页面爬虫代码：爬取微信读书主页

import requests
url = "http://weread.qq.com"
try:
    r = requests.get(url)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text[:1000])
except:
    print("爬取失败")

效果展示：

爬取搜索引擎中的数据代码段：

import requests
keyword = "Python"
try:
    kv = {'wd':keyword}
    r = requests.get("http://www.baidu.com/s",params=kv)
    print(r.request.url)
    r.raise_for_status()
    print(len(r.text))
except:
    print("爬取失败")

效果展示：

爬取一个图片或一段视频：

import requests
import os
url = "blob:https://www.iqiyi.com/9eabe47d-2db2-43a5-b0da-702e73add90e"
root = "C:\\Users\\Administrator\\Desktop\\python_work\\vids\\"
path = root + url.split('/')[-1]
try:
    if not os.path.exists(root):
        os.mkdir(root)
    if not os.path.exists(path):
        r = requests.get(url)
        with open(path, 'wb') as f:
            f.write(r.content)
            f.close()
            print("文件成功保存")
    else:
        print("文件已存在")
except:
    print("爬取失败")

引用BeautifulSoup库：

 import requests
 r = requests.get("http://python123.io/ws/demo.html")
 demo = r.text
 from bs4 import BeautifulSoup
 soup = BeautifulSoup(demo, "html.parser")
 print(soup.prettify())

查找网页内URL链接：

>>> import requests
>>> r = requests.get("http://python123.io/ws/demo.html")
>>> demo = r.text;
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(demo, "html.parser")
>>> for link in soup.find_all('a')
>>> for link in soup.find_all('a'):
	print(link.get('href'))

效果展示：