pip install requests 安装requests库
基本顺序:
import requests r=requests.get("url路径") r.status_code #成功返回200 r.encooding=r.apparent_encoding r.rext
通用框架:
import requests def getHTMLText(url): try: r=requests.get(url,timeout=30) r.raise_for_status() #如果状态不是200,引发HTTPError异常 r.encoding=r.apparent_encoding return r.text except: return "产生异常" if __name__== "__main__": url="https://www.baidu.com" print(getHTMLText(url))
对于某些情况,模拟浏览器访问
import requests url="https://www.amazon.cn/gp/product/B01M8L5Z3Y" try: kv={'user-agent':'Mozilla/5.0'} r=requests.grt(url,headers=kv) r.raise_for_status() r.encoding=r.apparent_encoding print(r.text[1000:2000]) except: print(“爬取失败”)