python爬虫

requests

常用命令：
requests.get()
requests.post()

图片爬取

from io import BytesIO,StringIO
import requests
from PIL import Image
from pyquery import PyQuery as pq
import json

kw = {'q': 'png图片'}
response = requests.get("https://cn.bing.com/images/search?", params = kw);

print(response.url)
doc = pq(response.content.decode('utf-8'))

jsondata = json.loads(doc('a.iusc:first-child').attr('m'))

imgurl = jsondata['murl']
resimg = requests.get(imgurl)
f = BytesIO(resimg.content)  # 二进制写入内存（区别StringIO）
img = Image.open(f)
print(img.size)

post请求

import requests

formdata = {
    "q":"auto|||auto|||hello world"
}

url = "http://www.stl56.com/fanyi/?a=result"

headers={ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36 Edg/99.0.1150.36",
		  "Content-Type": "application/x-www-form-urlencoded"}

response = requests.post(url, data = formdata, headers = headers)

print (response.text)

JS动态获取解决方案

• 直接从 JavaScript 代码里采集内容（费时费力）， 用python的方式重新写出来
• 用 Python 的 第三方库运行 JavaScript，直接采集你在浏览器里看到的页面（这个可以有）。

Selenium + PhantomJS

https://selenium-python.readthedocs.io/index.html

https://phantomjs.org/documentation/

解析方案

• xpath
• bs4
• pyquery

问题方案

页面自动跳转解决：禁止重定向

requests.get("https://", params = kw, headers = headers, allow_redirects = False)

抓包工具

Wireshark：http://www.wireshark.org

资料

pyquery使用：https://zhuanlan.zhihu.com/p/393764169

XPath：https://www.jianshu.com/p/883c74f939ee

web scraper

有道翻译破解：https://blog.csdn.net/qq_36653505/article/details/80378869

http://httpbin.org/

selenium教程：https://blog.csdn.net/weixin_43157068/article/details/104331842

https://blog.csdn.net/WuLex/article/details/108582447

https://www.jianshu.com/p/6c82c965c014