Python 爬虫学习笔记1

Python 爬虫学习笔记1

part1 获取数据

模拟浏览器发起请求的几种方法（以百度为例）

1.库 urllib

1.1 urlopen

import urllib.request

url = "http://www.baidu.com"

response = urllib.request.urlopen(url)
print(response.read().decode('utf-8'))

1.2 Request

import urllib.request

url = "http://www.baidu.com"

request = urllib.request.Request(url)
response = urllib.request.urlopen(request)
print(response.read().decode('utf-8'))

2.库requests

2.1 requests

import requests

url = "http://www.baidu.com"

res = requests.get(url)
print(res.headers, res.url, res.text)

注：
requests.post()、requests.put()、requests.delete()等实现POST、PUT、DELETE等请求

添加参数

import requests

url = "http://www.baidu.com"

data = {
    'name': 'xiaohua',
    'age': '18'
}
res = requests.get(url, params=data)
print(res.headers, res.url, res.text)