爬虫第一天，urllib库中的URLopen

爬虫

基本流程

1、发起请求

requests
GET or import

response

①浏览器发送消息给该网址所在服务器HTTP Requests
②服务器根据内容，做出相应处理，把消息回传给浏览器HTTP Response
③浏览器收到Response信息后，进行解析

2、解析内容
3、获取相应内容
4、保存数据

Urllib库

请求模块
异常处理
解析模块

启动jupyter notebook
打开cmd cd
步骤①

URLopen

代码

urllib.request.urlopen(url,data+None,[timeout,]*,cafile=None,capath=None,cadefault=False,context=None)

import urllib.request
response = urllib.request.urlopen('http://www.baidu.com')
print(response.read().decode('utf-8'))

import urllib.parse
import urllib.request

data = bytes(urllib.parse.urlencode({'word':'hello'}),encoding='utf8')
response = urllib.request.urlopen('http://httpbin.org/post',data=data)
print(response.read())

import urllib.request

response = urllib.request.urlopen('http://httpbin.org/get',timeout=1)
print(response.read())

import socket
import urllib.request
import urllib.error

try:
    response = urllib.request.urlopen('http://httpbin.org/get',timeout=0.1)
except urllib.error.URLError as e:
    if isinstance(e.reason,socket.timeout):
        print('TIME OUT')