个人学习爬虫笔记之urllib模块的使用三

URL解析：

urlparse

就是将网址按规律分段；

  1 from urllib.parse import urlparse
  2 
  3 result = urlparse('http://www.baidu.com/index.html;user?id=5#comment')
  4 print(type(result), result)

结果是：

 <class 'urllib.parse.ParseResult'> ParseResult(scheme='http', netloc='www.baidu.com', path='/index.html', params='user', query='id=5', fragment='comment')

---

url可以分成六个字段； 

里边两个网址不同

  1 from urllib.parse import urlparse
  2 
  3 result = urlparse('http://www.baidu.com/index.html;user?id=5#comment', scheme='https')
  4 print( result)

  1 from urllib.parse import urlparse
  2 
  3 result = urlparse('www.baidu.com/index.html;user?id=5#comment', scheme='https')
  4 print( result)

向前拼接：将fragments所带的字段向前拼接

  1 from urllib.parse import urlparse
  2 
  3 result = urlparse('http://baidu.com/index.html;user?id=5#comment', allow_fragments=False)
  4 print( result)

  1 from urllib.parse import urlparse
  2 
  3 result = urlparse('http://baidu.com/index.html#comment', allow_fragments=False)
  4 print( result)

拼接链接：就是将各个字段拼接成一个链接；

  1 from urllib.parse import urlunparse
  2 
  3 data = ['http', 'www.baidu.com', 'index.html', 'user', 'a=6', 'comment']
  4 result = urlunparse(data)
  5 print(result)

拼接url:

urljoin方法；

  1 from urllib.parse import urljoin
  2 
  3 print(urljoin('http://www.baidu.com', 'FAQ.html'))
  4 print(urljoin('http://www.baidu.com', 'https://cuiqingcai.com/FAQ.html'))
  5 print(urljoin('http://www.baidu.com/about.html', 'https://cuiqingcai.com/FAQ.html'))
  6 print(urljoin('http://www.baidu.com/about.html', 'https://cuiqingcai.com/FAQ.html?question=2'))
  7 print(urljoin('http://www.baidu.com?wd=abc', 'https://cuiqingcai.com/index.php'))
  8 print(urljoin('http://www.baidu.com', '?category=2#comment'))
  9 print(urljoin('http://www.baidu.com#comment', '?category=2#comment'))

结果：

http://www.baidu.com/FAQ.html
https://cuiqingcai.com/FAQ.html
https://cuiqingcai.com/FAQ.html
https://cuiqingcai.com/FAQ.html?question=2
https://cuiqingcai.com/index.php
http://www.baidu.com?category=2#comment
http://www.baidu.com?category=2#comment

---

把列表里的内容加到链接里：

  1 from urllib.parse import urlencode
  2 
  3 params = {'name':'germey', 'age':'22'}
  4 base_url = 'http://www.baidu.com'
  5 url = base_url + urlencode(params)
  6 print(url)

结果是：

http://www.baidu.comname=germey&age=22