爬虫一些基本用法

一些学习笔记，整理一下

小结：

【1】re.findall  这个是当用import re 时候用的，而find_all是BeautifulSoup

【2】//打开网页用urlopen（右键网页，查看源代码就可看到）

        html = urlopen("https://morvanzhou.github.io/static/scraping/list.html").read().decode('utf-8')

        //用beautifulsoup“加工下”网页，具体是啥不想了解

         soup = BeautifulSoup(html, features='lxml')

         //这里也可以用select  语法 (“标签”)，（“标签”>"子标签"），（“.类名”），（“#id”），（"标签 #id"）
         img_links = soup.find_all("img", { "src": re.compile('.*?\.jpg')}) 

【3】scrapy 这个要上手感觉有些费劲，目前前两个感觉也够了

【4】关于正则表达式  import re

        . 表示任意符号      *表示数目   ？表示有没有都行  .*?匹配任意的东东     (.*?.jpg) 括号里面的是输出部分

 

from urllib.request import urlopen
# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/basic-structure.html").read().decode('utf-8')
print(html)


import re
res = re.findall(r"<title>(.+?)</title>", html)
print("\nPage title is: ", res[0])
# Page title is:  Scraping tutorial 1 | 莫烦Python


res = re.findall(r"<p>(.*?)</p>", html, flags=re.DOTALL)    # re.DOTALL if multi line
print("\nPage paragraph is: ", res[0])
# Page paragraph is:
#     这是一个在 <a href="https://morvanzhou.github.io/">莫烦Python</a>
#     <a href="https://morvanzhou.github.io/tutorials/scraping">爬虫教程</a> 中的简单测试.


res = re.findall(r'href="(.*?)"', html)
print("\nAll links: ", res)
# All links:  ['https://morvanzhou.github.io/static/img/description/tab_icon.png', 'https://morvanzhou.github.io/', 'https://morvanzhou.github.io/tutorials/scraping']

 

 

 

（2）

from bs4 import BeautifulSoup
from urllib.request import urlopen

# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/basic-structure.html").read().decode('utf-8')

soup = BeautifulSoup(html, features='lxml')
print(soup.h1)
print('\n', soup.p)

all_href = soup.find_all('a') // 通过标签查找

all_href = [l['href'] for l in all_href]
print('\n', all_href)

（3）

from bs4 import BeautifulSoup
from urllib.request import urlopen

# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/list.html").read().decode('utf-8')

soup = BeautifulSoup(html, features='lxml')
# 例如<div class="list-group has-bdb edit-btn-box">
month = soup.find_all('li', { "class": "month"})
for m in month:
    print(m.get_text())


jan = soup.find('ul', { "class": 'jan'})
d_jan = jan.find_all('li')              # use jan as a parent
for d in d_jan:
    print(d.get_text())

（4）

from bs4 import BeautifulSoup
from urllib.request import urlopen
import re

# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/table.html").read().decode('utf-8')

soup = BeautifulSoup(html, features='lxml')

img_links = soup.find_all("img", { "src": re.compile('.*?\.jpg')})
for link in img_links:
    print(link['src'])
<