一些学习笔记,整理一下
小结:
【1】re.findall 这个是当用import re 时候用的,而find_all是BeautifulSoup
【2】//打开网页用urlopen(右键网页,查看源代码就可看到)
html = urlopen("https://morvanzhou.github.io/static/scraping/list.html").read().decode('utf-8')
//用beautifulsoup“加工下”网页,具体是啥不想了解
soup = BeautifulSoup(html, features='lxml')
//这里也可以用select 语法 (“标签”),(“标签”>"子标签"),(“.类名”),(“#id”),("标签 #id")
img_links = soup.find_all("img", {
"src": re.compile('.*?\.jpg')})
【3】scrapy 这个要上手感觉有些费劲,目前前两个感觉也够了
【4】关于正则表达式 import re
. 表示任意符号 *表示数目 ?表示有没有都行 .*?匹配任意的东东 (.*?.jpg) 括号里面的是输出部分
from urllib.request import urlopen
# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/basic-structure.html").read().decode('utf-8')
print(html)
import re
res = re.findall(r"<title>(.+?)</title>", html)
print("\nPage title is: ", res[0])
# Page title is: Scraping tutorial 1 | 莫烦Python
res = re.findall(r"<p>(.*?)</p>", html, flags=re.DOTALL) # re.DOTALL if multi line
print("\nPage paragraph is: ", res[0])
# Page paragraph is:
# 这是一个在 <a href="https://morvanzhou.github.io/">莫烦Python</a>
# <a href="https://morvanzhou.github.io/tutorials/scraping">爬虫教程</a> 中的简单测试.
res = re.findall(r'href="(.*?)"', html)
print("\nAll links: ", res)
# All links: ['https://morvanzhou.github.io/static/img/description/tab_icon.png', 'https://morvanzhou.github.io/', 'https://morvanzhou.github.io/tutorials/scraping']
(2)
from bs4 import BeautifulSoup
from urllib.request import urlopen
# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/basic-structure.html").read().decode('utf-8')
soup = BeautifulSoup(html, features='lxml')
print(soup.h1)
print('\n', soup.p)
all_href = soup.find_all('a') // 通过标签查找
all_href = [l['href'] for l in all_href]
print('\n', all_href)
(3)
from bs4 import BeautifulSoup
from urllib.request import urlopen
# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/list.html").read().decode('utf-8')
soup = BeautifulSoup(html, features='lxml')
# 例如<div class="list-group has-bdb edit-btn-box">
month = soup.find_all('li', {
"class": "month"})
for m in month:
print(m.get_text())
jan = soup.find('ul', {
"class": 'jan'})
d_jan = jan.find_all('li') # use jan as a parent
for d in d_jan:
print(d.get_text())
(4)
from bs4 import BeautifulSoup
from urllib.request import urlopen
import re
# if has Chinese, apply decode()
html = urlopen("https://morvanzhou.github.io/static/scraping/table.html").read().decode('utf-8')
soup = BeautifulSoup(html, features='lxml')
img_links = soup.find_all("img", {
"src": re.compile('.*?\.jpg')})
for link in img_links:
print(link['src'])
<