BeautifulSoup即BS4的使用

BS4的作用和使用场景

BeautifulSoup模块的安装

pip install beautifulsoup4

同时我们要安装lxml

pip install lxml

如果不安装lxml，则BeautifulSoup会使用Python内置的解析器对文档进行解析。之所以使用lxml，是因为解析速度快
BeautifulSoup可以理解成Document对象，提供很方便的API来获取Tag，以及搜索相关的标签内容。

BS4的应用实例

使用Jupyter Notebook，可以通过以下代码提取知乎上的问题和回复内容

def scroll(driver):
    driver.execute_script(
        """
        
            var y = document.body.scrollTop;
            var step = 100;
            window.scroll(0,y);
            function f(){
                if(y<document.body.scrollHeight){
                    y += step;
                    window.scroll(0,y);
                    //alert("current height is " + y + " and maxHeight is " + document.body.scrollHeight)
                    setTimeout(f, 1000);
                    
                }
                else{
                    window.scroll(0, y);
                    document.title += "scroll-done";
                }
            }
            setTimeout(f, 1000);
        """
    )

def func():
    for story in storys:
        nameLabel = story.find('meta', itemprop="name")
        name = nameLabel["content"]
        storyText = story.find('div', class_="RichContent-inner")
        #拼接字符串并写入文本
        try:
            string=name+'-------------'+storyText.text+'\n'
            f.write(string)
            print('answer By' + str(name)+' has been finished')
        except Exception:
            print('Something is wrong while writing to txt')

from bs4 import BeautifulSoup
from selenium import webdriver
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome()
driver.get('https://www.zhihu.com/question/54732849')
time.sleep(10)
actions = ActionChains(driver)
actions.key_down(Keys.ESCAPE).key_up(Keys.ESCAPE).perform()
time.sleep(2)
scroll(driver)
time.sleep(60)
html=driver.page_source
soup=BeautifulSoup(html,'lxml')
storys=soup.find_all('div',class_="List-item")
time.sleep(5)
f=open('tmp.txt','w+')
func()
f.close()
driver.quit()

BS4的API文档

对于BeautifulSoup的其他用法，可以参考API文档，访问地址https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/