beautiful-soup是用来解析网页的利器。
我们先打开一个网页
code=200表示访问成功
from urllib.request import urlopen
url='http://www.xzw.com/astro/virgo/?appid=bds'
response=urlopen(url)
response.getcode()
content=response.read()
接下来,我们可以用beautiful-soup来解析这个网页,生成一个beautiful-soup对象
bs_obj=BeautifulSoup(urlopen(url), 'html.parser',from_encoding='utf-8')
bs_obj.title
<title>处女座2018年运势,处女座男生女生,处女座男生女生最配的星座,处女座的男人,Virgo-星座是按阴历还是阳历,十二星座速配查询 - 星座屋</title>
然后可以用bs对象的find方法来查找节点,我们通过传入节点的标签和属性作为参数,返回节点的list
bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'})
[<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>,
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>,
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>,
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>,
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>,
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>,
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>,
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>,
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>,
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>,
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>,
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>]
对于每个节点,我们可以获取它的名称,属性,文字
Node.name | 节点的名称 |
Node['href'] | 节点的属性 |
Node.get_text() | 节点的文字 |
node_list = bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'})
for node in node_list:
print(node.name,node['href'],node['title'],node.get_text())
a /astro/aries/ 白羊座
a /astro/taurus/ 金牛座
a /astro/gemini/ 双子座
a /astro/cancer/ 巨蟹座
a /astro/leo/ 狮子座
a /astro/virgo/ 处女座
a /astro/libra/ 天秤座
a /astro/scorpio/ 天蝎座
a /astro/sagittarius/ 射手座
a /astro/capricorn/ 摩羯座
a /astro/aquarius/ 水瓶座
a /astro/pisces/ 双鱼座
Children 节点
如果我们想拿到div标签后面的所有a标签,可以用node.children
for child in bs_obj.find('div', {'class':'a-nav'}).children:
print(child)
<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>
<em class="todayastro"></em>
更多的还有find_parent, find_next_sibling等,用法都差不多,可以查看文档