python Beautiful soup网页解析-星座网

beautiful-soup是用来解析网页的利器。

我们先打开一个网页

code=200表示访问成功

from urllib.request import urlopen
url='http://www.xzw.com/astro/virgo/?appid=bds'
response=urlopen(url)
response.getcode()
content=response.read()

接下来，我们可以用beautiful-soup来解析这个网页，生成一个beautiful-soup对象

bs_obj=BeautifulSoup(urlopen(url), 'html.parser',from_encoding='utf-8')

bs_obj.title

<title>处女座2018年运势,处女座男生女生,处女座男生女生最配的星座,处女座的男人,Virgo-星座是按阴历还是阳历,十二星座速配查询 - 星座屋</title>

然后可以用bs对象的find方法来查找节点，我们通过传入节点的标签和属性作为参数，返回节点的list

bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'})

[<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>,
 <a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>,
 <a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>,
 <a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>,
 <a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>,
 <a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>,
 <a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>,
 <a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>,
 <a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>,
 <a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>,
 <a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>,
 <a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>]

对于每个节点，我们可以获取它的名称，属性，文字

Node.name	节点的名称
Node['href']	节点的属性
Node.get_text()	节点的文字

node_list = bs_obj.find_all('a',{'target':'_blank','hidefocus':'true'})
for node in node_list:
    print(node.name,node['href'],node['title'],node.get_text())

a /astro/aries/ 白羊座
a /astro/taurus/ 金牛座
a /astro/gemini/ 双子座
a /astro/cancer/ 巨蟹座
a /astro/leo/ 狮子座
a /astro/virgo/ 处女座
a /astro/libra/ 天秤座
a /astro/scorpio/ 天蝎座
a /astro/sagittarius/ 射手座
a /astro/capricorn/ 摩羯座
a /astro/aquarius/ 水瓶座
a /astro/pisces/ 双鱼座

Children 节点

如果我们想拿到div标签后面的所有a标签，可以用node.children

for child in bs_obj.find('div', {'class':'a-nav'}).children:
    print(child)

<a class="icon0" hidefocus="true" href="/astro/aries/" target="_blank" title="白羊座"></a>
<a class="icon1" hidefocus="true" href="/astro/taurus/" target="_blank" title="金牛座"></a>
<a class="icon2" hidefocus="true" href="/astro/gemini/" target="_blank" title="双子座"></a>
<a class="icon3" hidefocus="true" href="/astro/cancer/" target="_blank" title="巨蟹座"></a>
<a class="icon4" hidefocus="true" href="/astro/leo/" target="_blank" title="狮子座"></a>
<a class="icon5" hidefocus="true" href="/astro/virgo/" target="_blank" title="处女座"></a>
<a class="icon6" hidefocus="true" href="/astro/libra/" target="_blank" title="天秤座"></a>
<a class="icon7" hidefocus="true" href="/astro/scorpio/" target="_blank" title="天蝎座"></a>
<a class="icon8" hidefocus="true" href="/astro/sagittarius/" target="_blank" title="射手座"></a>
<a class="icon9" hidefocus="true" href="/astro/capricorn/" target="_blank" title="摩羯座"></a>
<a class="icon10" hidefocus="true" href="/astro/aquarius/" target="_blank" title="水瓶座"></a>
<a class="icon11" hidefocus="true" href="/astro/pisces/" target="_blank" title="双鱼座"></a>
<em class="todayastro"></em>

更多的还有find_parent, find_next_sibling等，用法都差不多，可以查看文档