python get text,Python Beautifulsoup get_text（）没有获取所有文本

I'm trying to get all text from a html tag using beautifulsoup get_text() method. I use Python 2.7 and Beautifulsoup 4.4.0. It works for most of the times. However, this method can only get first paragraph from a tag sometimes. I can't figure out why. Please see the following example.

from bs4 import BeautifulSoup

import urllib2

job_url = "http://www.indeed.com/viewjob?jk=0f5592c8191a21af"

site = urllib2.urlopen(job_url).read()

soup = BeautifulSoup(site, "html.parser")

text = soup.find("span", {"class": "summary"}).get_text()

print text

I want to get all content from this indeed job description. Basically, I want to get all text in . However, utilize the code above, I can only get "Please note that this is a 1 year contract assignment. Candidates cannot start an assignment until background check and drug test is completed". Why I'm losing the rest of text? How can I get all text from this tag without specifying sub-tags?

Thanks a lot.

解决方案

Try it with a different parser like the lxml parser instead of the html.parser parser:

Replace:

soup = BeautifulSoup(site, "html.parser")

with:

soup = BeautifulSoup(site, "lxml")