python中bs4.element.tag,美丽的汤错误：'＆LT;类的bs4.element.Tag'＆GT;'对象有没有属性'的内容“？...

I'm writing a script that extracts the content out of an article and removes any unnecessary stuff eg. scripts and styling. Beautiful Soup keeps raising the following exception:

'' object has no attribute 'contents'

Here's the code of the trim function (element is the HTML element that contains the content of the webpage):

def trim(element):

elements_to_remove = ('script', 'style', 'link', 'form', 'object', 'iframe')

for i in elements_to_remove:

remove_all_elements(element, i)

attributes_to_remove = ('class', 'id', 'style')

for i in attributes_to_remove:

remove_all_attributes(element, i)

remove_all_comments(element)

# Remove divs that have more non-p elements than p elements

for div in element.find_all('div'):

p = len(div.find_all('p'))

img = len(div.find_all('img'))

li = len(div.find_all('li'))

a = len(div.find_all('a'))

if p == 0 or img > p or li > p or a > p:

div.decompose()

Looking at the stack trace, the problem seems to be coming from this method right after the for statement:

# Remove divs that have more non-p elements than p elements

for div in element.find_all('div'):

p = len(div.find_all('p')) #

I don't get why this instance of bs4.element.Tag doesn't have the attribute 'contents'? I tried it out on an actual webpage and the element was full of p's and img's...

Here's the traceback (This is part of a Django project I'm working on):

Environment:

Request Method: POST

Request URL: http://localhost:8000/read/add/

Django Version: 1.4.1

Python Version: 2.7.3

Installed Applications:

('django.contrib.auth',

'django.contrib.contenttypes',

'django.contrib.sessions',

'django.contrib.sites',

'django.contrib.messages',

'django.contrib.staticfiles',

'home',

'account',

'read',

'review')

Installed Middleware:

('django.middleware.common.CommonMiddleware',

'django.contrib.sessions.middleware.SessionMiddleware',

'django.middleware.csrf.CsrfViewMiddleware',

'django.contrib.auth.middleware.AuthenticationMiddleware',

'django.contrib.messages.middleware.MessageMiddleware')

Traceback:

File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response

111. response = callback(request, *callback_args, **callback_kwargs)

File "/home/marco/sandra/read/views.py" in add

24. Article.objects.create_article(request.user, url)

File "/home/marco/sandra/read/models.py" in create_article

11. title, content = logic.process_html(web_page.read())

File "/home/marco/sandra/read/logic.py" in process_html

7. soup = htmlbarber.give_haircut(BeautifulSoup(html_code, 'html5lib'))

File "/home/marco/sandra/read/htmlbarber/__init__.py" in give_haircut

45. scissor.trim(element)

File "/home/marco/sandra/read/htmlbarber/scissor.py" in trim

35. p = len(div.find_all('p'))

File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in find_all

1128. return self._find_all(name, attrs, text, limit, generator, **kwargs)

File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in _find_all

413. return [element for element in generator

File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in descendants

1140. if not len(self.contents):

File "/home/marco/.virtualenvs/sandra/local/lib/python2.7/site-packages/bs4/element.py" in __getattr__

924. "'%s' object has no attribute '%s'" % (self.__class__, tag))

Exception Type: AttributeError at /read/add/

Exception Value: '' object has no attribute 'contents'

Here's the source code of remove_all_* functions:

def remove_all_elements(element_to_clean, unwanted_element_name):

for to_remove in element_to_clean.find_all(unwanted_element_name):

to_remove.decompose()

def remove_all_attributes(element_to_clean, unwanted_attribute_name):

for to_inspect in [element_to_clean] + element_to_clean.find_all():

try:

del to_inspect[unwanted_attribute_name]

except KeyError:

pass

def remove_all_comments(element_to_clean):

for comment in element_to_clean.find_all(text=lambda text:isinstance(text, Comment)):

comment.extract()

解决方案

I think the problem is that in remove_all_elements or somewhere else in your code you are deleting the contents attribute of some of your tags.

It looks like this is happening when you call to_remove.decompose(). Here is the source for that method:

def decompose(self):

"""Recursively destroys the contents of this tree."""

self.extract()

i = self

while i is not None:

next = i.next_element

i.__dict__.clear()

i = next

Here is what happens if you call this function manually:

>> soup = BeautifulSoup('

hi

')

>>> d0 = soup.find_all('div')[0]

>>> d0

hi

>>> d0.decompose()

>>> d0

Traceback (most recent call last):

...

Traceback (most recent call last):

AttributeError: '' object has no attribute 'contents'

It appears that once you have called decompose on a tag you must never attempt to use that tag again. I'm not quite sure where this is happening though.

One thing I would try checking is that len(element.__dict__) > 0 at all times in your trim() function.