python html 鼠标点击,使用Python中的beautifulsoup在html页面上调用“on-click事件”的问题...

我试图刮取网页上出现的所有项目的名称，但默认情况下只有18个项目在页面上可见&我的代码只抓取这些项目。您可以通过单击“ShowAll”按钮查看所有项目，但该按钮使用的是Javascript。在

经过一些研究，我发现PyQt模块可以用来解决这个涉及javascript按钮的问题&我使用了它，但仍然无法调用“on click”事件。参考代码如下：import csv

import urllib2

import sys

import time

from bs4 import BeautifulSoup

from PyQt4.QtGui import *

from PyQt4.QtCore import *

from PyQt4.QtWebKit import *

class Render(QWebPage):

def __init__(self, url):

self.app = QApplication(sys.argv)

QWebPage.__init__(self)

self.loadFinished.connect(self._loadFinished)

self.mainFrame().load(QUrl(url))

self.app.exec_()

def _loadFinished(self, result):

self.frame = self.mainFrame()

self.app.quit()

url = 'http://www.att.com/shop/wireless/devices/smartphones.html'

r = Render(url)

jsClick = var evObj = document.createEvent('MouseEvents');

evObj.initEvent('click', true, true );

this.dispatchEvent(evObj);

allSelector = "a#deviceShowAllLink" # This is the css selector you actually need

allButton = r.frame.documentElement().findFirst(allSelector)

allButton.evaluateJavaScript(jsClick)

page = allButton

soup = BeautifulSoup(page)

soup.prettify()

with open('Smartphones_26decv1.0.csv', 'wb') as csvfile:

spamwriter = csv.writer(csvfile, delimiter=',')

spamwriter.writerow(["Date","Day of Week","Device Name","Price"])

items = soup.findAll('a', {"class": "clickStreamSingleItem"},text=True)

prices = soup.findAll('div', {"class": "listGrid-price"})

for item, price in zip(items, prices):

textcontent = u' '.join(price.stripped_strings)

if textcontent:

spamwriter.writerow([time.strftime("%Y-%m-%d"),time.strftime("%A") ,unicode(item.string).encode('utf8').strip(),textcontent])

我在这方面面临的错误如下：

^{pr2}$

有人能帮我调用这个“onclick”事件，这样我就可以为所有人收集数据了吗物品。什么我的无知，因为我是编程新手。在