I'm trying to scrape a real estate website for listings. It has an aspx form that has to filled out before submission.
All I care about is multifamily properties in Oregon, however. So this was my first attempt:
driver = webdriver.Firefox()
driver.get("http://www.cbre.us/PropertyListings/Pages/Properties-for-Sale.aspx")
#Searching for multifamily residences
selectPropertyType = driver.find_element_by_id("ForSalePropertyType")
selectPropertyType.select_by_value("70")
#In the state of Oregon
selectState = driver.find_element_by_id("ForSaleState_ListBox1")
selectState.select_by_value("OR")
#Submit form
submitBtn = driver.find_element_by_id("ForSaleLooplinkSubmit")
submitBtn.click()
#Wait for results to load
WebDriverWait(driver, 5)
When I run this script it gives an error "Can't locate element "ForSalePropertyType". What am I doing wrong here? Thanks in advance.
解决方案
This element located inside an iframe. You have to switch to it's context:
driver.switch_to.frame("ctl00_PlaceHolderMain_IFrameContent_IFrameContent")
# searching for multifamily residences
selectPropertyType = driver.find_element_by_id("ForSalePropertyType")
selectPropertyType.select_by_value("70")
To get back to default context:
driver.switch_to.default_content()
As a side note, be aware of the policies listed in the Disclaimer / Terms of use, specifically:
You agree that you will not: (a) impersonate any person or entity or
misrepresent your affiliation with any other person or entity; (b)
engage in spamming, flooding, harvesting of e-mail addresses or other
personal information, spidering, screen scraping, database scraping,
or any other activity with the purpose of obtaining lists of users or
any other information, including specifically, property listings
available through the site; (c) send chain letters or pyramid schemes
via the site; or (d) attempt to gain unauthorized access to other
computer systems through the site. You agree that you will not use the
site in any manner that could damage, disable, overburden, or impair
the site or interfere with any other party's use and enjoyment of the
site.