python - Why doesn't Selenium find elements? - Stack Overflow

admin管理员组

文章数量:1312971

I want to scrape the address of the houses from the website, but the code returns no output, can someone tell me where the problem is from?

here is my code:

from selenium import webdriver
from selenium.webdrivermon.by import By

url= "/?new_homes=include&q=england+&orig_q=united+kingdom&view_type=list&pn=1"
driver=webdriver.Chrome()
driver.get(url)
# Find elements using By.CLASS_NAME
houses = driver.find_elements(By.CLASS_NAME, "w6xkpv1 w6xkpv4")

for house in houses:
    # Find address within each house element
    address = house.find_element(By.XPATH, './/*[@id="main-content"]/div[2]/div/div/section/div[1]/div[2]/div[1]/div/div/div/div[1]/a/h2').text
    print(address)

I want to scrape the address of the houses from the website, but the code returns no output, can someone tell me where the problem is from?

here is my code:

from selenium import webdriver
from selenium.webdrivermon.by import By

url= "https://www.zoopla.co.uk/house-prices/england/?new_homes=include&q=england+&orig_q=united+kingdom&view_type=list&pn=1"
driver=webdriver.Chrome()
driver.get(url)
# Find elements using By.CLASS_NAME
houses = driver.find_elements(By.CLASS_NAME, "w6xkpv1 w6xkpv4")

for house in houses:
    # Find address within each house element
    address = house.find_element(By.XPATH, './/*[@id="main-content"]/div[2]/div/div/section/div[1]/div[2]/div[1]/div/div/div/div[1]/a/h2').text
    print(address)

• python
• selenium-webdriver
• web-scraping

Share Improve this question Follow edited Jan 31 at 21:47 Robert 8,6935454 gold badges129129 silver badges170170 bronze badges asked Jan 31 at 21:03 Chioma OkoroaforChioma Okoroafor 5711 silver badge77 bronze badges 7

• 1 When you debug, which operation isn't finding the HTML element(s) it's looking for? When you observe that HTML during debugging, does it contain the target element(s)? Can you include that HTML in the question? (Can be edited to only show the relevant structure and not the entire DOM, of course.) – David Commented Jan 31 at 21:05
• the houses = driver.find_elements(By.CLASS_NAME, "w6xkpv1 w6xkpv4") – Chioma Okoroafor Commented Jan 31 at 21:08
• 1 Do you know if zoopla allows web scraping? They may be actively blocking you. In that case check if they offer alternative ways to query their data (e.g., an API). – Robert Commented Jan 31 at 21:09
• What do you get when you add print(houses)? – Robert Commented Jan 31 at 21:13
• 2 The obvious answer is that driver.find_elements(By.CLASS_NAME, "w6xkpv1 w6xkpv4") found no matching elements, therefore it returned an empty list, and so the next loop for house in houses was empty and did not run at all. – John Gordon Commented Jan 31 at 22:45

 |  Show 2 more comments

2 Answers 2

Sorted by: Reset to default 1

You could take a completely different approach using CSS selectors as follows:

from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdrivermon.by import By

url = "https://www.zoopla.co.uk/house-prices/england/?new_homes=include&q=england+&orig_q=united+kingdom&view_type=list&pn=1"

# handle elements that may not be currently visible
def etext(e):
    if e:
        if t := e.text.strip():
            return t
        if (p := e.get_property("textContent")) and isinstance(p, str):
            return p.strip()
    return ""

with Chrome() as driver:
    driver.get(url)
    wait = WebDriverWait(driver, 10)
    ec = EC.presence_of_all_elements_located
    sel = By.CSS_SELECTOR, "div[data-testid=result-item] h2"
    for h2 in wait.until(ec(sel)):
        print(etext(h2))

All the selectors are wrong. You can't just copy the content of the class attribute and expect it to work.

Class names can't contain spaces.
In HTML class attribute, space allows to set multiple class names for the element.
In CSS, class names separated by space means the second element is inside of the first one.

Also, XPATH is broken.

Use this instead:

from selenium import webdriver
from selenium.webdrivermon.by import By

url= "https://www.zoopla.co.uk/house-prices/england/?new_homes=include&q=england+&orig_q=united+kingdom&view_type=list&pn=1"
driver=webdriver.Chrome()
driver.get(url)

houses = driver.find_elements(By.CSS_SELECTOR, ".w6xkpv1.w6xkpv4 h2")

print(houses)

for address in houses:
    print(address.text)

本文标签： pythonWhy doesn39t Selenium find elementsStack Overflow