Web scraping with Selenium: Why your code works but returns an empty list still.
In this article, I’ll show how I scraped the restaurant, type, rating, popular items and price details off a page on Doordash, San Diego’s website. while maintaining the data integrity such that Restaurant A did not end up with Restaurant B’s menu or Restaurant D’s prices. This article states and solves some reasons why your list keeps returning empty even when your code is right. It could be because:
- The Menu and Prices are hidden until a dropdown button is clicked. The algorithm needs to click on this button so the details can be collected and parsed by Beautiful soup.
- Unlike other parameters like restaurant that had one value per tag for one store, Menu and prices had between two and three values each in one tag for one store.
- The multiple values per tag of Menu and Prices creates nested lists making it impossible to use get_text() function to get the values.
Selenium is that master key you want to have when it comes to scraping website. Since its process is human-like, it can bye-pass most firewalls meant to prevent libraries like Beautiful soup. The thing about websites is they are like human faces, they have unique features of their own, so when you are using Selenium to scrape, you really need to take the html/element inspection aspect serious, else you’ll be plunged down a rabbit hole and become madder than the mad hatter himself.
I handled the Menu and price value by:
- storing them in separate lists, without calling the get_text() function
- I then transformed each list into separate data frame
- A function was applied to all columns in each data frame to get the text values with an if condition to bypass None values.
- The columns in each data frame are combined to a single column
- Both processed data frames are merged with data frame containing the rest parameters through index.
Below is the full code.
Lobatan! (it translates to ‘All done!” in Yoruba)
After converting the final df to csv file, Below is a screenshot of my csv file, you can see the Restaurant, type, location, rating, Menu and corresponding prices.
