Q: Method to Scrape a python list of multiple URLs I scraped wikipedia pages for the URLs I need and appended it to an empty list in python.
I now need to scrape every URL in my list for specific information, like date, coordinates etc.
Given the structure of the HTML code, parent/sub-parent structure, a lot of the information cannot be linked to by tag alone. Or can it? See fact box in the following link: https://en.wikipedia.org/wiki/1987_Maryland_train_collision.
I am targeting my scraping on these fact boxes because most of them includes one.
I understand that you can put in a conditional statement to claim specific data from a set of data, with same HTML tag. However, I am not sure how to approach it.
So far i have the below:
list_of_urls = #my list of urls to be scraped


for i in list_of_urls:
        
        soup = BeautifulSoup(text, features="lxml")
        
        for item in soup.findAll('td',attrs={'class':'infobox-label'}):
            
            if item.find('td', attrs={'class':'infobox-data'})  == "date":
                print(item.find)
    
                date_info = item.get("infobox-data")
                print(date_info)

                #do something more..    

Any thoughts on the above?
Thank you for your time.

EDIT: Solved by applying Rusticus methods.. 


A: The structure of what you're examining looks like this:


<tr>
  <th scope="row" class="infobox-label" style="white-space:nowrap;padding-right:0.65em;">Date</th>
  <td class="infobox-data" style="line-height:1.3em;">January 4, 1987 <br>1:30 PM</td>
</tr>




*

*Note that the "infobox-label" is in a TH tag not a TD tag.

*item.find is a method, you probably intended "print(item)"

*Once you've found the TH tag, you will want to move to the TD tag to get the value.  There are several ways to do this, I think the simplest is to reference "item.parent.td"

Maybe you're looking for something like this:
    for item in soup.findAll('th',attrs={'class':'infobox-label'}):
        
        if item.text  == "Date":
            print(item)

            date_info = item.parent.td.text
            print(date_info)

Alternatively just:
soup.select_one('.infobox').find('th', text="Date").parent.td.text.strip()

For Coordinates:
soup.select_one('.infobox').find('th', text="Coordinates").parent.td.select_one('.geo-dec').text.strip()

