<p>I had to execute a JavaScript code that opens the pdf link in a new window/tab. After that, just closed this newly opened window/tab and continued with the script execution. Here is the updated code:</p>
<pre><code>import os
import time
import csv

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

ncbi_url = &quot;https://www.ncbi.nlm.nih.gov/&quot;

download_dir = &quot;pdf_downloaded&quot;

if not os.path.exists(download_dir):
    os.mkdir(download_dir)
    
options = Options()
options.set_preference('permissions.default.stylesheet', 2)
options.set_preference('permissions.default.image', 2)
options.set_preference('browser.download.folderList', 2)
options.set_preference('browser.download.dir', os.path.abspath(download_dir))
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
options.set_preference('pdfjs.disabled', True)

###############################################################################
def download_pmc_pdf(pmcid_key):
    browser = webdriver.Firefox(options=options)
    pmc_url = ncbi_url + &quot;pmc/articles/&quot; + pmcid_key + &quot;/&quot;
    browser.get(pmc_url)
    try:
        wait = WebDriverWait(browser, 10)
        pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))

        # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
        for link in pdf_links:
            href = link.get_attribute('href')
            if '.pdf' in href:
                # Open the pdf link in a new window
                browser.execute_script(f'window.open(&quot;{href}&quot;,&quot;_blank&quot;);')
                break

        time.sleep(15)  # delay for 10 seconds
        # After the file is downloaded:
        with open(f'{download_dir}/file_mapping.csv', 'a', newline='') as file:
            writer = csv.writer(file)
            writer.writerow([pmcid_key, os.path.basename(href)])
        ## Switch to the new window and close it 
        ## (this is a hack to deal with bowser freezing upon downloading)
        browser.switch_to.window(browser.window_handles[-1])
        browser.close()

    except Exception as e:
        print(e)
    finally:
        browser.quit()  # quit the browser outside the loop    
###############################################################################
pmcid = &quot;2788555&quot;   
download_pmc_pdf(pmcid)
</code></pre>

CC BY-SA 4.0
2023-07-14T15:57:30.880Z
2023-07-14T16:02:37.583Z
2023-07-14T16:02:37.583Z
Wasim Aftab
<p>Pubmed Central stores pdfs for all available open access articles on a public-facing ftp server (<a href="https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/" rel="nofollow noreferrer">https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/</a>). You can query the database using the OA Web Service API, e.g. with cURL</p>
<pre><code># for pubmed id &quot;10026098&quot;
curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098
&lt;OA&gt;&lt;responseDate&gt;2023-07-17 01:39:13&lt;/responseDate&gt;&lt;request id=&quot;PMC10026098&quot;&gt;https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098&lt;/request&gt;&lt;records returned-count=&quot;2&quot; total-count=&quot;2&quot;&gt;&lt;record id=&quot;PMC10026098&quot; citation=&quot;Ther Adv Med Oncol. 2023 Mar 18; 15:17588359231161409&quot; license=&quot;CC BY-NC&quot; retracted=&quot;no&quot;&gt;&lt;link format=&quot;tgz&quot; updated=&quot;2023-03-23 12:10:30&quot; href=&quot;ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/83/eb/PMC10026098.tar.gz&quot; /&gt;&lt;link format=&quot;pdf&quot; updated=&quot;2023-03-20 09:34:33&quot; href=&quot;ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf&quot; /&gt;&lt;/record&gt;&lt;/records&gt;&lt;/OA&gt;

# get the pdf link from the API (ftp storage)
curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\&quot;\(ftp.*.pdf\)\&quot;.*/\1/p'
ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf

# download the pdf
pdf=&quot;<span class="math-container">$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
curl -O "$</span>f&quot;

# download many pdfs from a list of ids
cat list_of_ids.txt
10026098
9341183

while read -r id_from_list
do
    pdf=&quot;<span class="math-container">$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$</span>id_from_list | sed -n 's/.*\&quot;\(ftp.*.pdf\)\&quot;.*/\1/p')&quot;
    curl -O &quot;$pdf&quot;
done &lt; list_of_ids.txt
</code></pre>
<p>This approach is preferable to selenium scraping for many reasons, and if you have any problems the NCBI provides documentation (<a href="https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart</a>) and offers support (you can email them at pubmedcentral@ncbi.nlm.nih.gov).</p>

CC BY-SA 4.0
2023-07-17T13:05:17.273Z
Thank you for your time! Unfortunately, it did not work, returning the error `curl: (23) Failed writing received data to disk/application`
Furthermore, I've noticed a potential issue. What if the article listed on PubMed is not available on PMC but on other platforms, such as the publisher's website? For example, PMID:37230009
Wasim Aftab
CC BY-SA 4.0
2023-07-17T23:15:03.573Z
You're welcome; just trying to help. That cURL error is very likely related to your curl install and file permissions (e.g. https://askubuntu.com/questions/1356327/cant-write-to-a-hidden-path-using-curl); you could try using `wget` instead, e.g. `wget -O - https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 2> /dev/null | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'` should work. And as for articles on Pubmed that aren't on PMC, your question specifies 'articles on PMC', so that is what my answer addresses. If you want to download pdfs from any source, that is a totally different question.
jared_mamrot
CC BY-SA 4.0
2023-07-19T14:41:55.783Z
Your solution is crashing because pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')" is empty for the pmcids that I used: 8968431, 8570779, 8290288, 8254498. But it worked for the pmcid that you provided. It seems pdfs corresponding to my pmcids are not present on the public-facing ftp server because when I tried your code with some of the pdfs that are listed there it worked. So, for my use case, browser automation is the only way so far
Wasim Aftab
CC BY-SA 4.0
2023-07-19T22:57:13.080Z
Ohh! Yep - I see the problem - only Open Access articles are accessible via the API (`<OA><responseDate>2023-07-19 18:55:39</responseDate><request>https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=8968431</request><error code="idIsNotOpenAccess">identifier 'PMC8968431' is not Open Access</error></OA>`). Thanks for taking the time to check!
jared_mamrot
CC BY-SA 4.0
2023-07-17T06:09:16.797Z
2023-07-17T06:09:16.797Z
jared_mamrot
<p>I'm trying to download pdf from pubmed central using selenium webdriver in python. The code is provided below which downloads the pdf fine but, it does not quit automatically. i.e. it is sort of stuck executing <code>browser.get(href)</code> line in <code>download_pmc_pdf()</code>. What could be the reason?</p>
<pre><code>import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
ncbi_url = &quot;https://www.ncbi.nlm.nih.gov/&quot;

download_dir = &quot;pdf_downloaded&quot;

if not os.path.exists(download_dir):
    os.mkdir(download_dir)
    
options = Options()
options.set_preference('permissions.default.stylesheet', 2)
options.set_preference('permissions.default.image', 2)
options.set_preference('browser.download.folderList', 2)
options.set_preference('browser.download.dir', os.path.abspath(download_dir))
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
options.set_preference('pdfjs.disabled', True)

browser = webdriver.Firefox(options=options)

###############################################################################
def download_pmc_pdf(pmcid_key):
    pmc_url = ncbi_url + &quot;pmc/articles/&quot; + pmcid_key + &quot;/&quot;
    browser.get(pmc_url)
    try:
        wait = WebDriverWait(browser, 10)
        pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
        
        # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
        for link in pdf_links:
            href = link.get_attribute('href')
            if '.pdf' in href:
                browser.get(href)
                break  # exit the loop as we have found the PDF link
        print(&quot;line beofre browser.quit()&quot;)
        browser.quit()  # quit the browser outside the loop
    except Exception as e:
        print(e)


    
###############################################################################
pmcid = &quot;2788555&quot;            
download_pmc_pdf(pmcid)
browser.quit()
</code></pre>

CC BY-SA 4.0
2023-07-14T14:25:14.893Z
This might be better-suited to stackoverflow since it's primarily a selenium question rather than a bioinformatics question, but for what it's worth, I see the same behavior when I try your script.  (I've used selenium a bit before but never tried `get` for anything other than HTML.)  I don't suppose the e-utils support PDF paper download, do they?
Jesse
CC BY-SA 4.0
2023-07-14T15:50:21.883Z
To the best of my knowledge, E-utils primarily supports text, not PDF downloads. For some PMC-listed articles, the XML, mediline records lack full text, hence the need for browser automation. Though e-utils is my preferred choice, this situation demands an alternative. Nonetheless, the issue is now resolved
Wasim Aftab
<python>
selenium webdriver is stuck after downloading and saving a pdf file from pubmed central website
# selenium webdriver is stuck after downloading and saving a pdf file from pubmed central website

(Asked by: username_0 on 2023-07-13)

I'm trying to download pdf from pubmed central using selenium webdriver in python. The code is provided below which downloads the pdf fine but, it does not quit automatically. i.e. it is sort of stuck executing browser.get(href) line in download_pmc_pdf(). What could be the reason?
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
ncbi_url = "https://www.ncbi.nlm.nih.gov/"

download_dir = "pdf_downloaded"

if not os.path.exists(download_dir):
    os.mkdir(download_dir)
    
options = Options()
options.set_preference('permissions.default.stylesheet', 2)
options.set_preference('permissions.default.image', 2)
options.set_preference('browser.download.folderList', 2)
options.set_preference('browser.download.dir', os.path.abspath(download_dir))
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
options.set_preference('pdfjs.disabled', True)

browser = webdriver.Firefox(options=options)

###############################################################################
def download_pmc_pdf(pmcid_key):
    pmc_url = ncbi_url + "pmc/articles/" + pmcid_key + "/"
    browser.get(pmc_url)
    try:
        wait = WebDriverWait(browser, 10)
        pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))
        
        # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
        for link in pdf_links:
            href = link.get_attribute('href')
            if '.pdf' in href:
                browser.get(href)
                break  # exit the loop as we have found the PDF link
        print("line beofre browser.quit()")
        browser.quit()  # quit the browser outside the loop
    except Exception as e:
        print(e)


    
###############################################################################
pmcid = "2788555"            
download_pmc_pdf(pmcid)
browser.quit()

**Tags:** python
> **Comments:**
> **username_1**: This might be better-suited to stackoverflow since it's primarily a selenium question rather than a bioinformatics question, but for what it's worth, I see the same behavior when I try your script.  (I've used selenium a bit before but never tried `get` for anything other than HTML.)  I don't suppose the e-utils support PDF paper download, do they?  
> **username_0**: To the best of my knowledge, E-utils primarily supports text, not PDF downloads. For some PMC-listed articles, the XML, mediline records lack full text, hence the need for browser automation. Though e-utils is my preferred choice, this situation demands an alternative. Nonetheless, the issue is now resolved  

---

## 2 answers

### Answer by username_0 on 2023-07-14. Score: 2

I had to execute a JavaScript code that opens the pdf link in a new window/tab. After that, just closed this newly opened window/tab and continued with the script execution. Here is the updated code:
import os
import time
import csv

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

ncbi_url = "https://www.ncbi.nlm.nih.gov/"

download_dir = "pdf_downloaded"

if not os.path.exists(download_dir):
    os.mkdir(download_dir)
    
options = Options()
options.set_preference('permissions.default.stylesheet', 2)
options.set_preference('permissions.default.image', 2)
options.set_preference('browser.download.folderList', 2)
options.set_preference('browser.download.dir', os.path.abspath(download_dir))
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
options.set_preference('pdfjs.disabled', True)

###############################################################################
def download_pmc_pdf(pmcid_key):
    browser = webdriver.Firefox(options=options)
    pmc_url = ncbi_url + "pmc/articles/" + pmcid_key + "/"
    browser.get(pmc_url)
    try:
        wait = WebDriverWait(browser, 10)
        pdf_links = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.pmc-sidebar__formats li a')))

        # Iterate over all links and navigate to the link containing '.pdf' in the href attribute
        for link in pdf_links:
            href = link.get_attribute('href')
            if '.pdf' in href:
                # Open the pdf link in a new window
                browser.execute_script(f'window.open("{href}","_blank");')
                break

        time.sleep(15)  # delay for 10 seconds
        # After the file is downloaded:
        with open(f'{download_dir}/file_mapping.csv', 'a', newline='') as file:
            writer = csv.writer(file)
            writer.writerow([pmcid_key, os.path.basename(href)])
        ## Switch to the new window and close it 
        ## (this is a hack to deal with bowser freezing upon downloading)
        browser.switch_to.window(browser.window_handles[-1])
        browser.close()

    except Exception as e:
        print(e)
    finally:
        browser.quit()  # quit the browser outside the loop    
###############################################################################
pmcid = "2788555"   
download_pmc_pdf(pmcid)


---

### Answer by username_2 on 2023-07-17. Score: 1

Pubmed Central stores pdfs for all available open access articles on a public-facing ftp server (https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/). You can query the database using the OA Web Service API, e.g. with cURL
# for pubmed id "10026098"
curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098
2023-07-17 01:39:13https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098

# get the pdf link from the API (ftp storage)
curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'
ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_pdf/83/eb/10.1177_17588359231161409.PMC10026098.pdf

# download the pdf
pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
curl -O "$f"

# download many pdfs from a list of ids
cat list_of_ids.txt
10026098
9341183

while read -r id_from_list
do
    pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')"
    curl -O "$pdf"
done < list_of_ids.txt

This approach is preferable to selenium scraping for many reasons, and if you have any problems the NCBI provides documentation (https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/#indart) and offers support (you can email them at pubmedcentral@ncbi.nlm.nih.gov).

> **Comments:**
> **username_0**: Thank you for your time! Unfortunately, it did not work, returning the error `curl: (23) Failed writing received data to disk/application` Furthermore, I've noticed a potential issue. What if the article listed on PubMed is not available on PMC but on other platforms, such as the publisher's website? For example, PMID:37230009  
> **username_2**: You're welcome; just trying to help. That cURL error is very likely related to your curl install and file permissions (e.g. https://askubuntu.com/questions/1356327/cant-write-to-a-hidden-path-using-curl); you could try using `wget` instead, e.g. `wget -O - https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=10026098 2> /dev/null | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p'` should work. And as for articles on Pubmed that aren't on PMC, your question specifies 'articles on PMC', so that is what my answer addresses. If you want to download pdfs from any source, that is a totally different question.  
> **username_0**: Your solution is crashing because pdf="$(curl https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=$id_from_list | sed -n 's/.*\"\(ftp.*.pdf\)\".*/\1/p')" is empty for the pmcids that I used: 8968431, 8570779, 8290288, 8254498. But it worked for the pmcid that you provided. It seems pdfs corresponding to my pmcids are not present on the public-facing ftp server because when I tried your code with some of the pdfs that are listed there it worked. So, for my use case, browser automation is the only way so far  
> **username_2**: Ohh! Yep - I see the problem - only Open Access articles are accessible via the API (`<OA><responseDate>2023-07-19 18:55:39</responseDate><request>https://www.ncbi.nlm.nih.gov/pmc/utils/oa/oa.fcgi?id=8968431</request><error code="idIsNotOpenAccess">identifier 'PMC8968431' is not Open Access</error></OA>`). Thanks for taking the time to check!  

---

