<p>As @vkkodali pointed out, the problem turned out to be that you shouldn't use multithreading.</p>

CC BY-SA 4.0
2020-07-11T23:43:22.900Z
2020-07-11T23:43:22.900Z
An Ignorant Wanderer
<p>The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.</p>
<p><strong>In an argument to an EDirect command:</strong></p>
<pre><code>esearch -db nuccore -query 'some query' -api_key 12345
</code></pre>
<p><strong>In a UNIX environment:</strong></p>
<p>You can either run this just before you are running some edirect commands or add the following line to your <code>.bashrc</code> file.</p>
<pre><code>export NCBI_API_KEY=12345
</code></pre>
<p><strong>In the environment, from a Perl script:</strong></p>
<pre><code>$ENV{NCBI_API_KEY} = &quot;12345&quot;
</code></pre>

CC BY-SA 4.0
2020-07-10T00:48:42.473Z
did as you said, yet I still get "too many requests" for quite a few
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-10T00:55:16.293Z
Even went as far as setting the sleep time to 10, and the step size for the for loop to 1, and still get too many requests
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-10T02:13:40.723Z
I typically don't use Python for this sort of thing. I just run the command in a bash shell. I suspect the reason for the too many requests error is because you are using multithreading. Does your code simultaneously run 10 instances of the `fetch_metadata` function? May be try without the multithreading to see if it works?
vkkodali
CC BY-SA 4.0
2020-07-10T02:19:48.120Z
yes that worked. Interesting, thanks!
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-10T00:36:27.757Z
2020-07-10T00:36:27.757Z
vkkodali
<p>I had the same problem. I was making a lot of requests and with the thread it blocked very quickly. But I solved the problem with multiprocessing. Pool and <code>imap_unordered</code>. I can even put <code>ThreadPoolExecutor</code> inside the function. I'm making an application in Gradio (hence the <code>gr.Process()</code>).</p>
<p>Then there are a few subtleties, if you see correctly, there's an infinite loop in process_gene if there's a 429 error or something else. If it's a 429 error it's just that the server doesn't want to because too many requests have been made, if it's another error you should stop and restart the gene or at least analyse the error better. If, for example, it's a 500 error, it's just that the server is down for maintenance, so you might as well try again.</p>
<p>Then I make packets from my gene list, because when in doubt, I prefer to send in packets of 50 to avoid overloading the various processor cores. And to finish everything off, I use <code>pool.close()</code> and <code>pool.join()</code> to make sure it finishes all the processes and &quot;empties&quot; the cores to send the next group of genes.</p>
<p>Now, some information about how I tested and what I have concluded. I tried it on a Windows on Windows 10, 8GB RAM, almost 10 year old i5 processor (4 cores) and I tried it on a Mac Mini M2 8GB RAM, it has 8 cores. As you can see in my code (<code>multiprocessing.Pool(processes=multiprocessing.cpu_count())</code>), I used all the cores. With Windows, I manage to extract 90 gene_names per minute (not with the script below, I have another script where each <code>process_gene()</code> makes between 4 and 5 requests in a row + web-scraping, the script below is just an example). With Windows, sometimes I still get error 429 but as there's a loop, I get through it. In theory, the Mac should be 2x faster because it has 8 cores. But as the API limits to 10 requests per second, I'm stuck at 90 gene_names per minute.</p>
<p>I strongly advise you to use the API key, but you're already doing that.</p>
<p>I hope this helps :)</p>
<pre class="lang-py prettyprint-override"><code>import random
import time
import requests

def process_gene(gene_id):
    while True:
        url = f&quot;https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&amp;id={gene_id}&amp;retmode=json&amp;rettype=xml&amp;api_key={YOUR_API_KEY}&quot;
        response = requests.get(url)

        if response.status_code == 200:
            response_data = response.json()
            try:
                gene_info = response_data['result'][str(gene_id)]
                gene_name = gene_info['name']

                print(f&quot;Response 200: Info for {gene_id} {gene_name} retrieved.&quot;)
                return gene_name, f&quot;Response 200: Info for {gene_id} {gene_name} retrieved.&quot;, gene_id
            
            except Exception as e:
                print(f&quot;Response 200: Info for {gene_id} not found.&quot;)
                return &quot;Error 200&quot;, f&quot;Info for {gene_id} not found.&quot;, gene_id

        elif response.status_code == 429:
            print(f&quot;Error 429: API rate limit exceeded during get {gene_id} info, try again: {response.text}&quot;)
            time.sleep(random.uniform(0.25, 0.5))
        else:
            print(f&quot;Error {response.status_code}: {response.text}&quot;)
            time.sleep(random.uniform(0.25, 0.5))
    

def your_function(list_gene, progress=gr.Progress()):
    result = []
    result_error = []
    gene_ids = list_gene.strip().split(&quot;\n&quot;)
    total_genes = len(gene_ids)
    progress(0.0, desc=&quot; Extracting... ⚠️ PLEASE WAIT UNTIL COMPLETION&quot;)

    gene_groups = [gene_ids[i:i + 50] for i in range(0, len(gene_ids), 50)]
    i = 0
    j = 0

    for group in gene_groups:
        j += 1
        with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
            results = pool.imap_unordered(process_gene,[gene_id for gene_id in group])

            for result_output, message, gene_id in results:
                if result_output == &quot;Error 200&quot;:
                    result_error.append(message)
                else:
                    result.append(result_output)

                i += 1
                progress(i / total_genes, desc=f&quot; Extracting {gene_id}... {j}/{len(gene_groups)} - {i}/{total_genes}&quot;)

            pool.close()
            pool.join()
            
    return result, result_error
</code></pre>

CC BY-SA 4.0
2024-09-06T11:05:36.993Z
2024-09-09T14:56:13.323Z
2024-09-09T14:56:13.323Z
Julien Minniti
<p>I keep getting a &quot;too many requests error&quot; when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.</p>
<p>Here is my (Python) code:</p>
<pre><code>import subprocess
import concurrent.futures
import time

bioprojects = [
    &quot;PRJNA644722&quot;,
    &quot;PRJNA644892&quot;
]

all_metadata_generators = []

def fetch_metadata(bioproject):
    return subprocess.run(f&quot;esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo&quot;, shell=True, capture_output=True, text=True)

with concurrent.futures.ThreadPoolExecutor() as executor:
    for i in range(0, len(bioprojects), 10):
        all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
        time.sleep(1)


for metadata_generator in all_metadata_generators:
    for metadata in metadata_generator:
        print(metadata)

</code></pre>

CC BY-SA 4.0
2020-07-09T21:52:07.963Z
Ahh... this was often rumoured and there whispers on the wind of departmental bans - just rumours. You bang on with the `sleep`, I would just increase it until the signal detection police are happy. If you use a try: exception loop you could even try and attempt to attenuate the `sleep` time to just about the detection limit. Goodluck
M__
CC BY-SA 4.0
2020-07-09T22:00:46.303Z
@Michael thing is here `sleep` is technically not even doing anything since I only have 2 bioproject ids. So I can't even do 2 requests in more than one second without problems
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-09T23:02:46.327Z
where do you have the API key? In your `.bashrc` file? I would run the same command from the command line to check if you still get the 429 error. If you don't then you may have to specify the API key in the python command again.
vkkodali
CC BY-SA 4.0
2020-07-09T23:11:47.843Z
@vkkodali how do you specify it on the command line? I couldn't find it in the e-utilities documentation
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-09T23:14:42.257Z
@vkkodali ah I see now that actually the API key was not set. I set it from Biopython (and I'm not using Biopython in my code) so obviously it wasn't going to be used above. Silly me!
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-09T23:15:55.587Z
@vkkodali but how do you pass it to the esearch / efetch commands?
An Ignorant Wanderer
<python><phylogenetics>
Too many requests error when querying NCBI
# Too many requests error when querying NCBI

(Asked by: username_0 on 2020-07-09)

I keep getting a "too many requests error" when querying the NCBI SRA database, even though I'm running less than 10 requests per second, and I have an API key, which supposedly should allow me to run 10 per second.
Here is my (Python) code:
import subprocess
import concurrent.futures
import time

bioprojects = [
    "PRJNA644722",
    "PRJNA644892"
]

all_metadata_generators = []

def fetch_metadata(bioproject):
    return subprocess.run(f"esearch -db sra -query '{bioproject}[bioproject]' | efetch -format runinfo", shell=True, capture_output=True, text=True)

with concurrent.futures.ThreadPoolExecutor() as executor:
    for i in range(0, len(bioprojects), 10):
        all_metadata_generators.append(executor.map(fetch_metadata, bioprojects[i:i+10]))
        time.sleep(1)


for metadata_generator in all_metadata_generators:
    for metadata in metadata_generator:
        print(metadata)

**Tags:** python, phylogenetics
> **Comments:**
> **username_1**: Ahh... this was often rumoured and there whispers on the wind of departmental bans - just rumours. You bang on with the `sleep`, I would just increase it until the signal detection police are happy. If you use a try: exception loop you could even try and attempt to attenuate the `sleep` time to just about the detection limit. Goodluck  
> **username_0**: @Michael thing is here `sleep` is technically not even doing anything since I only have 2 bioproject ids. So I can't even do 2 requests in more than one second without problems  
> **username_2**: where do you have the API key? In your `.bashrc` file? I would run the same command from the command line to check if you still get the 429 error. If you don't then you may have to specify the API key in the python command again.  
> **username_0**: @vkkodali how do you specify it on the command line? I couldn't find it in the e-utilities documentation  
> **username_0**: @vkkodali ah I see now that actually the API key was not set. I set it from Biopython (and I'm not using Biopython in my code) so obviously it wasn't going to be used above. Silly me!  
> **username_0**: @vkkodali but how do you pass it to the esearch / efetch commands?  

---

## 3 answers

### Answer by username_0 on 2020-07-11. Score: 2 | Accepted answer

As @vkkodali pointed out, the problem turned out to be that you shouldn't use multithreading.


---

### Answer by username_2 on 2020-07-10. Score: 4

The execution of your EDirect commands may be limited unless you are using an API Key, which you can create from your MyNCBI account. Your API Key can be used in one of several different ways.
In an argument to an EDirect command:
esearch -db nuccore -query 'some query' -api_key 12345

In a UNIX environment:
You can either run this just before you are running some edirect commands or add the following line to your .bashrc file.
export NCBI_API_KEY=12345

In the environment, from a Perl script:
$ENV{NCBI_API_KEY} = "12345"

> **Comments:**
> **username_0**: did as you said, yet I still get "too many requests" for quite a few  
> **username_0**: Even went as far as setting the sleep time to 10, and the step size for the for loop to 1, and still get too many requests  
> **username_2**: I typically don't use Python for this sort of thing. I just run the command in a bash shell. I suspect the reason for the too many requests error is because you are using multithreading. Does your code simultaneously run 10 instances of the `fetch_metadata` function? May be try without the multithreading to see if it works?  
> **username_0**: yes that worked. Interesting, thanks!  

---

### Answer by username_3 on 2024-09-06. Score: 3

I had the same problem. I was making a lot of requests and with the thread it blocked very quickly. But I solved the problem with multiprocessing. Pool and imap_unordered. I can even put ThreadPoolExecutor inside the function. I'm making an application in Gradio (hence the gr.Process()).
Then there are a few subtleties, if you see correctly, there's an infinite loop in process_gene if there's a 429 error or something else. If it's a 429 error it's just that the server doesn't want to because too many requests have been made, if it's another error you should stop and restart the gene or at least analyse the error better. If, for example, it's a 500 error, it's just that the server is down for maintenance, so you might as well try again.
Then I make packets from my gene list, because when in doubt, I prefer to send in packets of 50 to avoid overloading the various processor cores. And to finish everything off, I use pool.close() and pool.join() to make sure it finishes all the processes and "empties" the cores to send the next group of genes.
Now, some information about how I tested and what I have concluded. I tried it on a Windows on Windows 10, 8GB RAM, almost 10 year old i5 processor (4 cores) and I tried it on a Mac Mini M2 8GB RAM, it has 8 cores. As you can see in my code (multiprocessing.Pool(processes=multiprocessing.cpu_count())), I used all the cores. With Windows, I manage to extract 90 gene_names per minute (not with the script below, I have another script where each process_gene() makes between 4 and 5 requests in a row + web-scraping, the script below is just an example). With Windows, sometimes I still get error 429 but as there's a loop, I get through it. In theory, the Mac should be 2x faster because it has 8 cores. But as the API limits to 10 requests per second, I'm stuck at 90 gene_names per minute.
I strongly advise you to use the API key, but you're already doing that.
I hope this helps :)
import random
import time
import requests

def process_gene(gene_id):
    while True:
        url = f"https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=gene&id={gene_id}&retmode=json&rettype=xml&api_key={YOUR_API_KEY}"
        response = requests.get(url)

        if response.status_code == 200:
            response_data = response.json()
            try:
                gene_info = response_data['result'][str(gene_id)]
                gene_name = gene_info['name']

                print(f"Response 200: Info for {gene_id} {gene_name} retrieved.")
                return gene_name, f"Response 200: Info for {gene_id} {gene_name} retrieved.", gene_id
            
            except Exception as e:
                print(f"Response 200: Info for {gene_id} not found.")
                return "Error 200", f"Info for {gene_id} not found.", gene_id

        elif response.status_code == 429:
            print(f"Error 429: API rate limit exceeded during get {gene_id} info, try again: {response.text}")
            time.sleep(random.uniform(0.25, 0.5))
        else:
            print(f"Error {response.status_code}: {response.text}")
            time.sleep(random.uniform(0.25, 0.5))
    

def your_function(list_gene, progress=gr.Progress()):
    result = []
    result_error = []
    gene_ids = list_gene.strip().split("\n")
    total_genes = len(gene_ids)
    progress(0.0, desc=" Extracting... ⚠️ PLEASE WAIT UNTIL COMPLETION")

    gene_groups = [gene_ids[i:i + 50] for i in range(0, len(gene_ids), 50)]
    i = 0
    j = 0

    for group in gene_groups:
        j += 1
        with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
            results = pool.imap_unordered(process_gene,[gene_id for gene_id in group])

            for result_output, message, gene_id in results:
                if result_output == "Error 200":
                    result_error.append(message)
                else:
                    result.append(result_output)

                i += 1
                progress(i / total_genes, desc=f" Extracting {gene_id}... {j}/{len(gene_groups)} - {i}/{total_genes}")

            pool.close()
            pool.join()
            
    return result, result_error


---

