<p>I once needed to download more than 1000 files from SRA, and I found a blog post that is very useful (<a href="https://reneshbedre.github.io/blog/fqutil.html" rel="nofollow noreferrer">https://reneshbedre.github.io/blog/fqutil.html</a>). The idea is to use aspera to fetch SRA files. This is fast becasue of aspera. Then use fasterq-dump to convert SRA file to fastq. fasterq-dump is faster than fastq-dump partly because it does not compress the output file. I think compressing is the rate-limiting step here. After this, you can use pigz, which is a multi-threading version of gzip to compress the fastq files. Note that if you want to download files to a computing cluster, you can submit jobs for fasterq-dump and pigz. This makes things much faster. You cannot submit jobs for fetching SRA, because you need to connect to internet.</p>

CC BY-SA 4.0
2020-07-13T15:59:35.270Z
Thanks for sharing this. So I tried running prefetch with aspera as explained in the blog, but I get unrecognized option when I include the '-a' option. I do have aspera installed. Did you have this issue by any chance?
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-13T16:32:43.377Z
seems like aspera is no longer supported https://github.com/ncbi/sra-tools/issues/255
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-13T16:33:18.447Z
however, prefetch is still very fast, thanks for sharing this, update your answer to reflect this and I'll accept it!
An Ignorant Wanderer
CC BY-SA 4.0
2020-07-13T15:07:51.370Z
2020-07-13T15:07:51.370Z
Phoenix Mu
<p>I have the following code to download data from the SRA using multithreading in Python. After running this a few times now (for testing purposes), I keep getting denied access to the data. Not sure how to fix this. In particular, <code>cat</code>ing the output files gives:</p>
<pre><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;&lt;Error&gt;&lt;Code&gt;AccessDenied&lt;/Code&gt;&lt;Message&gt;Access Denied&lt;/Message&gt;&lt;RequestId&gt;cb0f0e98-cafb-1dd7-9b7b-d8c49756ec52&lt;/RequestId&gt;&lt;HostId&gt;QeDGVwBXYp61J0B4_OUTn7UsEsiQEec0n18DAeR0kaE&lt;/HostId&gt;&lt;/Error&gt;
</code></pre>
<p>Here is my code:</p>
<pre><code>import threading
import requests
import time 

start = time.perf_counter()

class MyThread(threading.Thread):
    def __init__(self, url):
        threading.Thread.__init__(self)
        self.url = url
        self.result = None
        self.filename = url.split('/')[-1]
    def run(self):
        res = requests.get(self.url)
        with open(self.filename, 'wb') as f:
            f.write(res.content)
urls = [
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.1',
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.2',
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.1',
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.2']

threads = [MyThread(url, ) for url in urls]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')
</code></pre>

<python>
What is the best way to get a large number of RNA seq data from SRA in Python without being denied access
# What is the best way to get a large number of RNA seq data from SRA in Python without being denied access

(Asked by: username_0 on 2020-07-12)

I have the following code to download data from the SRA using multithreading in Python. After running this a few times now (for testing purposes), I keep getting denied access to the data. Not sure how to fix this. In particular, cating the output files gives:
AccessDeniedAccess Deniedcb0f0e98-cafb-1dd7-9b7b-d8c49756ec52QeDGVwBXYp61J0B4_OUTn7UsEsiQEec0n18DAeR0kaE

Here is my code:
import threading
import requests
import time 

start = time.perf_counter()

class MyThread(threading.Thread):
    def __init__(self, url):
        threading.Thread.__init__(self)
        self.url = url
        self.result = None
        self.filename = url.split('/')[-1]
    def run(self):
        res = requests.get(self.url)
        with open(self.filename, 'wb') as f:
            f.write(res.content)
urls = [
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.1',
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000001/SRR000001.2',
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.1',
        'https://sra-downloadb.be-md.ncbi.nlm.nih.gov/sos1/sra-pub-run-1/SRR000002/SRR000002.2']

threads = [MyThread(url, ) for url in urls]
for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

**Tags:** python

---

## 1 answers

### Answer by username_1 on 2020-07-13. Score: 0 | Accepted answer

I once needed to download more than 1000 files from SRA, and I found a blog post that is very useful (<https://reneshbedre.github.io/blog/fqutil.html>). The idea is to use aspera to fetch SRA files. This is fast becasue of aspera. Then use fasterq-dump to convert SRA file to fastq. fasterq-dump is faster than fastq-dump partly because it does not compress the output file. I think compressing is the rate-limiting step here. After this, you can use pigz, which is a multi-threading version of gzip to compress the fastq files. Note that if you want to download files to a computing cluster, you can submit jobs for fasterq-dump and pigz. This makes things much faster. You cannot submit jobs for fetching SRA, because you need to connect to internet.

> **Comments:**
> **username_0**: Thanks for sharing this. So I tried running prefetch with aspera as explained in the blog, but I get unrecognized option when I include the '-a' option. I do have aspera installed. Did you have this issue by any chance?  
> **username_0**: seems like aspera is no longer supported https://github.com/ncbi/sra-tools/issues/255  
> **username_0**: however, prefetch is still very fast, thanks for sharing this, update your answer to reflect this and I'll accept it!  

---

