<p>Answers to this question can be found on <a href="https://www.biostars.org/p/91885/" rel="noreferrer">biostars</a>. To give you a short summary:</p>

<ol>
<li><p>Faster way might be to use the <code>parallel-fastq-dump</code>, as suggested in this <a href="https://www.biostars.org/p/91885/#300457" rel="noreferrer">answer</a>. I never tested that tool though, my own experience is that <code>prefetch</code> is more stable than <code>fastq-dump</code> command.</p></li>
<li><p>In that same post it seems that the Japanese mirror seems faster, don't know if that is still valid.</p></li>
<li><p>The reason this is never optimized might be that downloading from <code>SRA</code> is not a task you'll have to do repeatedly, as described here in this <a href="https://www.biostars.org/p/91885/#91895" rel="noreferrer">post</a>.</p></li>
</ol>

CC BY-SA 4.0
2019-04-24T10:02:46.080Z
Thanks, comprehensive answer. Is it this site https://www.ddbj.nig.ac.jp/  ?
M__
CC BY-SA 4.0
2019-04-24T10:17:37.637Z
https://www.ddbj.nig.ac.jp/dra/index-e.html, yes it is the same.
benn
CC BY-SA 4.0
2019-04-24T08:36:07.600Z
2019-04-24T08:36:07.600Z
benn
<p>You can try <code>wget</code> to download SRA files from NCBI server. This seems to be 2x faster in terms of CPU time than <code>prefetch</code> and has the advantage of not clogging your <code>/home/ncbi/public/sra/</code> directory in case you have space limitation. The tutorial for <code>wget</code> downloading method is available at <a href="https://github.com/johanzi/ncbi_tutorial" rel="nofollow noreferrer">https://github.com/johanzi/ncbi_tutorial</a>.</p>

<p>Here my timing of the <code>prefetch</code> method:</p>

<pre><code>time prefetch SRR3300113
real    3m5.609s
user    0m6.332s
sys     0m2.076s
</code></pre>

<p>Versus the <code>wget</code> method:</p>

<pre><code>time wget ftp://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR330/SRR3300113/SRR3300113.sra
real    3m4.696s
user    0m0.456s
sys     0m3.332s
</code></pre>

<p>That makes a user+sys time of 8.402s for <code>prefetch</code> and 3.788s for <code>wget</code>. Not so much difference for real time though (about a second).</p>

CC BY-SA 4.0
2019-04-29T15:56:38.050Z
Hello Johan, welcome to BIoinformatics.SE. Is there a reason to think that `wget` would be faster? Intuitively I would guess that it's the server that is clogged, then using a different program to query the server does not sound like something that would help. (by the way, my inability to see how it could help does not mean that it does not :-)
Kamil S Jaron
CC BY-SA 4.0
2019-04-30T14:55:09.850Z
Hi Kamil, the `wget` process is about 2x faster in terms of CPU time than `prefetch` method based on my test:
Johan Zicola
CC BY-SA 4.0
2019-05-01T07:47:41.540Z
`user` + `sys` sort of tells you how long CPUs were active during the `real` time of execution (i.e. for user `real` is what matters, you can check [this](https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1)). So, what I would get out of your nice test is, that the `wget` is a more optimized tool, but both of them are fetching the data from exactly the same place and the limiting factor is the speed of their ftp.
Kamil S Jaron
CC BY-SA 4.0
2019-05-01T11:52:46.773Z
Yes. I would say the main advantage of `wget` is that you can choose where to put your downloaded SRA files while it goes in default into your home directory `/home/user/ncbi/public/sra` with `prefetch`. It can be a problem when you work on a server and you have limited disk usage for your home directory.
Johan Zicola
CC BY-SA 4.0
2021-03-20T02:53:44.593Z
You can choose where to put the cache, at least as of a few versions ago, via vdb-config. I agree it's a total pain that it defaults to the home directory instead of the current one, but it's a fixable problem.
GenesRus
CC BY-SA 4.0
2019-04-29T11:59:58.650Z
2019-05-01T14:34:27.090Z
2019-05-01T14:34:27.090Z
Johan Zicola
<p>I have 2 .csv files that contains a list of accession codes. For example for this <a href="https://www.ncbi.nlm.nih.gov/biosample?LinkName=bioproject_biosample_all&amp;from_uid=520914" rel="nofollow noreferrer">experiment</a> a .csv file will contain:</p>
<pre><code>SAMN10870244
SAMN10870243
SAMN10870242
SAMN10870241
SAMN10870240
SAMN10870239
SAMN10870238
SAMN10870237
SAMN10870236
SAMN10870235
SAMN10870234
SAMN10870233
</code></pre>
<p>Then I run the following</p>
<pre><code>for i in <span class="math-container">$(cat *.csv); do echo $</span>{i}; /usr/bin/sratoolkit/bin/fastq-dump --gzip ${i} &amp; done
</code></pre>
<ol>
<li>Some of my downloads takes a lot of time days. Is there a faster way to get stuff from NCBI?</li>
<li>Are there mirrors of NCBI to accelerate the process?</li>
<li>Why is it so slow?</li>
</ol>
<hr />
<p><strong>EDIT 2024</strong><br />
You might want to look into:</p>
<ul>
<li><a href="https://github.com/kblin/ncbi-acc-download" rel="nofollow noreferrer"><code>ncbi-acc-download</code></a>: A CLI tool for efficient downloads from NCBI Entrez.</li>
<li><a href="https://nsdpy.readthedocs.io/en/latest/main.html" rel="nofollow noreferrer"><code>NSDPY</code></a>: A Python library for batch downloading with additional features like taxonomic sorting.</li>
</ul>

<phylogenetics>
How do you download stuff from NCBI fast?
# How do you download stuff from NCBI fast?

(Asked by: username_0 on 2019-04-24)

I have 2 .csv files that contains a list of accession codes. For example for this experiment a .csv file will contain:
SAMN10870244
SAMN10870243
SAMN10870242
SAMN10870241
SAMN10870240
SAMN10870239
SAMN10870238
SAMN10870237
SAMN10870236
SAMN10870235
SAMN10870234
SAMN10870233

Then I run the following
for i in $(cat *.csv); do echo ${i}; /usr/bin/sratoolkit/bin/fastq-dump --gzip ${i} & done


Some of my downloads takes a lot of time days. Is there a faster way to get stuff from NCBI?
Are there mirrors of NCBI to accelerate the process?
Why is it so slow?


EDIT 2024
You might want to look into:

ncbi-acc-download: A CLI tool for efficient downloads from NCBI Entrez.
NSDPY: A Python library for batch downloading with additional features like taxonomic sorting.

**Tags:** phylogenetics

---

## 2 answers

### Answer by username_1 on 2019-04-24. Score: 6

Answers to this question can be found on [biostars](https://www.biostars.org/p/91885/). To give you a short summary:

1. Faster way might be to use the `parallel-fastq-dump`, as suggested in this [answer](https://www.biostars.org/p/91885/#300457). I never tested that tool though, my own experience is that `prefetch` is more stable than `fastq-dump` command.
2. In that same post it seems that the Japanese mirror seems faster, don't know if that is still valid.
3. The reason this is never optimized might be that downloading from `SRA` is not a task you'll have to do repeatedly, as described here in this [post](https://www.biostars.org/p/91885/#91895).

> **Comments:**
> **username_2**: Thanks, comprehensive answer. Is it this site https://www.ddbj.nig.ac.jp/  ?  
> **username_1**: https://www.ddbj.nig.ac.jp/dra/index-e.html, yes it is the same.  

---

### Answer by username_3 on 2019-04-29. Score: 1

You can try wget to download SRA files from NCBI server. This seems to be 2x faster in terms of CPU time than prefetch and has the advantage of not clogging your /home/ncbi/public/sra/ directory in case you have space limitation. The tutorial for wget downloading method is available at https://github.com/johanzi/ncbi_tutorial.
Here my timing of the prefetch method:
time prefetch SRR3300113
real    3m5.609s
user    0m6.332s
sys     0m2.076s

Versus the wget method:
time wget ftp://ftp-trace.ncbi.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR330/SRR3300113/SRR3300113.sra
real    3m4.696s
user    0m0.456s
sys     0m3.332s

That makes a user+sys time of 8.402s for prefetch and 3.788s for wget. Not so much difference for real time though (about a second).

> **Comments:**
> **username_4**: Hello Johan, welcome to BIoinformatics.SE. Is there a reason to think that `wget` would be faster? Intuitively I would guess that it's the server that is clogged, then using a different program to query the server does not sound like something that would help. (by the way, my inability to see how it could help does not mean that it does not :-)  
> **username_3**: Hi Kamil, the `wget` process is about 2x faster in terms of CPU time than `prefetch` method based on my test:  
> **username_4**: `user` + `sys` sort of tells you how long CPUs were active during the `real` time of execution (i.e. for user `real` is what matters, you can check [this](https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1)). So, what I would get out of your nice test is, that the `wget` is a more optimized tool, but both of them are fetching the data from exactly the same place and the limiting factor is the speed of their ftp.  
> **username_3**: Yes. I would say the main advantage of `wget` is that you can choose where to put your downloaded SRA files while it goes in default into your home directory `/home/user/ncbi/public/sra` with `prefetch`. It can be a problem when you work on a server and you have limited disk usage for your home directory.  
> **username_5**: You can choose where to put the cache, at least as of a few versions ago, via vdb-config. I agree it's a total pain that it defaults to the home directory instead of the current one, but it's a fixable problem.  

---

