<p>Apparently there is no <strong>100% reliable</strong> solution, but in the most cases, you should be able to use the <a href="https://github.com/enasequence/enaBrowserTools" rel="nofollow noreferrer">enaBrowserTools</a> python scripts (enaDataGet) to quite reliably download data.</p>

<p>The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.</p>

<p>This answer is based on kind assistance of Suran from ebi helpdesk.</p>

CC BY-SA 4.0
2018-05-11T15:27:03.167Z
Did you reach them through the email?
llrs
CC BY-SA 4.0
2018-05-11T17:15:28.647Z
Do, they are usually very responsive and helpful.
Konrad Rudolph
CC BY-SA 4.0
2018-05-11T14:38:27.583Z
2018-05-25T11:52:34.537Z
2018-05-25T11:52:34.537Z
Kamil S Jaron
<p>Here are two python functions I use for getting stuff from ENA based on an accession:
</p>

<pre class="lang-py prettyprint-override"><code>import os
import itertools
from urllib.request import urlopen

def fetch_ENA_files(accession):
    """Get the names of the files matching the ENA accession"""

    # Get the paths to the files
    url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&amp;result=read_run&amp;fields=fastq_ftp" \
          % accession

    try:
        paths = urlopen(url).readlines()[1:]
    except:
        E.debug("couldn't access %s" % url)
        raise

    paths = list(itertools.chain.from_iterable(
        [p.strip().split(";") for p in paths]))
    filenames = [os.path.basename(x) for x in paths]
    dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
    return filenames, dl_paths

def fetch_ENA(dl_path, outdir, protocol="ascp"):
    """Fetch fastq from ENA given accession"""

    if protocol == "ascp":
        statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                       era-fasp@fasp.sra.ebi.ac.uk:/%(dl_path)s %(outdir)s """ % locals()
    elif protocol == "http":
        fn = os.path.basename(dl_path)
        outfile = os.path.join(outdir, fn)
        statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()

    return statement
</code></pre>

<p><code>fetch_ENA_files</code> queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found. </p>

<p><code>fetch_ENA(dl_path, outdir, protocol)</code> takes a download path and downloads it to <code>outdir</code> using with ftp or ascp.</p>

<p>You might use it like:</p>

<pre class="lang-py prettyprint-override"><code>filenames, paths = fetch_ENA_files("ERR2135453")
for dlp in paths:
    bash_statement = fetch_ENA(dlp, "downloads", "ascp")
    os.system(bash_statement)
</code></pre>

<p>(we don't we actaully use our modules to send it to the cluster)</p>

CC BY-SA 4.0
2018-05-11T17:06:32.320Z
I had to decode the paths from bytes to strings to get this to work. It doesn't actually work for the accession in question so I guess that's further evidence that there's no way to reliably fetch data sets that are too recent to be in SRA
heathobrien
CC BY-SA 4.0
2018-05-14T08:47:22.537Z
Sorry, should have pointed out that this is python 2 not 3, hence the encoding problems.
Ian Sudbery
CC BY-SA 4.0
2018-05-11T15:44:23.840Z
2018-05-11T17:06:35.770Z
2018-05-11T17:06:35.770Z
Ian Sudbery
<p>I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA (<a href="https://www.ebi.ac.uk/ena/data/view/ERR2135453" rel="nofollow noreferrer">ERR2135453</a>), <a href="https://www.ncbi.nlm.nih.gov/sra/?term=ERR2135453" rel="nofollow noreferrer">not in SRA</a>. And I found that my download script has its limitations.</p>

<p>My bash script looks like this (requires aspera <code>ascp</code> and <code>fastq-dump</code>) and I wanted to make changes that are commented now</p>

<pre><code>#!/bin/bash

ACCESION=$1
OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh

# if [ ${ACCESION:0:3} == SRA ]; then
    URL=anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
# else
#    URL=ena_url
# fi    

ascp -QT -i $OPENSSH $URL

fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
rm data/"$ACCESION".sra
</code></pre>

<p>The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession (<code>anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/</code>), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is <code>ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453</code>. Where <code>003</code> there comes from? Also not all reads are in <code>err</code> subdirectory. </p>

<p>Is there an easy way how to download reads from ENA knowing just the accession number?</p>

<p>--- edit ---</p>

<p>I found on the relevant ENA <a href="https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera" rel="nofollow noreferrer">site about downloading data using aspera</a> that ftp urls should be possible to derive only from accession also in ENA.</p>

<blockquote>
  <p>Submitted read data files are organised by submission accession number
  under vol1/ directory in ftp.sra.ebi.ac.uk:</p>
  
  <p><a href="ftp://ftp.sra.ebi.ac.uk/vol1/" rel="nofollow noreferrer">ftp://ftp.sra.ebi.ac.uk/vol1/</a>/</p>
  
  <p>where  contains the first 6 letters and
  numbers of the SRA Submission accession.</p>
  
  <p>For example, the files submitted in the SRA Submission ERA007448 are
  available at: <a href="ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/" rel="nofollow noreferrer">ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/</a>.</p>
</blockquote>

<p>The reason why there is this strange <code>003</code> will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).</p>

<p>However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?</p>

<data-download><sratoolkit>
A reliable fetcher of short read using SRA/ENA accession
# A reliable fetcher of short read using SRA/ENA accession

(Asked by: username_0 on 2018-05-10)

I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA (ERR2135453), not in SRA. And I found that my download script has its limitations.
My bash script looks like this (requires aspera ascp and fastq-dump) and I wanted to make changes that are commented now
#!/bin/bash

ACCESION=$1
OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh

# if [ ${ACCESION:0:3} == SRA ]; then
    URL=anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
# else
#    URL=ena_url
# fi    

ascp -QT -i $OPENSSH $URL

fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
rm data/"$ACCESION".sra

The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession (anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453. Where 003 there comes from? Also not all reads are in err subdirectory. 
Is there an easy way how to download reads from ENA knowing just the accession number?
--- edit ---
I found on the relevant ENA site about downloading data using aspera that ftp urls should be possible to derive only from accession also in ENA.

Submitted read data files are organised by submission accession number
  under vol1/ directory in ftp.sra.ebi.ac.uk:
ftp://ftp.sra.ebi.ac.uk/vol1//
where  contains the first 6 letters and
  numbers of the SRA Submission accession.
For example, the files submitted in the SRA Submission ERA007448 are
  available at: ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/.

The reason why there is this strange 003 will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).
However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?

**Tags:** data-download, sratoolkit

---

## 2 answers

### Answer by username_0 on 2018-05-11. Score: 1 | Accepted answer

Apparently there is no **100% reliable** solution, but in the most cases, you should be able to use the [enaBrowserTools](https://github.com/enasequence/enaBrowserTools) python scripts (enaDataGet) to quite reliably download data.

The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.

This answer is based on kind assistance of Suran from ebi helpdesk.

> **Comments:**
> **username_1**: Did you reach them through the email?  
> **username_2**: Do, they are usually very responsive and helpful.  

---

### Answer by username_3 on 2018-05-11. Score: 1

Here are two python functions I use for getting stuff from ENA based on an accession:

import os
import itertools
from urllib.request import urlopen

def fetch_ENA_files(accession):
    """Get the names of the files matching the ENA accession"""

    # Get the paths to the files
    url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&result=read_run&fields=fastq_ftp" \
          % accession

    try:
        paths = urlopen(url).readlines()[1:]
    except:
        E.debug("couldn't access %s" % url)
        raise

    paths = list(itertools.chain.from_iterable(
        [p.strip().split(";") for p in paths]))
    filenames = [os.path.basename(x) for x in paths]
    dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
    return filenames, dl_paths

def fetch_ENA(dl_path, outdir, protocol="ascp"):
    """Fetch fastq from ENA given accession"""

    if protocol == "ascp":
        statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                       era-fasp@fasp.sra.ebi.ac.uk:/%(dl_path)s %(outdir)s """ % locals()
    elif protocol == "http":
        fn = os.path.basename(dl_path)
        outfile = os.path.join(outdir, fn)
        statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()

    return statement

fetch_ENA_files queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found. 
fetch_ENA(dl_path, outdir, protocol) takes a download path and downloads it to outdir using with ftp or ascp.
You might use it like:
filenames, paths = fetch_ENA_files("ERR2135453")
for dlp in paths:
    bash_statement = fetch_ENA(dlp, "downloads", "ascp")
    os.system(bash_statement)

(we don't we actaully use our modules to send it to the cluster)

> **Comments:**
> **username_4**: I had to decode the paths from bytes to strings to get this to work. It doesn't actually work for the accession in question so I guess that's further evidence that there's no way to reliably fetch data sets that are too recent to be in SRA  
> **username_3**: Sorry, should have pointed out that this is python 2 not 3, hence the encoding problems.  

---

