<p>The main problem is that your script is inadvertently treating a set of file and directory names as a single SRA accession number, all in one variable.  You should update it to work with each accession separately and make sure they really are accessions.</p>
<hr />
<p>On this line you're taking all the names of things in your working directory (minus the script), each on a separate line, and trimming .sra off the end of each line:</p>
<pre><code>sraID=$(ls -1 | grep -v run.sh | sed 's/\.sra$//')
</code></pre>
<p>In your case that list was apparently just a directory named &quot;sra&quot;.  (Even though I don't have that directory on my end, you can see it in your output, from the first of two <code>ls *</code> at the end, where you have <code>sra/SRR12332899.sra</code>.)  So later on it tries to <code>fastq-dump sra</code>, which gives you that &quot;cannot be opened as database or table&quot; error.  (<code>vdb-validate</code> can take a directory as input, so it doesn't complain.)  If I run your script from scratch in an empty directory with SRR12332899, it actually seems to run OK, aside from removing everything it creates at the end.  But if you run it with a second accession you see the problem, since it then tries things like <code>vdb-validate &quot;several accessions and filenames separated by newlines&quot;</code>.</p>
<p>I think a proper fix would involve a bunch of changes, but here are some tips, at least:</p>
<ul>
<li>use subdirectories to keep things organized so you don't have to juggle multiple types of file in one place</li>
<li><code>set -x</code> at the start of a script will show you each command as it happens, which can be very helpful for troubleshooting</li>
<li>be careful with quotes and whitespace in bash.  If you want to loop over each word in a variable you can do <code>for thing in $var; do ...; done</code> (unquoted) but if you run into filenames with whitespace, it breaks them up.  Another pattern is <code>command | while read thing; do ...; done</code> which will go line-by-line.  (In that case if you were working with a variable storing the command output, you might do something like <code>echo &quot;$output&quot; | while...</code>, quoted that time, since you'd want to preserve the newlines.)</li>
<li><code>ls *</code> will expand each file/directory name as separate arguments (so bash really ends up calling <code>ls &quot;sra&quot; ...whatever else...</code>), and when ls sees directory names is lists their contents too.  If you just want to show all the files/directories you can leave off the asterisk altogether, or if you wanted to give directory names but not have it list their contents, you can use <code>ls -d</code>.</li>
<li>you don't necessarily have to call <code>ls</code> to store the names as a block of text.  You can loop over each filename, including those with spaces, directly in bash (e.g. <code>for thing in *; do echo &quot;path: '$thing'&quot;; done</code>)</li>
<li>be careful with the rm command, especially combined with wildcards, and especially with <code>-rf</code>.  If <code>sraID</code> happened to be empty somehow, think of what <code>rm -rf &quot;$sraID&quot;*</code> would actually mean.  (And yes, scripts can remove themselves!)</li>
</ul>

CC BY-SA 4.0
2023-08-06T16:31:57.907Z
will work on your suggestion and show you what i have done
PesKchan
CC BY-SA 4.0
2023-08-05T21:40:09.380Z
2023-08-05T21:40:09.380Z
Jesse
<p>I have a very basic objective which I want to give list of id to a ncbi-srafastq tool kit which will prefetch the id which is in .sra file extension and then it will run fastq-dump on the file to split the files into fastq files, single file if there is single end file or paired end file if its paired end or two set of files.</p>
<p>This is my code</p>
<pre><code>#!/bin/bash

input=$1

a=<span class="math-container">$(basename "$</span>input&quot;)
id=${a/.id/}

# Download SRA data
prefetch &quot;$id&quot;

# Extract SRA ID from the downloaded data
#sraID=<span class="math-container">$(ls -1 | grep -v run.sh)
sraID=$</span>(ls -1 | grep -v run.sh | sed 's/\.sra<span class="math-container">$//')
# Validate the downloaded SRA data
vdb-validate "$</span>sraID&quot;

# Dump fastq data from the SRA file
fastq-dump &quot;$sraID&quot;  --gzip

# Concatenate fastq files for read1, read2, and unpaired reads (if available)
for s in &quot;<span class="math-container">$sraID"
do
  echo "$</span>s&quot;
  if [ &quot;<span class="math-container">$s" != "$</span>id&quot; ]
  then
    [ -f &quot;<span class="math-container">${s}_1.fastq" ] &amp;&amp; cat "$</span>{s}_1.fastq&quot; &gt;&gt; &quot;<span class="math-container">${id}_R1.fastq"
    [ -f "$</span>{s}_2.fastq&quot; ] &amp;&amp; cat &quot;<span class="math-container">${s}_2.fastq" &gt;&gt; "$</span>{id}_R2.fastq&quot;
    [ -f &quot;<span class="math-container">${s}.fastq" ] &amp;&amp; cat "$</span>{s}.fastq&quot; &gt;&gt; &quot;${id}.fastq&quot;
  fi
done

# Compress fastq files using pigz
#for i in &quot;<span class="math-container">${id}"*.fastq
#do
  #echo "$</span>i&quot;
 # pigz &quot;$i&quot; &amp; sleep 1
#done
#wait

# Remove the original uncompressed fastq file if both read1 and read2 files are present
[ -f &quot;<span class="math-container">${id}_R1.fastq.gz" ] &amp;&amp; [ -f "$</span>{id}_R2.fastq.gz&quot; ] &amp;&amp; [ -f &quot;<span class="math-container">${id}.fastq.gz" ] &amp;&amp; rm "$</span>{id}.fastq.gz&quot;

# Test deleting intermediate folder
echo &quot;Space before tmp folder delete&quot;
echo &quot;Removing tmp folder <span class="math-container">$sraID "
ls *
rm -rf "$</span>sraID&quot;*
ls *
echo 'done'
</code></pre>
<p>I run using as small example file as such</p>
<pre><code>./run.sh SRR12332899
</code></pre>
<p>The error I get is this</p>
<pre><code>2023-08-05T12:37:32 prefetch.3.0.6: Current preference is set to retrieve SRA Normalized Format files with full base quality scores.
2023-08-05T12:37:33 prefetch.3.0.6: 1) Downloading 'SRR12332899'...
2023-08-05T12:37:33 prefetch.3.0.6: SRA Normalized Format file is being retrieved, if this is different from your preference, it may be due to current file availability.
2023-08-05T12:37:33 prefetch.3.0.6:  Downloading via HTTPS...
2023-08-05T12:40:48 prefetch.3.0.6:  HTTPS download succeed
2023-08-05T12:40:48 prefetch.3.0.6:  'SRR12332899' is valid
2023-08-05T12:40:48 prefetch.3.0.6: 1) 'SRR12332899' was downloaded successfully
2023-08-05T12:40:48 prefetch.3.0.6: 'SRR12332899' has 0 unresolved dependencies
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' metadata: md5 ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Table 'SEQUENCE' metadata: md5 ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'ALTREAD': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'QUALITY': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'READ': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'SPOT_GROUP': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'sra/SRR12332899.sra' contains only unaligned reads
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' is consistent
2023-08-05T12:40:49 fastq-dump.3.0.6 err: item not found while constructing within virtual database module - the path 'sra' cannot be opened as database or table
fastq-dump quit with error code 3
sra
Space before tmp folder delete
Removing tmp folder sra
run.sh

sra:
SRR12332899.sra
run.sh
done
</code></pre>
<p>If I do just</p>
<p><code>prefetch SRR12332899</code> and then <code>fastq-dump SRR12332899</code></p>
<p>I get a SRR12332899.fastq  SRR12332899.sra</p>
<p>This is for one file since I want to download list of files using the script which is not working.</p>
<p>Any suggestion would be really helpful</p>

CC BY-SA 4.0
2023-08-07T11:09:54.333Z
Note that you are catting an already gzipped files since you used --gzip in fastq-dump, so the suffix of the output must be .gz

It would be muchfaster to simply search for the dataset at sra-explorer.info and get fastq download links right away.
ATpoint
<fastq><shell><ncbi>
ncbi fastq dump error in loop
# ncbi fastq dump error in loop

(Asked by: username_0 on 2023-08-05)

I have a very basic objective which I want to give list of id to a ncbi-srafastq tool kit which will prefetch the id which is in .sra file extension and then it will run fastq-dump on the file to split the files into fastq files, single file if there is single end file or paired end file if its paired end or two set of files.
This is my code
#!/bin/bash

input=$1

a=$(basename "$input")
id=${a/.id/}

# Download SRA data
prefetch "$id"

# Extract SRA ID from the downloaded data
#sraID=$(ls -1 | grep -v run.sh)
sraID=$(ls -1 | grep -v run.sh | sed 's/\.sra$//')
# Validate the downloaded SRA data
vdb-validate "$sraID"

# Dump fastq data from the SRA file
fastq-dump "$sraID"  --gzip

# Concatenate fastq files for read1, read2, and unpaired reads (if available)
for s in "$sraID"
do
  echo "$s"
  if [ "$s" != "$id" ]
  then
    [ -f "${s}_1.fastq" ] && cat "${s}_1.fastq" >> "${id}_R1.fastq"
    [ -f "${s}_2.fastq" ] && cat "${s}_2.fastq" >> "${id}_R2.fastq"
    [ -f "${s}.fastq" ] && cat "${s}.fastq" >> "${id}.fastq"
  fi
done

# Compress fastq files using pigz
#for i in "${id}"*.fastq
#do
  #echo "$i"
 # pigz "$i" & sleep 1
#done
#wait

# Remove the original uncompressed fastq file if both read1 and read2 files are present
[ -f "${id}_R1.fastq.gz" ] && [ -f "${id}_R2.fastq.gz" ] && [ -f "${id}.fastq.gz" ] && rm "${id}.fastq.gz"

# Test deleting intermediate folder
echo "Space before tmp folder delete"
echo "Removing tmp folder $sraID "
ls *
rm -rf "$sraID"*
ls *
echo 'done'

I run using as small example file as such
./run.sh SRR12332899

The error I get is this
2023-08-05T12:37:32 prefetch.3.0.6: Current preference is set to retrieve SRA Normalized Format files with full base quality scores.
2023-08-05T12:37:33 prefetch.3.0.6: 1) Downloading 'SRR12332899'...
2023-08-05T12:37:33 prefetch.3.0.6: SRA Normalized Format file is being retrieved, if this is different from your preference, it may be due to current file availability.
2023-08-05T12:37:33 prefetch.3.0.6:  Downloading via HTTPS...
2023-08-05T12:40:48 prefetch.3.0.6:  HTTPS download succeed
2023-08-05T12:40:48 prefetch.3.0.6:  'SRR12332899' is valid
2023-08-05T12:40:48 prefetch.3.0.6: 1) 'SRR12332899' was downloaded successfully
2023-08-05T12:40:48 prefetch.3.0.6: 'SRR12332899' has 0 unresolved dependencies
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' metadata: md5 ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Table 'SEQUENCE' metadata: md5 ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'ALTREAD': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'QUALITY': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'READ': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Column 'SPOT_GROUP': checksums ok
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'sra/SRR12332899.sra' contains only unaligned reads
2023-08-05T12:40:48 vdb-validate.3.0.6 info: Database 'SRR12332899.sra' is consistent
2023-08-05T12:40:49 fastq-dump.3.0.6 err: item not found while constructing within virtual database module - the path 'sra' cannot be opened as database or table
fastq-dump quit with error code 3
sra
Space before tmp folder delete
Removing tmp folder sra
run.sh

sra:
SRR12332899.sra
run.sh
done

If I do just
prefetch SRR12332899 and then fastq-dump SRR12332899
I get a SRR12332899.fastq  SRR12332899.sra
This is for one file since I want to download list of files using the script which is not working.
Any suggestion would be really helpful

**Tags:** fastq, shell, ncbi
> **Comments:**
> **username_1**: Note that you are catting an already gzipped files since you used --gzip in fastq-dump, so the suffix of the output must be .gz  It would be muchfaster to simply search for the dataset at sra-explorer.info and get fastq download links right away.  

---

## 1 answers

### Answer by username_2 on 2023-08-05. Score: 3 | Accepted answer

The main problem is that your script is inadvertently treating a set of file and directory names as a single SRA accession number, all in one variable.  You should update it to work with each accession separately and make sure they really are accessions.

On this line you're taking all the names of things in your working directory (minus the script), each on a separate line, and trimming .sra off the end of each line:
sraID=$(ls -1 | grep -v run.sh | sed 's/\.sra$//')

In your case that list was apparently just a directory named "sra".  (Even though I don't have that directory on my end, you can see it in your output, from the first of two ls * at the end, where you have sra/SRR12332899.sra.)  So later on it tries to fastq-dump sra, which gives you that "cannot be opened as database or table" error.  (vdb-validate can take a directory as input, so it doesn't complain.)  If I run your script from scratch in an empty directory with SRR12332899, it actually seems to run OK, aside from removing everything it creates at the end.  But if you run it with a second accession you see the problem, since it then tries things like vdb-validate "several accessions and filenames separated by newlines".
I think a proper fix would involve a bunch of changes, but here are some tips, at least:

use subdirectories to keep things organized so you don't have to juggle multiple types of file in one place
set -x at the start of a script will show you each command as it happens, which can be very helpful for troubleshooting
be careful with quotes and whitespace in bash.  If you want to loop over each word in a variable you can do for thing in $var; do ...; done (unquoted) but if you run into filenames with whitespace, it breaks them up.  Another pattern is command | while read thing; do ...; done which will go line-by-line.  (In that case if you were working with a variable storing the command output, you might do something like echo "$output" | while..., quoted that time, since you'd want to preserve the newlines.)
ls * will expand each file/directory name as separate arguments (so bash really ends up calling ls "sra" ...whatever else...), and when ls sees directory names is lists their contents too.  If you just want to show all the files/directories you can leave off the asterisk altogether, or if you wanted to give directory names but not have it list their contents, you can use ls -d.
you don't necessarily have to call ls to store the names as a block of text.  You can loop over each filename, including those with spaces, directly in bash (e.g. for thing in *; do echo "path: '$thing'"; done)
be careful with the rm command, especially combined with wildcards, and especially with -rf.  If sraID happened to be empty somehow, think of what rm -rf "$sraID"* would actually mean.  (And yes, scripts can remove themselves!)

> **Comments:**
> **username_0**: will work on your suggestion and show you what i have done  

---

