<p>Using a <code>for</code> loop for this is <a href="https://mywiki.wooledge.org/ParsingLs" rel="nofollow noreferrer">by far the safer option</a>. However, you need to <a href="https://unix.stackexchange.com/a/131767/22222">quote your variables</a> and also make sure you're passing the right variable name. While you're at it, you can also remove the extension so that your filtered file name can be <code>foo_out.vcf</code> instead of <code>foo.vcf_out.vcf</code> (which is what your loop would create. Putting all that together:</p>

<pre><code>for f in *.vcf; do 
    echo "filtering <span class="math-container">$f"; 
bcftools filter -i 'QUAL &gt; 1000' -o filter/"$</span>{f//.vcf}"_out.vcf "$f"; done
fi
</code></pre>

CC BY-SA 4.0
2018-12-03T19:52:24.207Z
I like the "${f//.vcf}", thanks - my filenames have been growing unnecessarily long!
Joanne
CC BY-SA 4.0
2018-12-03T16:12:51.863Z
2018-12-03T16:12:51.863Z
terdon
<p>For one-liners, <code>xargs</code> is often more convenient than the <code>for</code> loop:</p>

<pre class="lang-shell prettyprint-override"><code>ls *.vcf | sed s,.vcf$,, \
  | xargs -i echo 'bcftools filter -i "QUAL &gt; 1000" -o "filter/{}_out.vcf" "{}.vcf"' \
  | parallel
</code></pre>

<p>Note that your command line (with @WouterDeCoster's fix) writes output to <code>foo.vcf_out.vcf</code>. My command line writes to <code>foo_out.vcf</code>.</p>

CC BY-SA 4.0
2018-12-03T16:17:40.440Z
Xargs is indeed convenient, but parsing `ls` is very fragile and will break on any non-standard file name. Why not `for f in *vcf; do printf 'bcftools filter -i "QUAL > 1000" -o filter/"%s" "%s"\n' "${f//.vcf}_out.vcf" "$f"; done | parallel`? That seems as simple as `xargs` but can deal with arbitrary file names.
terdon
CC BY-SA 4.0
2018-12-03T17:19:39.100Z
@terdon I don't see that is an issue. You can simply add quotation marks.
user172818
CC BY-SA 4.0
2018-12-03T17:27:49.653Z
Yes, that will protect from spaces, but not from newlines. And OK, having newlines in filenames isn't something that happens very often but it is possible. I freely admit, however, that I may just be nitpicking because I'm too used to hanging out on [unix.se] where this sort of thing is considered very important and any example of parsing `ls` is met with horror. Still, since the shell loop is safe and painless, it just seems better to use that instead.
terdon
CC BY-SA 4.0
2018-12-02T02:56:35.130Z
2018-12-03T17:18:56.133Z
2018-12-03T17:18:56.133Z
user172818
<p>I assume your shell is looking for the variable <code>$f_out</code>, but can't find it because you did not define it. It probably generated a hidden file with the name <code>.vcf</code> in your output directory.</p>

<p>Instead, use the following: <code>-o <span class="math-container">${f}_out.vcf</code>. This will "protect/limit" the bash variable to just <code>$</span>f</code> rather than <code>$f_out</code>.</p>

<p>When encountering issues like this it's worth putting <code>echo</code> in your loop right before your command, in this case bcftools (adding quotes whenever necessary). That would show you the command as how it is going to be executed and would show you, in this case, that the output name is incorrect.</p>

CC BY-SA 4.0
2018-12-03T17:44:09.363Z
This works thanks! But I wonder why I've had a similar loop working fine with `$f.SnpSift.vcf` as my output? Does it understand the `.` differently than the `_` after `$f`?
Joanne
CC BY-SA 4.0
2018-12-03T19:14:24.803Z
Yes that's right, and easy to test using a dummy variable and `echo`
Wouter De Coster
CC BY-SA 4.0
2018-12-06T09:14:55.877Z
Alternatively, instead of (or in addition to) `echo`-ing you can use `set -u` which will give an error on undefined variables (this assumes you use Bash or Zsh as your shell (not sure about others)).
ph0t0nix
CC BY-SA 4.0
2018-12-01T09:57:56.710Z
2018-12-01T09:57:56.710Z
Wouter De Coster
<p>Probably a silly oversight on my part, but I'm trying to filter all the vcfs in a directory with bcftools using a simple loop. My basic command is working fine:</p>

<pre><code>bcftools filter -i 'QUAL &gt; 1000' -o filter/file1out.vcf file1in.vcf 
</code></pre>

<p>but when I try to loop it, it echoes through each filename, but generates no output files or error messages.</p>

<pre><code>for f in *.vcf; do echo "filtering <span class="math-container">$f"; bcftools filter -i 'QUAL &gt; 1000' -o filter/$</span>f_out.vcf $f; done
</code></pre>

<p>Am I not handling the output direction properly? Any help much appreciated!</p>

<vcf><bcftools>
bcftools filtering all files in a directory
# bcftools filtering all files in a directory

(Asked by: username_0 on 2018-11-30)

Probably a silly oversight on my part, but I'm trying to filter all the vcfs in a directory with bcftools using a simple loop. My basic command is working fine:
bcftools filter -i 'QUAL > 1000' -o filter/file1out.vcf file1in.vcf 

but when I try to loop it, it echoes through each filename, but generates no output files or error messages.
for f in *.vcf; do echo "filtering $f"; bcftools filter -i 'QUAL > 1000' -o filter/$f_out.vcf $f; done

Am I not handling the output direction properly? Any help much appreciated!

**Tags:** vcf, bcftools

---

## 3 answers

### Answer by username_3 on 2018-12-01. Score: 5 | Accepted answer

I assume your shell is looking for the variable `$f_out`, but can't find it because you did not define it. It probably generated a hidden file with the name `.vcf` in your output directory.

Instead, use the following: `-o ${f}_out.vcf`. This will "protect/limit" the bash variable to just `$f` rather than `$f_out`.

When encountering issues like this it's worth putting `echo` in your loop right before your command, in this case bcftools (adding quotes whenever necessary). That would show you the command as how it is going to be executed and would show you, in this case, that the output name is incorrect.

> **Comments:**
> **username_0**: This works thanks! But I wonder why I've had a similar loop working fine with `$f.SnpSift.vcf` as my output? Does it understand the `.` differently than the `_` after `$f`?  
> **username_3**: Yes that's right, and easy to test using a dummy variable and `echo`  
> **username_4**: Alternatively, instead of (or in addition to) `echo`-ing you can use `set -u` which will give an error on undefined variables (this assumes you use Bash or Zsh as your shell (not sure about others)).  

---

### Answer by username_1 on 2018-12-03. Score: 2

Using a for loop for this is by far the safer option. However, you need to quote your variables and also make sure you're passing the right variable name. While you're at it, you can also remove the extension so that your filtered file name can be foo_out.vcf instead of foo.vcf_out.vcf (which is what your loop would create. Putting all that together:
for f in *.vcf; do 
    echo "filtering $f"; 
bcftools filter -i 'QUAL > 1000' -o filter/"${f//.vcf}"_out.vcf "$f"; done
fi

> **Comments:**
> **username_0**: I like the "${f//.vcf}", thanks - my filenames have been growing unnecessarily long!  

---

### Answer by username_2 on 2018-12-02. Score: 1

For one-liners, xargs is often more convenient than the for loop:
ls *.vcf | sed s,.vcf$,, \
  | xargs -i echo 'bcftools filter -i "QUAL > 1000" -o "filter/{}_out.vcf" "{}.vcf"' \
  | parallel

Note that your command line (with @WouterDeCoster's fix) writes output to foo.vcf_out.vcf. My command line writes to foo_out.vcf.

> **Comments:**
> **username_1**: Xargs is indeed convenient, but parsing `ls` is very fragile and will break on any non-standard file name. Why not `for f in *vcf; do printf 'bcftools filter -i "QUAL > 1000" -o filter/"%s" "%s"\n' "${f//.vcf}_out.vcf" "$f"; done | parallel`? That seems as simple as `xargs` but can deal with arbitrary file names.  
> **username_2**: @terdon I don't see that is an issue. You can simply add quotation marks.  
> **username_1**: Yes, that will protect from spaces, but not from newlines. And OK, having newlines in filenames isn't something that happens very often but it is possible. I freely admit, however, that I may just be nitpicking because I'm too used to hanging out on [unix.se] where this sort of thing is considered very important and any example of parsing `ls` is met with horror. Still, since the shell loop is safe and painless, it just seems better to use that instead.  

---

