<p>We created this over the pandemic when we needed to merge loads of VCFs which all have the same header and first few columns (ie same variants)</p>
<p><a href="https://github.com/iqbal-lab-org/ivcfmerge" rel="nofollow noreferrer">https://github.com/iqbal-lab-org/ivcfmerge</a></p>
<p>That page gives detailed instructions but i'll paste a couple of examples
of how you can use it</p>
<p>== In Python
=4.1.1 If the number of input files is small (can be opened all at once)</p>
<pre><code>from contextlib import ExitStack
from ivcfmerge import ivcfmerge

filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to

with ExitStack() as stack:
    files = map(lambda fname: stack.enter_context(open(fname)), filenames)
    with open(output_path) as outfile:
        ivcfmerge(files, outfile)
</code></pre>
<p>=4.1.2 If the number of input files is big (cannot be opened all at once)
from ivcfmerge import ivcfmerge_batch</p>
<pre><code>filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to
batch_size = 1000    # How many files to open and merge at once

ivcfmerge_batch(filenames, output_path, batch_size)
</code></pre>
<p>Pretty sure we initially did batched bcftools merge, I can't honestly remember why we moved to this. Thankfully not having to do these merges very often</p>

CC BY-SA 4.0
2023-04-18T22:50:42.480Z
Thanks Zam. Unfortunately I fail the first two assumptions listed in the README.
Michael Hall
CC BY-SA 4.0
2023-04-18T08:04:23.903Z
2023-04-18T08:16:02.730Z
2023-04-18T08:16:02.730Z
Zamin Iqbal
<p>I ended up using an iterative approach (inspired by <a href="https://shicheng-guo.github.io/bioinformatics/1923/02/28/bcftools-merge" rel="nofollow noreferrer">this post</a>), which took 1.5 hours - so a massive speed up (and that was just the sequential version).</p>
<p>Note: <a href="https://github.com/sharkdp/fd" rel="nofollow noreferrer"><code>fd</code></a> is analogous to <code>find</code>, but way nicer to use.</p>
<p>First, merge subsets of <em>n</em> VCFs</p>
<pre><code># size of the subsets
n=200
# create file of filenames for each subset
fd -e vcf . path/to/vcfs/ | split -l $n --additional-suffix '.fofn' - subset_vcfs
</code></pre>
<p><code>-e vcf</code> tells <code>fd</code> to find all files with a <code>.vcf</code> extension, you can obviously change this to <code>.bcf</code>/<code>.vcf.gz</code> etc. if necessary. <code>split</code> splits the list of all VCFs files into subsets of size <code>-l</code> (200) and creates the file of filenames with the prefix <code>subset_vcfs</code>. It will add a random suffix to this - e.g. <code>subset_vcfsav</code>. Plus the additional suffix <code>.fofn</code>. See the <a href="https://www.man7.org/linux/man-pages/man1/split.1.html" rel="nofollow noreferrer"><code>split</code> manual</a> for all options.</p>
<p>Now iterate through these subsets and merge the VCFs listed in them.</p>
<h2>Subset merge option 1 - sequential</h2>
<p>This just merges the VCFs one at a time</p>
<pre class="lang-bash prettyprint-override"><code>for sub in *.fofn;
do
    echo &quot;Merging subset ${sub}...&quot;
    out_vcf=&quot;merge.${sub}.bcf&quot;
    bcftools merge --file-list &quot;$sub&quot; -o &quot;$out_vcf&quot;
    bcftools index &quot;$out_vcf&quot;
done
</code></pre>
<h2>Subset merge option 2 - parallel</h2>
<p>You can easily parallelise the previous step using <code>fd</code> on the subset files. See <a href="https://mbh.sh/2020-06-22-cheap-parallelisation/" rel="nofollow noreferrer">this post</a> for more details. Or basic <a href="https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop">bash <code>for</code> loop parallelisation</a>.</p>
<p>Create a script (<code>merge_subset.sh</code>) of what we want to run on each subset</p>
<pre class="lang-bash prettyprint-override"><code>#!/usr/bin/env bash
sub=&quot;$1&quot;
echo &quot;Merging subset ${sub}...&quot;
out_vcf=&quot;merge.${sub}.bcf&quot;
bcftools merge --file-list &quot;$sub&quot; -o &quot;$out_vcf&quot;
bcftools index &quot;$out_vcf&quot;
</code></pre>
<p>Now we run this script with each subset file of filenames using <code>fd</code></p>
<pre><code># number of processes to use. remove this option to use all available
jobs=4
fd -j $jobs -e fofn -x bash merge_subset.sh '{}'
</code></pre>
<h2>Final merge</h2>
<p>Now we merge all of these VCFs into our final VCF</p>
<pre><code>ls merge.*.bcf &gt; merged.fofn
bcftools merge -o merged.bcf --file-list merged.fofn
</code></pre>

CC BY-SA 4.0
2023-04-19T01:52:02.690Z
2023-04-19T23:32:39.223Z
2023-04-19T23:32:39.223Z
Michael Hall
<p>Perform a tree based merge of the files.</p>
<ul>
<li>Sort the files into groups that can easily be merged in a measurable amount of time.</li>
<li>Merge each group into a separate Variant Call Format (VCF) file
<ul>
<li>Has the benefit that files can also be grouped by type / source / run</li>
<li>Further benefit that small merges can be checked to ensure you like the output before proceeding to the next stage.</li>
<li>Further benefit that there are also subgroup files available to be distributed or shared separately.</li>
<li>Downside, extra storage space until you have a final product.</li>
</ul>
</li>
<li>Merge subfiles upward in stages until you arrive at a complete 44,000 entry file
<ul>
<li>If you're happy with the earlier test results, process can be automated with relatively simple iterative shell scripts.</li>
<li>Possibly faster, as the bulk merge needs to load every single file into memory, and you may be hitting a capacity bottleneck.</li>
<li>Worked in Fluid Dynamics for years, and often ran into a similar issue.  Could run many small cases representing a large case faster than running the large case, because large case would overwhelm memory and transfer capacity.</li>
<li>In some clusters you may also be causing nodes to silently crash, which then stalls the entire process without a visible notification.</li>
</ul>
</li>
</ul>

CC BY-SA 4.0
2023-04-18T17:55:31.967Z
2023-04-18T17:55:31.967Z
G. Putnam
<p>Can anyone suggest a strategy for speeding up VCF merging?</p>
<p>I have ~44,000 single-sample VCFs that I am trying to merge into a multi-sample VCF with <code>bcftools merge</code>, but the job keeps timing out on the cluster (just timed out after 4 days). I could obviously try running it for a couple of weeks, but I was hoping there is a better strategy someone can suggest?</p>
<p>The exact command I am running is</p>
<pre><code>bcftools merge --no-version -m all -i 'DP:avg,RO:avg,AO:avg' -F + -o merged.vcf --file-list merge.fofn
</code></pre>
<p>Happy to try any other tool that will give the same merge results.</p>

<vcf><bcftools><merge>
Strategy for merging many VCFs
# Strategy for merging many VCFs

(Asked by: username_0 on 2023-04-18)

Can anyone suggest a strategy for speeding up VCF merging?
I have ~44,000 single-sample VCFs that I am trying to merge into a multi-sample VCF with bcftools merge, but the job keeps timing out on the cluster (just timed out after 4 days). I could obviously try running it for a couple of weeks, but I was hoping there is a better strategy someone can suggest?
The exact command I am running is
bcftools merge --no-version -m all -i 'DP:avg,RO:avg,AO:avg' -F + -o merged.vcf --file-list merge.fofn

Happy to try any other tool that will give the same merge results.

**Tags:** vcf, bcftools, merge

---

## 3 answers

### Answer by username_1 on 2023-04-18. Score: 4

We created this over the pandemic when we needed to merge loads of VCFs which all have the same header and first few columns (ie same variants)
https://github.com/iqbal-lab-org/ivcfmerge
That page gives detailed instructions but i'll paste a couple of examples
of how you can use it
== In Python
=4.1.1 If the number of input files is small (can be opened all at once)
from contextlib import ExitStack
from ivcfmerge import ivcfmerge

filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to

with ExitStack() as stack:
    files = map(lambda fname: stack.enter_context(open(fname)), filenames)
    with open(output_path) as outfile:
        ivcfmerge(files, outfile)

=4.1.2 If the number of input files is big (cannot be opened all at once)
from ivcfmerge import ivcfmerge_batch
filenames = [...]    # List/iterator of relative/absolute paths to input files
output_path = '...'  # Where to write the merged VCF to
batch_size = 1000    # How many files to open and merge at once

ivcfmerge_batch(filenames, output_path, batch_size)

Pretty sure we initially did batched bcftools merge, I can't honestly remember why we moved to this. Thankfully not having to do these merges very often

> **Comments:**
> **username_0**: Thanks Zam. Unfortunately I fail the first two assumptions listed in the README.  

---

### Answer by username_0 on 2023-04-19. Score: 3

I ended up using an iterative approach (inspired by this post), which took 1.5 hours - so a massive speed up (and that was just the sequential version).
Note: fd is analogous to find, but way nicer to use.
First, merge subsets of n VCFs
# size of the subsets
n=200
# create file of filenames for each subset
fd -e vcf . path/to/vcfs/ | split -l $n --additional-suffix '.fofn' - subset_vcfs

-e vcf tells fd to find all files with a .vcf extension, you can obviously change this to .bcf/.vcf.gz etc. if necessary. split splits the list of all VCFs files into subsets of size -l (200) and creates the file of filenames with the prefix subset_vcfs. It will add a random suffix to this - e.g. subset_vcfsav. Plus the additional suffix .fofn. See the split manual for all options.
Now iterate through these subsets and merge the VCFs listed in them.
Subset merge option 1 - sequential
This just merges the VCFs one at a time
for sub in *.fofn;
do
    echo "Merging subset ${sub}..."
    out_vcf="merge.${sub}.bcf"
    bcftools merge --file-list "$sub" -o "$out_vcf"
    bcftools index "$out_vcf"
done

Subset merge option 2 - parallel
You can easily parallelise the previous step using fd on the subset files. See this post for more details. Or basic bash for loop parallelisation.
Create a script (merge_subset.sh) of what we want to run on each subset
#!/usr/bin/env bash
sub="$1"
echo "Merging subset ${sub}..."
out_vcf="merge.${sub}.bcf"
bcftools merge --file-list "$sub" -o "$out_vcf"
bcftools index "$out_vcf"

Now we run this script with each subset file of filenames using fd
# number of processes to use. remove this option to use all available
jobs=4
fd -j $jobs -e fofn -x bash merge_subset.sh '{}'

Final merge
Now we merge all of these VCFs into our final VCF
ls merge.*.bcf > merged.fofn
bcftools merge -o merged.bcf --file-list merged.fofn


---

### Answer by username_2 on 2023-04-18. Score: 1

Perform a tree based merge of the files.

* Sort the files into groups that can easily be merged in a measurable amount of time.
* Merge each group into a separate Variant Call Format (VCF) file
  + Has the benefit that files can also be grouped by type / source / run
  + Further benefit that small merges can be checked to ensure you like the output before proceeding to the next stage.
  + Further benefit that there are also subgroup files available to be distributed or shared separately.
  + Downside, extra storage space until you have a final product.
* Merge subfiles upward in stages until you arrive at a complete 44,000 entry file
  + If you're happy with the earlier test results, process can be automated with relatively simple iterative shell scripts.
  + Possibly faster, as the bulk merge needs to load every single file into memory, and you may be hitting a capacity bottleneck.
  + Worked in Fluid Dynamics for years, and often ran into a similar issue. Could run many small cases representing a large case faster than running the large case, because large case would overwhelm memory and transfer capacity.
  + In some clusters you may also be causing nodes to silently crash, which then stalls the entire process without a visible notification.


---

