<p>You should go with the CSI index as it seems like the BAI indexes cannot handle chromosomes longer than 2^29.</p>

<p>Your problem is that you're trying to merge the index files using <code>vcf-merge</code>. You should merge the VCF files, not the <code>.csi</code> index files.</p>

CC BY-SA 4.0
2020-05-14T08:55:30.327Z
Do you have a reference for this limitation? Is it mentioned in the docs somewhere?
terdon
CC BY-SA 4.0
2020-05-14T17:39:49.787Z
Yeah, I found it in a GitHub issue for the mosdepth/samtools repos: https://github.com/brentp/mosdepth/issues/41 https://github.com/samtools/samtools/issues/241
Ram RS
CC BY-SA 4.0
2020-05-14T18:14:02.323Z
This has come up on the mailing lists and forums often enough over the years to be part of the common lore. It's spelt out in the BAI spec in [SAMv1.pdf](http://samtools.github.io/hts-specs/SAMv1.pdf) §5.1.1 and will soon be mentioned in the `samtools index` manual page ([PR #1248](https://github.com/samtools/samtools/pull/1248)).
John Marshall
CC BY-SA 4.0
2020-05-14T23:59:45.620Z
Thanks for the advice, and sorry for the cross-post. You're right, CSI must be used if indexing is necessary since all of my chromosomes are larger than 2^29.
Brian
CC BY-SA 4.0
2020-05-13T23:08:08.410Z
2020-05-13T23:08:08.410Z
Ram RS
<p>I'm trying to use vcf-merge to combine 2 exome capture vcf files (~250K and ~330K in size) before trying it on all 96 samples. I'd appreciate any advice on the best way to do that! I've detailed what I've tried below. My issue seems to be with using tabix to convert the files to .tbi format.</p>
<p>Step 1: BGZIP</p>
<p>So far, I've zipped the files without issue:</p>
<pre><code>bgzip sample1.vcf
bgzip sample2.vcf
</code></pre>
<p>which produces:</p>
<pre><code>sample1.vcf.gz
sample2.vcf.gz
</code></pre>
<p>Step 2: TABIX</p>
<p>When I try to use this command:</p>
<pre><code>tabix -h -p vcf sample1.vcf.gz
</code></pre>
<p>the stderr is:</p>
<blockquote>
<p>Region 536999277..536999278 cannot be stored in a tbi index. Try using
a csi index with min_shift = 14, n_lvls &gt;= 6. tbx_index_build failed:
sample1.vcf.gz</p>
</blockquote>
<p>Using the -C option which works:</p>
<pre><code>tabix -C -h -p vcf sample1.vcf.gz
tabix -C -h -p vcf sample2.vcf.gz
</code></pre>
<p>which produces:
sample1.vcf.gz.csi
sample2.vcf.gz.csi</p>
<p>Step 3: VCF-MERGE</p>
<p>When I use this command:</p>
<pre><code>vcf-merge sample1.vcf.gz.csi sample2.vcf.gz.csi &gt; out.vcf.gz
</code></pre>
<p>the merge fails, and I get this stderr:</p>
<blockquote>
<p>Broken VCF header, no column names? at /usr/share/perl5/Vcf.pm line
172, &lt;<strong>ANONIO</strong>&gt; line 1. Vcf::throw(Vcf4_2=HASH(0x5645a760bc38),
&quot;Broken VCF header, no column names?&quot;) called at
/usr/share/perl5/Vcf.pm line 867
VcfReader::_read_column_names(Vcf4_2=HASH(0x5645a760bc38)) called at
/usr/share/perl5/Vcf.pm line 602
VcfReader::parse_header(Vcf4_2=HASH(0x5645a760bc38)) called at
/usr/bin/vcf-merge line 183 main::init_cols(HASH(0x5645a761f438),
Vcf4_2=HASH(0x5645a760b248)) called at /usr/bin/vcf-merge line 279
main::merge_vcf_files(HASH(0x5645a761f438)) called at
/usr/bin/vcf-merge line 12</p>
</blockquote>
<p>If I exclude the tabix files, the merge still fails and the stderr says:</p>
<blockquote>
<p>The column names not tab-separated? Could not load .tbi index of
sample1.vcf.gz. The command exited with an error. Is the file tabix
indexed?</p>
</blockquote>
<p>Any suggestions on how I can correctly merge these files?</p>

CC BY-SA 4.0
2020-05-13T23:28:03.333Z
Cross-posted on Biostars: https://www.biostars.org/p/437957/
Ram RS
CC BY-SA 4.0
2020-05-14T08:55:57.097Z
Would it be possible to share the files with us? Do you really have a chromosome that's 536999278 bases long? What species are you working on?
terdon
CC BY-SA 4.0
2020-05-14T23:55:44.067Z
Hi, I'm working with conifers which have very long chromosomes. As discusses below, the tbi cutoff is ~536,870,912 bp, so csi is needed.
Brian
<vcf><linux><vcftools><tabix>
VCF-merge fails due to tabix not producing .tbi files
# VCF-merge fails due to tabix not producing .tbi files

(Asked by: username_0 on 2020-05-13)

I'm trying to use vcf-merge to combine 2 exome capture vcf files (~250K and ~330K in size) before trying it on all 96 samples. I'd appreciate any advice on the best way to do that! I've detailed what I've tried below. My issue seems to be with using tabix to convert the files to .tbi format.
Step 1: BGZIP
So far, I've zipped the files without issue:
bgzip sample1.vcf
bgzip sample2.vcf

which produces:
sample1.vcf.gz
sample2.vcf.gz

Step 2: TABIX
When I try to use this command:
tabix -h -p vcf sample1.vcf.gz

the stderr is:

Region 536999277..536999278 cannot be stored in a tbi index. Try using
a csi index with min_shift = 14, n_lvls >= 6. tbx_index_build failed:
sample1.vcf.gz

Using the -C option which works:
tabix -C -h -p vcf sample1.vcf.gz
tabix -C -h -p vcf sample2.vcf.gz

which produces:
sample1.vcf.gz.csi
sample2.vcf.gz.csi
Step 3: VCF-MERGE
When I use this command:
vcf-merge sample1.vcf.gz.csi sample2.vcf.gz.csi > out.vcf.gz

the merge fails, and I get this stderr:

Broken VCF header, no column names? at /usr/share/perl5/Vcf.pm line
172, <ANONIO> line 1. Vcf::throw(Vcf4_2=HASH(0x5645a760bc38),
"Broken VCF header, no column names?") called at
/usr/share/perl5/Vcf.pm line 867
VcfReader::_read_column_names(Vcf4_2=HASH(0x5645a760bc38)) called at
/usr/share/perl5/Vcf.pm line 602
VcfReader::parse_header(Vcf4_2=HASH(0x5645a760bc38)) called at
/usr/bin/vcf-merge line 183 main::init_cols(HASH(0x5645a761f438),
Vcf4_2=HASH(0x5645a760b248)) called at /usr/bin/vcf-merge line 279
main::merge_vcf_files(HASH(0x5645a761f438)) called at
/usr/bin/vcf-merge line 12

If I exclude the tabix files, the merge still fails and the stderr says:

The column names not tab-separated? Could not load .tbi index of
sample1.vcf.gz. The command exited with an error. Is the file tabix
indexed?

Any suggestions on how I can correctly merge these files?

**Tags:** vcf, linux, vcftools, tabix
> **Comments:**
> **username_1**: Cross-posted on Biostars: https://www.biostars.org/p/437957/  
> **username_2**: Would it be possible to share the files with us? Do you really have a chromosome that's 536999278 bases long? What species are you working on?  
> **username_0**: Hi, I'm working with conifers which have very long chromosomes. As discusses below, the tbi cutoff is ~536,870,912 bp, so csi is needed.  

---

## 1 answers

### Answer by username_1 on 2020-05-13. Score: 2

You should go with the CSI index as it seems like the BAI indexes cannot handle chromosomes longer than 2^29.

Your problem is that you're trying to merge the index files using `vcf-merge`. You should merge the VCF files, not the `.csi` index files.

> **Comments:**
> **username_2**: Do you have a reference for this limitation? Is it mentioned in the docs somewhere?  
> **username_1**: Yeah, I found it in a GitHub issue for the mosdepth/samtools repos: https://github.com/brentp/mosdepth/issues/41 https://github.com/samtools/samtools/issues/241  
> **username_3**: This has come up on the mailing lists and forums often enough over the years to be part of the common lore. It's spelt out in the BAI spec in [SAMv1.pdf](http://samtools.github.io/hts-specs/SAMv1.pdf) §5.1.1 and will soon be mentioned in the `samtools index` manual page ([PR #1248](https://github.com/samtools/samtools/pull/1248)).  
> **username_0**: Thanks for the advice, and sorry for the cross-post. You're right, CSI must be used if indexing is necessary since all of my chromosomes are larger than 2^29.  

---

