<p>BED format can refer to the &quot;.txt&quot; file format that stores chromosomes and positions of genomic features as well as some other things like name. <code>bed</code> format can also refer to the plink format, which belongs to bed, bim, fam trio. The plink bed format stores all the genetic variants, but the program needs to refer to bim file for their positions.</p>
<p>Genome liftover obviously only needs the coordination of each SNPs, and it does not need individual genotypes at all. In fact, the bed format in PLINK does not provide any useful information. Knowing this means that providing the plink bed file is wrong. And this is pretty clear from the error message.</p>
<p>Moreover, since the three plink file always needs to be used together, most programs refer to them as <strong>plink file format</strong>. So whenever people say BED format, it is the txt file format.</p>

CC BY-SA 4.0
2020-08-27T13:39:59.777Z
2020-08-27T13:39:59.777Z
Phoenix Mu
<p>I am having strange issues lifting this dataset to hg37 from hg38. I am using the HGDP WGS data from <a href="ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/" rel="nofollow noreferrer">ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/</a>. The data set split by chromosome, so I first converted them to BED then merged them to a single file using plink1.9. However, when trying to lift to hg37/19 using either <a href="http://hgdownload.cse.ucsc.edu/admin/exe/linux.x86_64/" rel="nofollow noreferrer">liftOver</a>  or <a href="https://crossmap.readthedocs.io/en/latest/#" rel="nofollow noreferrer">CrossMap v0.5.1</a> I get strange errors. I tried both the chain from <a href="ftp://ftp.ensembl.org/pub/assembly_mapping/homo_sapiens/" rel="nofollow noreferrer">ensemble.org</a> and <a href="http://hgdownload.cse.ucsc.edu/goldenPath/hg38/liftOver/" rel="nofollow noreferrer">from UCSC</a>, I tried them both compressed and uncompressed. I am starting to think I need to try the conversion on the VCF files but these are so large I would prefer not to duplicate them unnecessarily.</p>
<h1>Data</h1>
<p>The data looks fine. If we look at the BIM:</p>
<pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP<span class="math-container">$ head hgdp_hg38_chrpos.bim
1   1:10153 0   10153   G   A
1   1:10163 0   10163   C   T
1   1:10180 0   10180   C   T
1   1:10250 0   10250   C   A
1   1:10257 0   10257   C   A
1   1:10291 0   10291   T   C
1   1:10297 0   10297   T   C
1   1:10327 0   10327   C   T
1   1:10330 0   10330   A   C
1   1:10333 0   10333   T   C
user@desktop:/media/luks8tb1/data/genomics/HGDP$</span> tail hgdp_hg38_chrpos.bim
Y   Y:56887228  0   56887228    C   T
Y   Y:56887285  0   56887285    C   A
Y   Y:56887316  0   56887316    A   G
Y   Y:56887463  0   56887463    A   C
Y   Y:56887491  0   56887491    A   C
Y   Y:56887503  0   56887503    T   G
Y   Y:56887583  0   56887583    G   A
Y   Y:56887631  0   56887631    T   C
Y   Y:56887837  0   56887837    A   G
Y   Y:56887844  0   56887844    C   T
</code></pre>
<p>The data are in chrpos variant names here but I had the same result using the original rsid.</p>
<h1>Chains</h1>
<p>Chains look fine, and I am able to read them into R without problems:</p>
<pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ head GRCh38_to_GRCh37.chain
chain 1 1 248956422 + 10000 297968 1 249250621 + 10000 267719 2
167417 80249 50000
40302

chain 1 1 248956422 + 347968 501617 1 249250621 - 248779253 248932902 3
153649

chain 1 1 248956422 + 585988 12949384 1 249250621 + 521368 13009210 4
1044707 1 0
3709 3 0
user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail GRCh38_to_GRCh37.chain

chain 1 chrY 57227415 + 56821509 57217415 Y 59373566 + 58967656 59363566 11183
85168 0 1
78580 0 1
63496 0 1
3158 1 0
30382 0 1
54147 0 1
80974
</code></pre>
<h1>CrossMap</h1>
<pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ CrossMap.py bed GRCh38_to_GRCh37.chain hgdp_hg38_chrpos.bed hgdp_hg37_chrpos.bed
@ 2020-08-25 20:04:26: Read the chain file:  GRCh38_to_GRCh37.chain
Traceback (most recent call last):
  File &quot;/home/user/.local/bin/CrossMap.py&quot;, line 166, in &lt;module&gt;
    crossmap_bed_file(mapTree, in_file, out_file)
  File &quot;/home/user/.local/lib/python3.6/site-packages/cmmodule/mapbed.py&quot;, line 32, in crossmap_bed_file
    for line in ireader.reader(inbed):
  File &quot;/home/user/.local/lib/python3.6/site-packages/cmmodule/ireader.py&quot;, line 27, in reader
    yield l.decode('utf8').strip().replace(&quot;\r&quot;, &quot;&quot;)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 20: invalid continuation byte
</code></pre>
<h1>liftOver</h1>
<pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.bed hg38ToHg19.over.chain hgdp_hg37.bed hgdp_hg37_errors.bed
Reading liftover chains
Mapping coordinates
Data format error: expecting at least 3 fields in BED file (hgdp_hg38.bed)
</code></pre>
<h1>Trying the PED file with liftOver</h1>
<p>I converted BED file to PED with plink1.9:</p>
<pre><code>plink --bfile hgdp_hg38 --recode --out hgdp_hg38
</code></pre>
<p>Then:</p>
<pre><code>user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.ped hg38ToHg19.over.chain hgdp_hg37.ped hgdp_hg37_errors.ped
Reading liftover chains
Mapping coordinates
invalid unsigned integer: &quot;HGDP00001&quot;
</code></pre>
<p><a href="https://www.google.com/search?q=%22liftover%22%20%22invalid%20unsigned%20integer%22" rel="nofollow noreferrer">There's a few question on Google about this, but nothing that looks useful.</a></p>
<p>CrossMap does not support PED.</p>

<plink><liftover>
Lifting from hg38 to hg37
# Lifting from hg38 to hg37

(Asked by: username_0 on 2020-08-25)

I am having strange issues lifting this dataset to hg37 from hg38. I am using the HGDP WGS data from ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/. The data set split by chromosome, so I first converted them to BED then merged them to a single file using plink1.9. However, when trying to lift to hg37/19 using either liftOver  or CrossMap v0.5.1 I get strange errors. I tried both the chain from ensemble.org and from UCSC, I tried them both compressed and uncompressed. I am starting to think I need to try the conversion on the VCF files but these are so large I would prefer not to duplicate them unnecessarily.
Data
The data looks fine. If we look at the BIM:
user@desktop:/media/luks8tb1/data/genomics/HGDP$ head hgdp_hg38_chrpos.bim
1   1:10153 0   10153   G   A
1   1:10163 0   10163   C   T
1   1:10180 0   10180   C   T
1   1:10250 0   10250   C   A
1   1:10257 0   10257   C   A
1   1:10291 0   10291   T   C
1   1:10297 0   10297   T   C
1   1:10327 0   10327   C   T
1   1:10330 0   10330   A   C
1   1:10333 0   10333   T   C
user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail hgdp_hg38_chrpos.bim
Y   Y:56887228  0   56887228    C   T
Y   Y:56887285  0   56887285    C   A
Y   Y:56887316  0   56887316    A   G
Y   Y:56887463  0   56887463    A   C
Y   Y:56887491  0   56887491    A   C
Y   Y:56887503  0   56887503    T   G
Y   Y:56887583  0   56887583    G   A
Y   Y:56887631  0   56887631    T   C
Y   Y:56887837  0   56887837    A   G
Y   Y:56887844  0   56887844    C   T

The data are in chrpos variant names here but I had the same result using the original rsid.
Chains
Chains look fine, and I am able to read them into R without problems:
user@desktop:/media/luks8tb1/data/genomics/HGDP$ head GRCh38_to_GRCh37.chain
chain 1 1 248956422 + 10000 297968 1 249250621 + 10000 267719 2
167417 80249 50000
40302

chain 1 1 248956422 + 347968 501617 1 249250621 - 248779253 248932902 3
153649

chain 1 1 248956422 + 585988 12949384 1 249250621 + 521368 13009210 4
1044707 1 0
3709 3 0
user@desktop:/media/luks8tb1/data/genomics/HGDP$ tail GRCh38_to_GRCh37.chain

chain 1 chrY 57227415 + 56821509 57217415 Y 59373566 + 58967656 59363566 11183
85168 0 1
78580 0 1
63496 0 1
3158 1 0
30382 0 1
54147 0 1
80974

CrossMap
user@desktop:/media/luks8tb1/data/genomics/HGDP$ CrossMap.py bed GRCh38_to_GRCh37.chain hgdp_hg38_chrpos.bed hgdp_hg37_chrpos.bed
@ 2020-08-25 20:04:26: Read the chain file:  GRCh38_to_GRCh37.chain
Traceback (most recent call last):
  File "/home/user/.local/bin/CrossMap.py", line 166, in 
    crossmap_bed_file(mapTree, in_file, out_file)
  File "/home/user/.local/lib/python3.6/site-packages/cmmodule/mapbed.py", line 32, in crossmap_bed_file
    for line in ireader.reader(inbed):
  File "/home/user/.local/lib/python3.6/site-packages/cmmodule/ireader.py", line 27, in reader
    yield l.decode('utf8').strip().replace("\r", "")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 20: invalid continuation byte

liftOver
user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.bed hg38ToHg19.over.chain hgdp_hg37.bed hgdp_hg37_errors.bed
Reading liftover chains
Mapping coordinates
Data format error: expecting at least 3 fields in BED file (hgdp_hg38.bed)

Trying the PED file with liftOver
I converted BED file to PED with plink1.9:
plink --bfile hgdp_hg38 --recode --out hgdp_hg38

Then:
user@desktop:/media/luks8tb1/data/genomics/HGDP$ liftOver hgdp_hg38.ped hg38ToHg19.over.chain hgdp_hg37.ped hgdp_hg37_errors.ped
Reading liftover chains
Mapping coordinates
invalid unsigned integer: "HGDP00001"

There's a few question on Google about this, but nothing that looks useful.
CrossMap does not support PED.

**Tags:** plink, liftover

---

## 1 answers

### Answer by username_1 on 2020-08-27. Score: 2 | Accepted answer

BED format can refer to the ".txt" file format that stores chromosomes and positions of genomic features as well as some other things like name. `bed` format can also refer to the plink format, which belongs to bed, bim, fam trio. The plink bed format stores all the genetic variants, but the program needs to refer to bim file for their positions.

Genome liftover obviously only needs the coordination of each SNPs, and it does not need individual genotypes at all. In fact, the bed format in PLINK does not provide any useful information. Knowing this means that providing the plink bed file is wrong. And this is pretty clear from the error message.

Moreover, since the three plink file always needs to be used together, most programs refer to them as **plink file format**. So whenever people say BED format, it is the txt file format.


---

