<p>The ID history converter is available as a <a href="https://github.com/Ensembl/ensembl-tools/tree/release/103/scripts/id_history_converter" rel="nofollow noreferrer">Perl script</a>.</p>
<p>This script accesses these <a href="http://www.ensembl.org/info/docs/api/core/core_schema.html#gene_archive" rel="nofollow noreferrer">MySQL tables</a>, so you could query these tables directly.</p>
<p>Note that the mapping session is only one release at a time, so you would need to run 25 queries to go from release 75 to 100.</p>

CC BY-SA 4.0
2021-03-22T14:38:04.890Z
Thank you for your answer. To use the perl script, I wonder if there is any tutorial to refer to. I am particularly interested in knowing how to "tell" the script to convert from a given specific release to other specific release e.g. from 75 to 76.
user345394
CC BY-SA 4.0
2021-03-22T14:57:50.050Z
It just gives you the whole history, you would need to parse out the releases you want.
Emily_Ensembl
CC BY-SA 4.0
2021-12-25T20:17:13.593Z
`IDmapper.pl` does the job, but I wish there was an easy way to install its dependency called `Bio::EnsEMBL::Registry`. I feel that perhaps a wrapper of sorts, for example in the Anaconda system, would be immensely helpful. (Please somebody make it happen! :D )
user345394
CC BY-SA 4.0
2021-03-22T13:51:44.683Z
2021-03-22T13:58:59.217Z
2021-03-22T13:58:59.217Z
Emily_Ensembl
<p>Here's one way to do it programmatically using the biomaRt package (your potential solution #1):</p>
<pre><code>library(biomaRt)
library(dplyr)

mart37 &lt;- useMart(biomart = &quot;ENSEMBL_MART_ENSEMBL&quot;,
                  host    = &quot;grch37.ensembl.org&quot;,
                  path    = &quot;/biomart/martservice&quot;,
                  dataset = &quot;hsapiens_gene_ensembl&quot;)

mart38 &lt;- useMart(biomart = &quot;ENSEMBL_MART_ENSEMBL&quot;,
                  path    = &quot;/biomart/martservice&quot;,
                  dataset = &quot;hsapiens_gene_ensembl&quot;)

#pick all genes on chr22 from build 37 as a test dataset::
out37 &lt;- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                              'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                  filters = c('chromosome_name'),
                  values = list(&quot;22&quot;),
                  mart = mart37) %&gt;% #empty string CCDS ID's
  filter(ccds != &quot;&quot;)

#NOTE:: to check which filters can be used to query mart37, run this command:
#View(listFilters(mart37))

out38 &lt;- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                              'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                        filters = c('ccds'),
                        values = list(out37$ccds),
                        mart = mart38)
</code></pre>
<p>What this does is query the build 38 DB via the CCDS ID's of build 37. You can include other variables in this <code>out38</code> dataframe by simply adding to the attributes argument above. And in order to see all of the potential variables that can be included, you just have to run <code>View(listFilters(mart37))</code> or <code>View(listFilters(mart38))</code>, depending on which build you are interested in.</p>
<p>Does this solve your problem?</p>

CC BY-SA 4.0
2021-03-25T22:04:02.017Z
your solution could indeed be useful. I am just curious which ensembl release numbers are used when `mart37` and `mart38` were built. Is there a way to specify that?
user345394
CC BY-SA 4.0
2021-03-26T13:10:34.870Z
`mart37` corresponds to the GRCh37 release, while `mart38` corresponds to GRCh38. This is specified in the `host=` argument in the code above. `mart38` is the default, and so is not specified. Does this answer your question?
h3ab74
CC BY-SA 4.0
2021-03-26T13:56:09.347Z
P.S. if you're referring to the version number of the ensembl release, you can obtain this by adding `'ensembl_gene_id_version'` to the `attributes` array in the code above.
h3ab74
CC BY-SA 4.0
2021-03-26T22:53:22.967Z
By Ensembl release number I meant [version numbers of the assemblies maintained by ensembl](https://www.ensembl.info/category/01-release/), e.g. release 75 corresponds to GRCh37 for human genome.
user345394
CC BY-SA 4.0
2021-03-26T22:57:28.020Z
`ensembl_gene_id_version` could be useful if it is possible to know to which release that belongs to. e.g. how would I know if `x.3` id belongs to genome release 90 or 91? To get this information, I would have to run 'Ensembl IDmapper' which I would like to avoid, because of the issue I mentioned in my question.
user345394
CC BY-SA 4.0
2021-03-25T16:15:02.477Z
2021-03-25T16:15:02.477Z
h3ab74
<p>3 solutions that I had to unfortunately rule out.</p>
<p><strong>Potential solution #1</strong></p>
<p>As far as I know, the conventional way is to do this is by using CCDS ids (e.g. using biomart). However, the issue is that, as I see on the <a href="https://www.ncbi.nlm.nih.gov/projects/CCDS/CcdsBrowse.cgi?REQUEST=SHOW_STATISTICS" rel="nofollow noreferrer">CCDS project page</a>, the last release of CCDS ids was made almost 3 years ago (!) and it seems to be compatible with Ensembl release 92. Then how should one convert ids to the latest Ensembl release e.g. 100?</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>CCDS Release</th>
<th>Species</th>
<th>NCBI Annotation Release</th>
<th>Ensembl Annotation Release</th>
<th>Assembly Name</th>
<th>Assembly ID</th>
<th>Made Public</th>
</tr>
</thead>
<tbody>
<tr>
<td>22</td>
<td>Homo sapiens</td>
<td>109</td>
<td>92</td>
<td>GRCh38.p12</td>
<td>GCF_000001405.38</td>
<td>06/14/2018</td>
</tr>
</tbody>
</table>
</div>
<p><strong>Potential solution #2</strong></p>
<p>I am also aware of Ensembl's <a href="http://useast.ensembl.org/Homo_sapiens/Tools/IDMapper" rel="nofollow noreferrer">ID History Converter</a>. However, I am looking for a programmatic approach. Also, I noticed that if I try to convert ~&gt;20K ids with ID History Converter, in my experience, there is always some issue. e.g. an error I got when I tried to convert ~70K ensembl transcript ids.
<a href="https://i.sstatic.net/9Ie1t.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9Ie1t.png" alt="enter image description here" /></a></p>
<p>As a side note, I wonder why this ID conversion functionality is not available via Ensembl's REST API. To me, id conversion seems to be a basic enough task that almost every bioinformatician has to do. So it would be immensely helpful to be someday able to do this programmatically using Ensembl's mighty a REST API.</p>
<p><strong>Potential solution #3</strong></p>
<p>Finally, as another potential solution, over at Ensembl's FTP website, located within the directory of the release to which I am interested in converting my ids to (release 100, target release), I saw that according to a <a href="http://ftp.ensembl.org/pub/release-100/README" rel="nofollow noreferrer">README file</a>, &quot;ensembl_archive_&quot; directory should contain &quot;Data on historical Ensembl releases&quot;. So, I wondered if this directory contains a map between ids from different releases. However, without any extra info about column names, contents, file formats etc, I couldn't really figure out how to use those files. So, I had to rule this one out too.</p>
<pre><code>Index of /pub/release-100/mysql/ensembl_archive_100/
../
CHECKSUMS                                          26-Mar-2020 14:31                 132
ens_release.txt.gz                                 26-Mar-2020 14:31                1098
ensembl_archive_100.sql.gz                         26-Mar-2020 14:31                 943
release_species.txt.gz                             26-Mar-2020 14:31               39620
species.txt.gz                                     26-Mar-2020 14:31                6669
</code></pre>
<p>Of note, I saw a similar &quot;mysql&quot; directory over at the directory of the release whose ids I want to convert (source release): <code>http://ftp.ensembl.org/pub/grch37/release-100/mysql/homo_sapiens_core_100_37/ </code>. However, again, I do not understand its contents without any extra information.</p>
<p>So, finally, I am not sure what is the best way to programmatically convert Ensembl ids from between different releases. As a test case scenario, I want to convert all the protein_coding transcript ids from GRCh37 (release 75) to GRCh38 (release 100). If anybody has any suggestions about this, please kindly let me know.</p>

<gene><ensembl><identifiers><conversion>
What is the best way to programmatically convert Ensembl ids from one release to other? e.g. transcript ids from release 75 to 100
# What is the best way to programmatically convert Ensembl ids from one release to other? e.g. transcript ids from release 75 to 100

(Asked by: username_0 on 2021-03-19)

3 solutions that I had to unfortunately rule out.
Potential solution #1
As far as I know, the conventional way is to do this is by using CCDS ids (e.g. using biomart). However, the issue is that, as I see on the CCDS project page, the last release of CCDS ids was made almost 3 years ago (!) and it seems to be compatible with Ensembl release 92. Then how should one convert ids to the latest Ensembl release e.g. 100?




CCDS Release
Species
NCBI Annotation Release
Ensembl Annotation Release
Assembly Name
Assembly ID
Made Public




22
Homo sapiens
109
92
GRCh38.p12
GCF_000001405.38
06/14/2018




Potential solution #2
I am also aware of Ensembl's ID History Converter. However, I am looking for a programmatic approach. Also, I noticed that if I try to convert ~>20K ids with ID History Converter, in my experience, there is always some issue. e.g. an error I got when I tried to convert ~70K ensembl transcript ids.

As a side note, I wonder why this ID conversion functionality is not available via Ensembl's REST API. To me, id conversion seems to be a basic enough task that almost every bioinformatician has to do. So it would be immensely helpful to be someday able to do this programmatically using Ensembl's mighty a REST API.
Potential solution #3
Finally, as another potential solution, over at Ensembl's FTP website, located within the directory of the release to which I am interested in converting my ids to (release 100, target release), I saw that according to a README file, "ensembl_archive_" directory should contain "Data on historical Ensembl releases". So, I wondered if this directory contains a map between ids from different releases. However, without any extra info about column names, contents, file formats etc, I couldn't really figure out how to use those files. So, I had to rule this one out too.
Index of /pub/release-100/mysql/ensembl_archive_100/
../
CHECKSUMS                                          26-Mar-2020 14:31                 132
ens_release.txt.gz                                 26-Mar-2020 14:31                1098
ensembl_archive_100.sql.gz                         26-Mar-2020 14:31                 943
release_species.txt.gz                             26-Mar-2020 14:31               39620
species.txt.gz                                     26-Mar-2020 14:31                6669

Of note, I saw a similar "mysql" directory over at the directory of the release whose ids I want to convert (source release): http://ftp.ensembl.org/pub/grch37/release-100/mysql/homo_sapiens_core_100_37/ . However, again, I do not understand its contents without any extra information.
So, finally, I am not sure what is the best way to programmatically convert Ensembl ids from between different releases. As a test case scenario, I want to convert all the protein_coding transcript ids from GRCh37 (release 75) to GRCh38 (release 100). If anybody has any suggestions about this, please kindly let me know.

**Tags:** gene, ensembl, identifiers, conversion

---

## 2 answers

### Answer by username_1 on 2021-03-22. Score: 1

The ID history converter is available as a [Perl script](https://github.com/Ensembl/ensembl-tools/tree/release/103/scripts/id_history_converter).

This script accesses these [MySQL tables](http://www.ensembl.org/info/docs/api/core/core_schema.html#gene_archive), so you could query these tables directly.

Note that the mapping session is only one release at a time, so you would need to run 25 queries to go from release 75 to 100.

> **Comments:**
> **username_0**: Thank you for your answer. To use the perl script, I wonder if there is any tutorial to refer to. I am particularly interested in knowing how to "tell" the script to convert from a given specific release to other specific release e.g. from 75 to 76.  
> **username_1**: It just gives you the whole history, you would need to parse out the releases you want.  
> **username_0**: `IDmapper.pl` does the job, but I wish there was an easy way to install its dependency called `Bio::EnsEMBL::Registry`. I feel that perhaps a wrapper of sorts, for example in the Anaconda system, would be immensely helpful. (Please somebody make it happen! :D )  

---

### Answer by username_2 on 2021-03-25. Score: 0

Here's one way to do it programmatically using the biomaRt package (your potential solution #1):
library(biomaRt)
library(dplyr)

mart37 <- useMart(biomart = "ENSEMBL_MART_ENSEMBL",
                  host    = "grch37.ensembl.org",
                  path    = "/biomart/martservice",
                  dataset = "hsapiens_gene_ensembl")

mart38 <- useMart(biomart = "ENSEMBL_MART_ENSEMBL",
                  path    = "/biomart/martservice",
                  dataset = "hsapiens_gene_ensembl")

#pick all genes on chr22 from build 37 as a test dataset::
out37 <- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                              'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                  filters = c('chromosome_name'),
                  values = list("22"),
                  mart = mart37) %>% #empty string CCDS ID's
  filter(ccds != "")

#NOTE:: to check which filters can be used to query mart37, run this command:
#View(listFilters(mart37))

out38 <- getBM(attributes = c('hgnc_symbol', 'chromosome_name',
                              'ensembl_gene_id', 'ensembl_transcript_id', 'ccds'),
                        filters = c('ccds'),
                        values = list(out37$ccds),
                        mart = mart38)

What this does is query the build 38 DB via the CCDS ID's of build 37. You can include other variables in this out38 dataframe by simply adding to the attributes argument above. And in order to see all of the potential variables that can be included, you just have to run View(listFilters(mart37)) or View(listFilters(mart38)), depending on which build you are interested in.
Does this solve your problem?

> **Comments:**
> **username_0**: your solution could indeed be useful. I am just curious which ensembl release numbers are used when `mart37` and `mart38` were built. Is there a way to specify that?  
> **username_2**: `mart37` corresponds to the GRCh37 release, while `mart38` corresponds to GRCh38. This is specified in the `host=` argument in the code above. `mart38` is the default, and so is not specified. Does this answer your question?  
> **username_2**: P.S. if you're referring to the version number of the ensembl release, you can obtain this by adding `'ensembl_gene_id_version'` to the `attributes` array in the code above.  
> **username_0**: By Ensembl release number I meant [version numbers of the assemblies maintained by ensembl](https://www.ensembl.info/category/01-release/), e.g. release 75 corresponds to GRCh37 for human genome.  
> **username_0**: `ensembl_gene_id_version` could be useful if it is possible to know to which release that belongs to. e.g. how would I know if `x.3` id belongs to genome release 90 or 91? To get this information, I would have to run 'Ensembl IDmapper' which I would like to avoid, because of the issue I mentioned in my question.  

---

