<p>My solution is simply.</p>
<ol>
<li>perform a web-based Blastp using the following parameters:</li>
</ol>
<ul>
<li>most dissimilar</li>
<li>5000 hits (maximum limit)</li>
<li>Restrict to Archaea (which is what you samples appear to be)*</li>
</ul>
<p>If you hit the limit of 5000 results, I would repeat the search for each class or phylum within Archaea. I can't remember if it was re-classified as a kingdom or was just another phylum. Concatenate all hits into a single fasta file.</p>
<ol start="2">
<li><p>Reg-ex your extremophiles database (personally I'd use pandas), so that the species name of each bacterium is recovered no other information is present. <em>Or</em> this is an easily accessible value within a hash/database (this is a better solution).</p>
</li>
<li><p>Use Biopython to loop through your blast fasta file and filter using <code>re</code> search against the species from your database. If you get a hit save that sequence.</p>
</li>
</ol>
<p>Ideally, the point 2 and 3 would be a pandas merge operation, but looping is easier.</p>
<p>*, If there are extremophiles which are not Archaea (I don't know much about them), then take each general taxonomic group and repeat the operation for the search criteria &quot;restrict the blastp to ... <em>specific group</em>&quot;.</p>
<hr />
<p><strong>Issues</strong> The issues with command line blastp, not via the web, is that access is greatly restricted during US working hours. If you are searching more than 100 hits and carrying out the search during the working day US time - they will not let it through. 100 hits or less is fine however. For &gt;100 hits (e.g. 5000) it's got to be done well outside US working hours/ weekends. Web-based blastp doesn't carry this restriction of access during US working hours. Basically, the API route is deprioritised for the web-based approach. I assume this is to stop bots - albeit of course a bot can automate through NCBI's webpage.</p>
<p><strong>Note</strong> The original solution by @terdon might easily work as a remote blastp because each query is restricted to &lt;100 - so it will always get through during working hours. What I am saying is if you're using Blast API it needs a carefully defined strategy, else shift to the web for blanket approaches.</p>
<p>Entrez operations from a commandline do work ok during US working day time - albeit it's got to be coded to deal with &quot;random&quot; server refusals otherwise it will break the code (sporadic queries are refused). This is just my recent experience nothing more.</p>
<p>Just to be clear - this is potentially a medium sized project. Once the strategy has been decided then specific code that supports this approach can be given in separate questions.</p>

CC BY-SA 4.0
2023-10-14T11:35:57.520Z
I deleted my answer because I couldn't get the entrez query to work well. Too many of the species names were problematic.
terdon
CC BY-SA 4.0
2023-10-14T12:21:10.263Z
@M__ your comment on the Issues is revelation to me as in the past I tried to BLAST  >100 proteins at the same time. And sometime it worked other time it did not. Now it makes perfect sense. I have also submitted a ticket to BLAST team, if they offer another solution I will share.
Supertech
CC BY-SA 4.0
2023-10-14T13:40:25.837Z
NCBI Entrez not working @terdon ... I've encountered bizarre **non-universal** foreign "link" key issues (I assume they're using SQL). In bacteria this is terrible: there were separate data and link structures present for one protein database and not the other, particularly how they linked to nucleotides ... it's disjointed. As a user I had to write two separate solutions in the code (I wouldn't mind but they never tell you either, got to figure it out). Whats happened is separate teams worked on "biologically the same" problem. The result is a *committee meeting*.
M__
CC BY-SA 4.0
2023-10-14T13:45:07.530Z
@Supertech ticket to blast support I tried this too. I hope you get better success than I had.  You said "[my] comments ... revelation", thanks, but upvotes and "accepts" are much more welcome than "revelations".
M__
CC BY-SA 4.0
2023-10-14T11:01:20.963Z
2023-10-14T13:37:29.497Z
2023-10-14T13:37:29.497Z
M__
<p>In the interface of <a href="https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastp&amp;PAGE_TYPE=BlastSearch&amp;LINK_LOC=blasthome" rel="nofollow noreferrer">blastp</a>, you can limit the search space to organisms you want to. Yo can do this by introducing name species or Tax IDs, you can add more than one, but I bet you don't wanna introduce the 1,500 one by one.</p>
<p>The only other way to do this, is by using blastp in its <a href="https://blast.ncbi.nlm.nih.gov/doc/blast-help/downloadblastdata.html" rel="nofollow noreferrer">command-line version</a>. You would need to build a custom database to be used as the searching space, and then run blast on your computer or server using this database. Bulding it is a pain in the ass, as far as I know. You need first to download all the proteins in all organisms in fasta format and then use this compilation as a database (makeblastdb command). I am not aware of a way to download all proteins from various organism given a list of taxonomic IDs, but if any, it has to be NCBI, Uniprot or similar. Then you can use blastp locally together with your custom database.</p>
<p>But why limiting the search space is such a way? You are biasing your results to what you want to get, instead of finding the stuff that actually is closest to your protein of interest. My suggestion is to use the online version of blastp with one of the various databases available and rather than pre-selecting organisms, you can exclude the ones from which you definitely don't want to get results from, and you can also exclude entire families/genera such as primates (taxid:9443), to name an example.</p>

CC BY-SA 4.0
2023-10-13T16:04:29.183Z
There are many valid reasons why you would want to limit the results. For instance, if you are looking for the protein's homologs in particular taxa, you don't care about what is closest.
terdon
CC BY-SA 4.0
2023-10-14T00:11:04.500Z
If you have noticed, database I want to use contains only (hyper)thermophilic organisms. That is exactly what I want.
Supertech
CC BY-SA 4.0
2023-10-14T00:12:43.433Z
There should be a way of downloading genomes based on the TaxIDs. I will send a ticket to the NCBI support team. In the past they have been very responsive. Maybe a feature they want to add if is not already possible.
Supertech
CC BY-SA 4.0
2023-10-14T08:48:29.507Z
@terdon, True, there are such cases or others. I still would suggest to first try to play with the "add/remove organism" boxes in the interface and then parse the blast output in order to search for the organisms of interest.
JRodrigoF
CC BY-SA 4.0
2023-10-13T15:58:22.000Z
2023-10-14T08:43:38.267Z
2023-10-14T08:43:38.267Z
JRodrigoF
<p>I have a list of organisms with their full scientific names and Taxonomy IDs, total ~1500. I want to BLAST my protein against this 1500 genome or proteome. How can I do that?</p>
<p>Here a sample protein that I want to search the following database:
<a href="http://togodb.org/db/thermobase" rel="nofollow noreferrer">http://togodb.org/db/thermobase</a>
Please note that database can be downloades as CSV file. I am very comfortable using web tools or programmatic tools such as Perl, Python, bash, or Entrez Programming Utilities.</p>
<pre><code>&gt;QCC44558.1 chitinase [Halobacterium salinarum] TaxID 641522631
MPHDRRSYLRTSSAVIASLLAASTPTSAADTPPEWDPDTVYTDGDKATFDGYVWEAKWWTKGDKPGADEW
GPWTQRRPVDGSPGGPTAAFTVTDTTVEPGTAVTVDASDTTGSVDSYEWAFGDGTTAAGVTASHTYDAAG
EYTIELTVTTGDGTTDTATQQITATTSPGDDEFKVVGYYPSWKGTDDYDFYPADVPFDQVTDVLYAFLDV
QPDGTVVLPDSDVDHESLLASFASLKQDRAADTRLKLSIGGWGLSPGFEDAAADQASRERFATTAVDLMR
TYDFDGIDVDWEHPGPRRGKCECGSAQGPANHVALLETVRDRLNDAEVEDGRTYDLSVANGGSDWNAALI
NHREVAAVVDDIYMMAYDFTGVWHGTAGLNAPIYGTPPDYPPSGDAQQYTLETTLTIWKEQGYWVDWMEW
EDHGDPVDDPGTLVLGMPFYGRGCNVENGIWDTFSLSEWQQGDPKYQNDVIPPGTWNDLRGPDDANTGAF
DYGDLAANYEGADAWTKQRNEQGGVPYLWNDSEGVFISYDDPTSIAAKVELAVAEDLGGVMIWELSQDYD
GTLLETINQHTP
</code></pre>

CC BY-SA 4.0
2023-10-13T15:36:02.017Z
@M__ Yes you are assessment is correct. I will consider breaking it down. I guess I have to build a local BLAST database first using TaxIDs. I will give try and maybe delete my question.
Supertech
CC BY-SA 4.0
2023-10-13T15:56:29.747Z
I will propose a different solution over the weekend. It's a lot easier than performing a local blast. My strong advice is to take the easy route. Whilst @terdon and I have done this style of calculation - there's much easier ways. This is because for major repeat projects - this level of coding is worth while - for what appears a one-off blast its not worth it. Also interogating NCBI via its link system is never fun (it's not standardised, i.e. thier databases don't use universal foreign keys). Basically solutions around remote blast only - there's a lot of caveats here and I'll describe them
M__
<blast><ncbi><blastp><makeblastdb><e-utilities>
How to BLAST a protein against a large set of organisms?
# How to BLAST a protein against a large set of organisms?

(Asked by: username_0 on 2023-10-13)

I have a list of organisms with their full scientific names and Taxonomy IDs, total ~1500. I want to BLAST my protein against this 1500 genome or proteome. How can I do that?
Here a sample protein that I want to search the following database:
http://togodb.org/db/thermobase
Please note that database can be downloades as CSV file. I am very comfortable using web tools or programmatic tools such as Perl, Python, bash, or Entrez Programming Utilities.
>QCC44558.1 chitinase [Halobacterium salinarum] TaxID 641522631
MPHDRRSYLRTSSAVIASLLAASTPTSAADTPPEWDPDTVYTDGDKATFDGYVWEAKWWTKGDKPGADEW
GPWTQRRPVDGSPGGPTAAFTVTDTTVEPGTAVTVDASDTTGSVDSYEWAFGDGTTAAGVTASHTYDAAG
EYTIELTVTTGDGTTDTATQQITATTSPGDDEFKVVGYYPSWKGTDDYDFYPADVPFDQVTDVLYAFLDV
QPDGTVVLPDSDVDHESLLASFASLKQDRAADTRLKLSIGGWGLSPGFEDAAADQASRERFATTAVDLMR
TYDFDGIDVDWEHPGPRRGKCECGSAQGPANHVALLETVRDRLNDAEVEDGRTYDLSVANGGSDWNAALI
NHREVAAVVDDIYMMAYDFTGVWHGTAGLNAPIYGTPPDYPPSGDAQQYTLETTLTIWKEQGYWVDWMEW
EDHGDPVDDPGTLVLGMPFYGRGCNVENGIWDTFSLSEWQQGDPKYQNDVIPPGTWNDLRGPDDANTGAF
DYGDLAANYEGADAWTKQRNEQGGVPYLWNDSEGVFISYDDPTSIAAKVELAVAEDLGGVMIWELSQDYD
GTLLETINQHTP

**Tags:** blast, ncbi, blastp, makeblastdb, e-utilities
> **Comments:**
> **username_0**: @M__ Yes you are assessment is correct. I will consider breaking it down. I guess I have to build a local BLAST database first using TaxIDs. I will give try and maybe delete my question.  
> **username_1**: I will propose a different solution over the weekend. It's a lot easier than performing a local blast. My strong advice is to take the easy route. Whilst @terdon and I have done this style of calculation - there's much easier ways. This is because for major repeat projects - this level of coding is worth while - for what appears a one-off blast its not worth it. Also interogating NCBI via its link system is never fun (it's not standardised, i.e. thier databases don't use universal foreign keys). Basically solutions around remote blast only - there's a lot of caveats here and I'll describe them  

---

## 2 answers

### Answer by username_1 on 2023-10-14. Score: 1 | Accepted answer

My solution is simply.

1. perform a web-based Blastp using the following parameters:

* most dissimilar
* 5000 hits (maximum limit)
* Restrict to Archaea (which is what you samples appear to be)\*

If you hit the limit of 5000 results, I would repeat the search for each class or phylum within Archaea. I can't remember if it was re-classified as a kingdom or was just another phylum. Concatenate all hits into a single fasta file.

2. Reg-ex your extremophiles database (personally I'd use pandas), so that the species name of each bacterium is recovered no other information is present. *Or* this is an easily accessible value within a hash/database (this is a better solution).
3. Use Biopython to loop through your blast fasta file and filter using `re` search against the species from your database. If you get a hit save that sequence.

Ideally, the point 2 and 3 would be a pandas merge operation, but looping is easier.

\*, If there are extremophiles which are not Archaea (I don't know much about them), then take each general taxonomic group and repeat the operation for the search criteria "restrict the blastp to ... *specific group*".

---

**Issues** The issues with command line blastp, not via the web, is that access is greatly restricted during US working hours. If you are searching more than 100 hits and carrying out the search during the working day US time - they will not let it through. 100 hits or less is fine however. For >100 hits (e.g. 5000) it's got to be done well outside US working hours/ weekends. Web-based blastp doesn't carry this restriction of access during US working hours. Basically, the API route is deprioritised for the web-based approach. I assume this is to stop bots - albeit of course a bot can automate through NCBI's webpage.

**Note** The original solution by @terdon might easily work as a remote blastp because each query is restricted to <100 - so it will always get through during working hours. What I am saying is if you're using Blast API it needs a carefully defined strategy, else shift to the web for blanket approaches.

Entrez operations from a commandline do work ok during US working day time - albeit it's got to be coded to deal with "random" server refusals otherwise it will break the code (sporadic queries are refused). This is just my recent experience nothing more.

Just to be clear - this is potentially a medium sized project. Once the strategy has been decided then specific code that supports this approach can be given in separate questions.

> **Comments:**
> **username_2**: I deleted my answer because I couldn't get the entrez query to work well. Too many of the species names were problematic.  
> **username_0**: @M__ your comment on the Issues is revelation to me as in the past I tried to BLAST  >100 proteins at the same time. And sometime it worked other time it did not. Now it makes perfect sense. I have also submitted a ticket to BLAST team, if they offer another solution I will share.  
> **username_1**: NCBI Entrez not working @terdon ... I've encountered bizarre **non-universal** foreign "link" key issues (I assume they're using SQL). In bacteria this is terrible: there were separate data and link structures present for one protein database and not the other, particularly how they linked to nucleotides ... it's disjointed. As a user I had to write two separate solutions in the code (I wouldn't mind but they never tell you either, got to figure it out). Whats happened is separate teams worked on "biologically the same" problem. The result is a *committee meeting*.  
> **username_1**: @Supertech ticket to blast support I tried this too. I hope you get better success than I had.  You said "[my] comments ... revelation", thanks, but upvotes and "accepts" are much more welcome than "revelations".  

---

### Answer by username_3 on 2023-10-13. Score: 1

In the interface of [blastp](https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastp&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome), you can limit the search space to organisms you want to. Yo can do this by introducing name species or Tax IDs, you can add more than one, but I bet you don't wanna introduce the 1,500 one by one.

The only other way to do this, is by using blastp in its [command-line version](https://blast.ncbi.nlm.nih.gov/doc/blast-help/downloadblastdata.html). You would need to build a custom database to be used as the searching space, and then run blast on your computer or server using this database. Bulding it is a pain in the ass, as far as I know. You need first to download all the proteins in all organisms in fasta format and then use this compilation as a database (makeblastdb command). I am not aware of a way to download all proteins from various organism given a list of taxonomic IDs, but if any, it has to be NCBI, Uniprot or similar. Then you can use blastp locally together with your custom database.

But why limiting the search space is such a way? You are biasing your results to what you want to get, instead of finding the stuff that actually is closest to your protein of interest. My suggestion is to use the online version of blastp with one of the various databases available and rather than pre-selecting organisms, you can exclude the ones from which you definitely don't want to get results from, and you can also exclude entire families/genera such as primates (taxid:9443), to name an example.

> **Comments:**
> **username_2**: There are many valid reasons why you would want to limit the results. For instance, if you are looking for the protein's homologs in particular taxa, you don't care about what is closest.  
> **username_0**: If you have noticed, database I want to use contains only (hyper)thermophilic organisms. That is exactly what I want.  
> **username_0**: There should be a way of downloading genomes based on the TaxIDs. I will send a ticket to the NCBI support team. In the past they have been very responsive. Maybe a feature they want to add if is not already possible.  
> **username_3**: @terdon, True, there are such cases or others. I still would suggest to first try to play with the "add/remove organism" boxes in the interface and then parse the blast output in order to search for the organisms of interest.  

---

