<p>If I understand correctly, your question boils down to: parsing a bunch of numbers in a row from a .csv file gives a different result from parsing a bunch of numbers as <em>rows</em> of a text file.  I that right?  I suggest that you look at the source of your text file.  For example, was the text file created on a Windows host (running bash, or in the command terminal?)?</p>

<p>Try using this modified version of the unix cat command:
<code>cat -et text_file.txt</code></p>

<p>The very end of each line will have a '\$' sign and that will show you if there are any extra non-visible characters in your rows.  For example, a DOS/Windows formatted file will usually have a ^M at the end of each row before the '$'.  If you don't strip off those line endings then they get added to your queries, even if you cannot see them.  You can strip off those linefeed (or are they carriage returns?) using a Perl one-liner like:
<code>perl -pe 's/\r//' badfile.txt &gt; goodfile.txt</code></p>

CC BY-SA 4.0
2018-08-02T00:46:21.807Z
Thanks, I suspect that between generating these files on my school's servers through bash to downloading them locally and manipulating them in Excel, that there must've been '$' characters like you said.  Since then, I've learned to do what I was doing in Excel on the command line and think I'll be able to avoid this altogether.
ljs
CC BY-SA 4.0
2018-07-27T03:54:22.750Z
2018-07-27T03:54:22.750Z
mdperry
<p>I've been searching TaxIDs against NCBI's Taxonomy DB to get taxonomic lineages for species.  I have successfully done this for 1,000's of TaxIDs that were returned to me by Blast+ blasts in a CSV.  (The first column in the CSV was the TaxID for the top hit of a sequence, and the other columns were for GI, Scientific Name, e-value, etc.).  I simply wrote a script that would take the first field, which is always the TaxID, and search it against Taxonomy to get lineages for the species those TaxIDs represented.  Everything worked fine!</p>

<p>However, some of the records returned from my original blast (that gave me the TaxIDs in the first place) would return two or more TaxIDs per record.  (I suspect because a sequence matched equally well with two or more different species?)  That's fine, but I had to remove these few exceptions before I searched the rest of the regular results against Taxonomy.  So I've got most of my data/lineages, but for about 1:10,0000 records that gave multiple TaxIDs, I would like to know the lineages of the multiple species that the blast suggested.</p>

<p>I now took all of those "more-than-one-taxid" hits, separated them out into columns in an excel sheet and saved it as a CSV, and am trying to search these against Taxonomy with essentially the same script I used for the rest of the data, but am getting an error for <em>every</em> taxID I search (see below).</p>

<p><strong>Complete Error</strong> (for each taxid searched against Taxonomy):</p>

<pre><code>400 Bad Request
&amp;rettype=full&amp;retmode=xml&amp;edirect_os=linux&amp;edirect=7.90&amp;tool=edirect&amp;email=x@xxx.localdomain'my&amp;id=464126
Result of do_post http request is
$VAR1 = bless( {
                 '_protocol' =&gt; 'HTTP/1.1',
                 '_content' =&gt; '&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
&lt;!DOCTYPE eEfetchResult PUBLIC "-//NLM//DTD efetch 20131226//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20131226/efetch.dtd"&gt;
&lt;eFetchResult&gt;
 &lt;ERROR&gt;WebEnv parameter is required&lt;/ERROR&gt;
&lt;/eFetchResult&gt;
',
                 '_rc' =&gt; 400,
                 '_headers' =&gt; bless( {
                                        'connection' =&gt; 'close',
                                        'set-cookie' =&gt; 'ncbi_sid=9FF573148F458940_CEC6SID; domain=.nih.gov; path=/; expires=Thu, 14 Mar 2019 22:36:03 GMT',
                                        'cache-control' =&gt; 'private',
                                        'access-control-allow-origin' =&gt; '*',
                                        'date' =&gt; 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'vary' =&gt; 'Accept-Encoding',
                                        'client-ssl-cert-issuer' =&gt; '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA',
                                        'client-ssl-cipher' =&gt; 'ECDHE-RSA-AES256-GCM-SHA384',
                                        'client-peer' =&gt; '130.14.29.110:443',
                                        'strict-transport-security' =&gt; 'max-age=31536000; includeSubDomains; preload',
                                        'client-date' =&gt; 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'content-type' =&gt; 'text/xml; charset=UTF-8',
                                        'x-xss-protection' =&gt; '1; mode=block',
                                        'client-transfer-encoding' =&gt; [
                                                                        'chunked'
                                                                      ],
                                        'server' =&gt; 'Finatra',
                                        'client-ssl-socket-class' =&gt; 'IO::Socket::SSL',
                                        'client-response-num' =&gt; 1,
                                        'ncbi-phid' =&gt; 'DAF6EB0134DF5AEC039F11D248910BE7.1',
                                        'ncbi-sid' =&gt; '9FF573148F458940_CEC6SID',
                                        '::std_case' =&gt; {
                                                          'set-cookie' =&gt; 'Set-Cookie',
                                                          'access-control-allow-origin' =&gt; 'Access-Control-Allow-Origin',
                                                          'client-ssl-cert-issuer' =&gt; 'Client-SSL-Cert-Issuer',
                                                          'client-ssl-cipher' =&gt; 'Client-SSL-Cipher',
                                                          'client-peer' =&gt; 'Client-Peer',
                                                          'strict-transport-security' =&gt; 'Strict-Transport-Security',
                                                          'client-date' =&gt; 'Client-Date',
                                                          'x-xss-protection' =&gt; 'X-XSS-Protection',
                                                          'client-transfer-encoding' =&gt; 'Client-Transfer-Encoding',
                                                          'client-ssl-socket-class' =&gt; 'Client-SSL-Socket-Class',
                                                          'client-response-num' =&gt; 'Client-Response-Num',
                                                          'ncbi-phid' =&gt; 'NCBI-PHID',
                                                          'ncbi-sid' =&gt; 'NCBI-SID',
                                                          'x-ua-compatible' =&gt; 'X-UA-Compatible',
                                                          'client-ssl-cert-subject' =&gt; 'Client-SSL-Cert-Subject',
                                                          'content-security-policy' =&gt; 'Content-Security-Policy'
                                                        },
                                        'x-ua-compatible' =&gt; 'IE=Edge',
                                        'client-ssl-cert-subject' =&gt; '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                        'content-security-policy' =&gt; 'upgrade-insecure-requests'
                                      }, 'HTTP::Headers' ),
                 '_msg' =&gt; 'Bad Request',
                 '_request' =&gt; bless( {
&amp;rettype=full&amp;retmode=xml&amp;edirect_os=linux&amp;edirect=7.90&amp;tool=edirect&amp;email=x@xxx.localdomain',
                                        '_uri' =&gt; bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                        '_headers' =&gt; bless( {
                                                               'user-agent' =&gt; 'libwww-perl/6.05',
                                                               'content-type' =&gt; 'application/x-www-form-urlencoded',
                                                               '::std_case' =&gt; {
                                                                                 'if-ssl-cert-subject' =&gt; 'If-SSL-Cert-Subject'
                                                                               }
                                                             }, 'HTTP::Headers' ),
                                        '_method' =&gt; 'POST',
                                        '_uri_canonical' =&gt; $VAR1-&gt;{'_request'}{'_uri'}
                                      }, 'HTTP::Request' )
               }, 'HTTP::Response' );
</code></pre>

<p><strong>What I've done:</strong></p>

<p>Here is a sample of the data I used that worked correctly and yielded lineages when searching the taxIDs (taxIDs in 1st column):</p>

<pre><code>549298,Contig1,Francisella halioticida,5.44e-126,gi|1213460577|gb|CP022132.1|,CP022132
240395,Contig2,Glandirana emeljanovi,4.74e-131,gi|1025812536|gb|KU641020.1|,KU641020
1069680,Contig3,Pneumocystis murina B123,2.34e-85,gi|1124120202|ref|XM_019757798.1|,XM_019757798
3871,Contig4,Lupinus angustifolius,1.89e-93,gi|1239725835|gb|CP023132.1|,CP023132
</code></pre>

<p>Here is a sample of the <strong>data that I am trying to get to work</strong> (when I search these manually against Taxonomy from their web page they seem to work and give lineages; it's just that I have just enough of them to make a non-programmatic solution really time consuming and tedious):</p>

<pre><code>258234
1316279
1316315
710996
711000
225338
985895
710996
1382731
77133
344338
</code></pre>

<p>This is the business end of <strong>the script that I used that worked for the  working data</strong>:</p>

<pre><code>#!/bin/bash
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do      
        #Firstly echo the Contig# (Query Seq. ID) followed by lineage information.
        echo -n "$f2," &gt;&gt; $2

        #Now get the lineage, pipe to xtract and parse XML for readability in output file
        efetch -db taxonomy \
        -id $f1 \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                ...
                -or Rank -equals species \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                &gt;&gt; $2
done &lt; "$1"
</code></pre>

<p>This is the slightly altered script that I am trying to use to make the not-working set of data yield results (and that produces the error):</p>

<pre><code>#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        efetch -db taxonomy \
        -id $line \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                ...
                -or Rank -equals species \
                -or Rank -equals subspecies \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                &gt;&gt; $2
done &lt; "$1"
</code></pre>

<p>As you can see, <strong>the only difference</strong> is that I am reading in entire lines for the TaxIDs list because it is simply a list with line breaks between the IDs as opposed to a true CSV with other values as in the example that works.  I've saved the taxIDs list as both a CSV and TXT and it doesn't make a difference.  Also: I can echo what $line is in the second script and it is exactly I'd expect--just the numbers of the taxID, <strong>so why would I be getting a 400 error for bad syntax on my query if it looks exactly like what I am sending in the first, working script?</strong></p>

<p>Thanks!</p>

CC BY-SA 3.0
2018-03-15T14:13:01.480Z
Hello, nice question. Just, could you please tame ([edit]) a bit the error log? It's very hard to look look at.
Kamil S Jaron
CC BY-SA 4.0
2018-07-30T20:51:18.827Z
Aside from your specific question slightly, you might be interested in looking at the `NCBITaxa()` tool in the ETE3 package. It basically does what your script does, but provides lots of other useful methods for querying the ncbi taxa database.
Joe Healey
<taxonomy><shell>
Using a Bash Script to search TaxIDs against NCBI's Taxonomy yields "400 Bad Request" error?
# Using a Bash Script to search TaxIDs against NCBI's Taxonomy yields "400 Bad Request" error?

(Asked by: username_0 on 2018-03-14)

I've been searching TaxIDs against NCBI's Taxonomy DB to get taxonomic lineages for species.  I have successfully done this for 1,000's of TaxIDs that were returned to me by Blast+ blasts in a CSV.  (The first column in the CSV was the TaxID for the top hit of a sequence, and the other columns were for GI, Scientific Name, e-value, etc.).  I simply wrote a script that would take the first field, which is always the TaxID, and search it against Taxonomy to get lineages for the species those TaxIDs represented.  Everything worked fine!
However, some of the records returned from my original blast (that gave me the TaxIDs in the first place) would return two or more TaxIDs per record.  (I suspect because a sequence matched equally well with two or more different species?)  That's fine, but I had to remove these few exceptions before I searched the rest of the regular results against Taxonomy.  So I've got most of my data/lineages, but for about 1:10,0000 records that gave multiple TaxIDs, I would like to know the lineages of the multiple species that the blast suggested.
I now took all of those "more-than-one-taxid" hits, separated them out into columns in an excel sheet and saved it as a CSV, and am trying to search these against Taxonomy with essentially the same script I used for the rest of the data, but am getting an error for every taxID I search (see below).
Complete Error (for each taxid searched against Taxonomy):
400 Bad Request
&rettype=full&retmode=xml&edirect_os=linux&edirect=7.90&tool=edirect&email=x@xxx.localdomain'my&id=464126
Result of do_post http request is
$VAR1 = bless( {
                 '_protocol' => 'HTTP/1.1',
                 '_content' => '


 WebEnv parameter is required

',
                 '_rc' => 400,
                 '_headers' => bless( {
                                        'connection' => 'close',
                                        'set-cookie' => 'ncbi_sid=9FF573148F458940_CEC6SID; domain=.nih.gov; path=/; expires=Thu, 14 Mar 2019 22:36:03 GMT',
                                        'cache-control' => 'private',
                                        'access-control-allow-origin' => '*',
                                        'date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'vary' => 'Accept-Encoding',
                                        'client-ssl-cert-issuer' => '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA',
                                        'client-ssl-cipher' => 'ECDHE-RSA-AES256-GCM-SHA384',
                                        'client-peer' => '130.14.29.110:443',
                                        'strict-transport-security' => 'max-age=31536000; includeSubDomains; preload',
                                        'client-date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'content-type' => 'text/xml; charset=UTF-8',
                                        'x-xss-protection' => '1; mode=block',
                                        'client-transfer-encoding' => [
                                                                        'chunked'
                                                                      ],
                                        'server' => 'Finatra',
                                        'client-ssl-socket-class' => 'IO::Socket::SSL',
                                        'client-response-num' => 1,
                                        'ncbi-phid' => 'DAF6EB0134DF5AEC039F11D248910BE7.1',
                                        'ncbi-sid' => '9FF573148F458940_CEC6SID',
                                        '::std_case' => {
                                                          'set-cookie' => 'Set-Cookie',
                                                          'access-control-allow-origin' => 'Access-Control-Allow-Origin',
                                                          'client-ssl-cert-issuer' => 'Client-SSL-Cert-Issuer',
                                                          'client-ssl-cipher' => 'Client-SSL-Cipher',
                                                          'client-peer' => 'Client-Peer',
                                                          'strict-transport-security' => 'Strict-Transport-Security',
                                                          'client-date' => 'Client-Date',
                                                          'x-xss-protection' => 'X-XSS-Protection',
                                                          'client-transfer-encoding' => 'Client-Transfer-Encoding',
                                                          'client-ssl-socket-class' => 'Client-SSL-Socket-Class',
                                                          'client-response-num' => 'Client-Response-Num',
                                                          'ncbi-phid' => 'NCBI-PHID',
                                                          'ncbi-sid' => 'NCBI-SID',
                                                          'x-ua-compatible' => 'X-UA-Compatible',
                                                          'client-ssl-cert-subject' => 'Client-SSL-Cert-Subject',
                                                          'content-security-policy' => 'Content-Security-Policy'
                                                        },
                                        'x-ua-compatible' => 'IE=Edge',
                                        'client-ssl-cert-subject' => '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                        'content-security-policy' => 'upgrade-insecure-requests'
                                      }, 'HTTP::Headers' ),
                 '_msg' => 'Bad Request',
                 '_request' => bless( {
&rettype=full&retmode=xml&edirect_os=linux&edirect=7.90&tool=edirect&email=x@xxx.localdomain',
                                        '_uri' => bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                        '_headers' => bless( {
                                                               'user-agent' => 'libwww-perl/6.05',
                                                               'content-type' => 'application/x-www-form-urlencoded',
                                                               '::std_case' => {
                                                                                 'if-ssl-cert-subject' => 'If-SSL-Cert-Subject'
                                                                               }
                                                             }, 'HTTP::Headers' ),
                                        '_method' => 'POST',
                                        '_uri_canonical' => $VAR1->{'_request'}{'_uri'}
                                      }, 'HTTP::Request' )
               }, 'HTTP::Response' );

What I've done:
Here is a sample of the data I used that worked correctly and yielded lineages when searching the taxIDs (taxIDs in 1st column):
549298,Contig1,Francisella halioticida,5.44e-126,gi|1213460577|gb|CP022132.1|,CP022132
240395,Contig2,Glandirana emeljanovi,4.74e-131,gi|1025812536|gb|KU641020.1|,KU641020
1069680,Contig3,Pneumocystis murina B123,2.34e-85,gi|1124120202|ref|XM_019757798.1|,XM_019757798
3871,Contig4,Lupinus angustifolius,1.89e-93,gi|1239725835|gb|CP023132.1|,CP023132

Here is a sample of the data that I am trying to get to work (when I search these manually against Taxonomy from their web page they seem to work and give lineages; it's just that I have just enough of them to make a non-programmatic solution really time consuming and tedious):
258234
1316279
1316315
710996
711000
225338
985895
710996
1382731
77133
344338

This is the business end of the script that I used that worked for the  working data:
#!/bin/bash
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do      
        #Firstly echo the Contig# (Query Seq. ID) followed by lineage information.
        echo -n "$f2," >> $2

        #Now get the lineage, pipe to xtract and parse XML for readability in output file
        efetch -db taxonomy \
        -id $f1 \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                ...
                -or Rank -equals species \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                >> $2
done < "$1"

This is the slightly altered script that I am trying to use to make the not-working set of data yield results (and that produces the error):
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        efetch -db taxonomy \
        -id $line \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                ...
                -or Rank -equals species \
                -or Rank -equals subspecies \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                >> $2
done < "$1"

As you can see, the only difference is that I am reading in entire lines for the TaxIDs list because it is simply a list with line breaks between the IDs as opposed to a true CSV with other values as in the example that works.  I've saved the taxIDs list as both a CSV and TXT and it doesn't make a difference.  Also: I can echo what $line is in the second script and it is exactly I'd expect--just the numbers of the taxID, so why would I be getting a 400 error for bad syntax on my query if it looks exactly like what I am sending in the first, working script?
Thanks!

**Tags:** taxonomy, shell
> **Comments:**
> **username_1**: Hello, nice question. Just, could you please tame ([edit]) a bit the error log? It's very hard to look look at.  
> **username_2**: Aside from your specific question slightly, you might be interested in looking at the `NCBITaxa()` tool in the ETE3 package. It basically does what your script does, but provides lots of other useful methods for querying the ncbi taxa database.  

---

## 1 answers

### Answer by username_3 on 2018-07-27. Score: 3 | Accepted answer

If I understand correctly, your question boils down to: parsing a bunch of numbers in a row from a .csv file gives a different result from parsing a bunch of numbers as *rows* of a text file. I that right? I suggest that you look at the source of your text file. For example, was the text file created on a Windows host (running bash, or in the command terminal?)?

Try using this modified version of the unix cat command:
`cat -et text_file.txt`

The very end of each line will have a '\$' sign and that will show you if there are any extra non-visible characters in your rows. For example, a DOS/Windows formatted file will usually have a ^M at the end of each row before the '$'. If you don't strip off those line endings then they get added to your queries, even if you cannot see them. You can strip off those linefeed (or are they carriage returns?) using a Perl one-liner like:
`perl -pe 's/\r//' badfile.txt > goodfile.txt`

> **Comments:**
> **username_0**: Thanks, I suspect that between generating these files on my school's servers through bash to downloading them locally and manipulating them in Excel, that there must've been '$' characters like you said.  Since then, I've learned to do what I was doing in Excel on the command line and think I'll be able to avoid this altogether.  

---

