<p>For generating the index, you should be using one of <code>-i</code> or <code>--index</code>, but not both:</p>

<pre><code>./kallisto index --make-unique -i index.fa.idx index.fa 
</code></pre>

<p>You need to point kallisto to the generated index file (usually has a <code>.idx</code> extension), rather than the fasta file:</p>

<pre><code>for file in *.fasta;
  do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
done
</code></pre>

CC BY-SA 3.0
2017-06-30T09:30:44.477Z
Why run it sequentially? Shouldn't this be parallelized, i.e. multiprocessed
Chris_Rands
CC BY-SA 3.0
2017-06-30T11:38:47.110Z
You're asking the wrong person; try adding that as a comment to the question. `Kallisto quant` allows multi-threaded operation with `-t <threads>`.
gringer
CC BY-SA 3.0
2017-06-30T15:18:44.913Z
I get the same error when I point to the index file

    Error: incompatible indices. Found version 0, expected version 10
ZincFingers
CC BY-SA 3.0
2017-06-30T16:05:52.333Z
maybe it's the index command that needs a bit of attention... I'll update the commands to include index generation
gringer
CC BY-SA 3.0
2017-06-30T16:20:11.787Z
I managed to get this to work after fiddling with the index generation:
' for f in *.fasta' do kallisto quant -i index --single -l 200 -s 0.1 -o "${file}-aligned" "${file)"; '

But it appears that it's repeatedly analyzing one of the 10 .fasta files I placed in the directory (also I'm not sure how to get that line to appear as a code)
ZincFingers
CC BY-SA 3.0
2017-06-30T16:24:38.760Z
@gringer - just an additional comment on multithreading: as of version `0.43.1`, the `-t <threads>` parameter doesn't work should you wish to look at the alignments (which `kallisto quant` can output upon supplying `--pseudobam`. I usually get around it by running several individual `kallisto quant` commands concurrently.
flatley176
CC BY-SA 3.0
2017-06-30T21:12:51.717Z
@gringer Using `-t` is fine but it’s actually almost certainly much more efficient to process the *files* in parallel rather than in a sequential loop.
Konrad Rudolph
CC BY-SA 3.0
2017-07-03T09:10:23.250Z
`-t` is not going to help, it's only useful if you've enabled bootstrapping, otherwise you can set it as high as you want but kallisto will still run single core.  Multi-threaded execution is used only for concurrently running the bootstrapping.
Matt Bashton
CC BY-SA 3.0
2017-07-03T09:11:50.737Z
If you're using tximport for downstream analysis in DESeq2 rather than Sleuth the bootstrapping won't be used as it's not taking into account quantification uncertainty via the bootstrapping procedure, so you need to consider if this is needed.
Matt Bashton
CC BY-SA 3.0
2017-06-29T23:43:58.127Z
2017-06-30T16:13:19.153Z
2017-06-30T16:13:19.153Z
gringer
<p><strong>Running the jobs in parallel</strong></p>

<p>Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular <a href="http://bioconductor.org/packages/release/bioc/vignettes/tximport/inst/doc/tximport.html" rel="nofollow noreferrer">tximport</a> route) you can do better using <a href="https://www.gnu.org/software/parallel/" rel="nofollow noreferrer">GNU parallel</a>, this will enable you to run as many jobs as you have execution threads on your system simultaneously.  Note that Kalliso only runs multithreaded with <code>-t</code> if you're using bootstrapping.</p>

<ol>
<li><p>You'll need to make the index first:</p>

<pre><code>kallisto index --make-unique -I index.fa.idx index.fa
</code></pre></li>
<li><p>Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:</p>

<pre><code>sample_1    sample_1_R1.fastq.gz
sample_2    sample_2_R1.fastq.gz
sample_3    sample_3_R1.fastq.gz
</code></pre>

<p>Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file. </p>

<p>You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:</p>

<pre><code>ls -1 *.fastq.gz &gt; editme
</code></pre></li>
<li><p>Next we create a list of jobs to run on our system using a utility script <code>Make_job_list_kallisto.sh</code> which consumes our tab delimited input file list above.  I've used your <code>-l</code> and <code>-s</code> parameters below, note that with paired end data normally these are estimated for you.</p>

<pre><code>#!/bin/bash -eu

[ $# -ne 2 ] &amp;&amp; { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: &lt; input tab delimited list &gt; &lt; output run list file &gt;\n\n" ; exit 1; }
set -o pipefail

# Get command-line args
INPUT_LIST=$1
OUTPUT=$2

# Set counter
COUNT=1
END=$(wc -l $INPUT_LIST | awk '{print $1}')

echo " "
echo " * Input file is: $INPUT_LIST"
echo " * Number of runs: $END"
echo " * Output job list for GNU parallel saved to: $OUTPUT"
echo " "

# Main bit of command-line for job
CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"

# Main Loop
[ -e $OUTPUT ] &amp;&amp; rm $OUTPUT
while [ $COUNT -le $END ];
do
    LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
    # Make file list
    echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
    echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" &gt;&gt; $OUTPUT
    ((COUNT++))
done
</code></pre>

<p>We use the script like so:</p>

<pre><code> Make_job_list_kallisto.sh input_file_list.txt job_list.txt
</code></pre></li>
<li><p>Now we have a job list for GNU parallel we can run our jobs in parallel like so:</p>

<pre><code>parallel --progress --jobs 4 --joblog kallisto_joblog.txt &lt; job_list.txt
</code></pre>

<p>This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file <code>kallisto_joblog.txt</code>.  </p></li>
</ol>

<p>For paired end data I already have these scripts + similar instructions on <a href="https://github.com/MattBashton/grolar" rel="nofollow noreferrer">GitHub</a> which people might find useful. </p>

CC BY-SA 3.0
2017-07-03T20:49:52.250Z
@ZincFingers, so `-I` upper case for specifying the index in my run generation scrip should in fact be lower case `-i`.  I've tested this and it works for me.  This might be your issue or you might have some other error.  If it's still not working try pasting in a few lines of your jobs list here.
Matt Bashton
CC BY-SA 3.0
2017-07-03T20:56:04.290Z
The error I was getting was my fault, I fixed it. Now, when I run the parallel command, I get an error from kallisto saying 

Error: missing read files
Error: file srr3884878.gz exists and is not a directory

It says this for all 17 of my files
ZincFingers
CC BY-SA 3.0
2017-07-03T21:35:21.343Z
I suspect you still have an issue with you input file to `Make_job_list_kallisto.sh` or the jobs it's generating, without seeing the input sample ID file list or the job list it's very difficult to debug.
Matt Bashton
CC BY-SA 3.0
2017-07-03T21:39:32.763Z
Check the diagnostic output of `Make_job_list_kallisto.sh` is sane, I get:
`* Input file is: test_for_BS.txt`
`* Number of runs: 4`
`* Output for GNU parallel saved to: test_job_list.txt`

`Working on 1 of 4 Sample ID: test1, Files ATCACG_L004_R1_001.fastq.gz`
`Working on 2 of 4 Sample ID: test2, Files GTCCGC_L004_R1_001.trimmed.fastq.gz`

etc.
Matt Bashton
CC BY-SA 3.0
2017-07-03T21:54:28.363Z
I think you might have forgotten to give an output dir or sample ID in the first column of your tab delimited input list you're giving to `Make_job_list_kallisto.sh` because it looks like kallisto is missing the variable which would be supplied to `-o` and treating the input fastq.gz file as the output dir, it's then not finding the gz.fastq file as it's trying to use that as the output dir.
Matt Bashton
CC BY-SA 3.0
2017-07-03T22:09:48.640Z
You were right! I fixed the input file and it seems to have resolved the issue. Many thanks for your fantastic solution and bearing with me while I trudged through this.
ZincFingers
CC BY-SA 3.0
2017-07-03T22:25:54.190Z
Glad you got it working!
Matt Bashton
CC BY-SA 3.0
2017-07-03T09:02:32.610Z
2017-07-03T20:50:27.253Z
2017-07-03T20:50:27.253Z
Matt Bashton
<p>I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:</p>

<pre><code>./kallisto index -i --index --make-unique index.fa 
</code></pre>

<p>This successfully creates an index file. Then, I tried this:</p>

<pre><code>for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
"$file"; done
</code></pre>

<p>While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:</p>

<pre><code>for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
$file-aligned  "$file"; done
</code></pre>

<p>I get an error that says "Error: incompatible indices. Found version 0, expected version 10. </p>

<p>I have now also tried the following:</p>

<pre><code>for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
"${file}-aligned" "${file}"; 
</code></pre>

<p>but I get an error saying the index file is not found.</p>

<hr>

<p>I am now running </p>

<pre><code>or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 
</code></pre>

<p>"$file"; done</p>

<p>This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly. </p>

<p>If I close terminal, start a new session and run the same command, I get</p>

<pre><code>Error: file not found ""
</code></pre>

CC BY-SA 3.0
2017-06-29T23:44:49.577Z
Why are you using fasta files for reads? Kallisto expects FASTQ files as input.
gringer
CC BY-SA 3.0
2017-06-30T07:55:48.413Z
Does kallisto use quality scores of reads? I had an impression that it does not, but I don't have a reference to backup it....
Kamil S Jaron
CC BY-SA 4.0
2021-05-12T18:15:09.330Z
I down-voted because this post does not pose a question.
winni2k
<rna-seq>
Run kallisto iteratively across many samples
# Run kallisto iteratively across many samples

(Asked by: username_0 on 2017-06-29)

I am on a Mac using UNIX. I am trying to use the kallisto quant command on all files in a directory (instead of manually entering them). Because I'm running the analysis against the same index file, I first enter the following:
./kallisto index -i --index --make-unique index.fa 

This successfully creates an index file. Then, I tried this:
for file in *.fasta; do kallisto quant --single -l 200 -s 0.1 -o $file-aligned 
"$file"; done

While the $file-aligned folders are created, they're empty. I get an error that states that the index file is missing. So I assumed I would need to specify the index file in the command line. I then tried this:
for file in *.fasta; do kallisto quant -i index.fa --single -l 200 -s 0.1 -o
$file-aligned  "$file"; done

I get an error that says "Error: incompatible indices. Found version 0, expected version 10. 
I have now also tried the following:
for file in *.fasta; do ./kallisto quant -i index --single - 200 -s 0.01 -o 
"${file}-aligned" "${file}"; 

but I get an error saying the index file is not found.

I am now running 
or file in *.fasta; do ./kallisto quant -i index --single -l 200 -s 0.1 -o $file-aligned 

"$file"; done
This sort of works. The index file error messages are gone, but it runs quant on the same file 10 times instead of each file once. If I run echo $file it spits back the file that my command is running repeatedly. 
If I close terminal, start a new session and run the same command, I get
Error: file not found ""

**Tags:** rna-seq
> **Comments:**
> **username_1**: Why are you using fasta files for reads? Kallisto expects FASTQ files as input.  
> **username_2**: Does kallisto use quality scores of reads? I had an impression that it does not, but I don't have a reference to backup it....  
> **username_3**: I down-voted because this post does not pose a question.  

---

## 2 answers

### Answer by username_7 on 2017-07-03. Score: 7 | Accepted answer

Running the jobs in parallel
Iteratively is one solution, but since you're not using bootstrapping, which can be run multithread (and is not needed if you're taking the popular tximport route) you can do better using GNU parallel, this will enable you to run as many jobs as you have execution threads on your system simultaneously.  Note that Kalliso only runs multithreaded with -t if you're using bootstrapping.

You'll need to make the index first:
kallisto index --make-unique -I index.fa.idx index.fa

Next lets define the input FASTQ files and the sample IDs you want to use using a tab-delimited input file like so:
sample_1    sample_1_R1.fastq.gz
sample_2    sample_2_R1.fastq.gz
sample_3    sample_3_R1.fastq.gz

Here the first column is the sample ID which will be used for output and the second separated by a tab character is the input file. 
You can produce this using a text editor but as a short-cut you can feed the list of fastq.gz you have into a list via:
ls -1 *.fastq.gz > editme

Next we create a list of jobs to run on our system using a utility script Make_job_list_kallisto.sh which consumes our tab delimited input file list above.  I've used your -l and -s parameters below, note that with paired end data normally these are estimated for you.
#!/bin/bash -eu

[ $# -ne 2 ] && { echo -en "\n*** This script generates jobs for GNU parallel. *** \n\n Error Nothing to do, usage: < input tab delimited list > < output run list file >\n\n" ; exit 1; }
set -o pipefail

# Get command-line args
INPUT_LIST=$1
OUTPUT=$2

# Set counter
COUNT=1
END=$(wc -l $INPUT_LIST | awk '{print $1}')

echo " "
echo " * Input file is: $INPUT_LIST"
echo " * Number of runs: $END"
echo " * Output job list for GNU parallel saved to: $OUTPUT"
echo " "

# Main bit of command-line for job
CMD="kallisto quant -i index.fa.idx --single -l 200 -s 0.1"

# Main Loop
[ -e $OUTPUT ] && rm $OUTPUT
while [ $COUNT -le $END ];
do
    LINE=( $(awk "NR==$COUNT" $INPUT_LIST) )
    # Make file list
    echo "Working on $COUNT of $END Sample ID: ${LINE[0]}, Files ${LINE[@]:1}"
    echo "$CMD -o ${LINE[0]} ${LINE[@]:1}" >> $OUTPUT
    ((COUNT++))
done

We use the script like so:
 Make_job_list_kallisto.sh input_file_list.txt job_list.txt

Now we have a job list for GNU parallel we can run our jobs in parallel like so:
parallel --progress --jobs 4 --joblog kallisto_joblog.txt < job_list.txt

This will run 4 of the kallisto jobs simultaneously (Increase this if you have more threads - don't mind consuming all the IO/CPU threads on your system) producing a nice progress report of those completed, additionally a job log will be written to the file kallisto_joblog.txt.  

For paired end data I already have these scripts + similar instructions on GitHub which people might find useful.

> **Comments:**
> **username_7**: @ZincFingers, so `-I` upper case for specifying the index in my run generation scrip should in fact be lower case `-i`.  I've tested this and it works for me.  This might be your issue or you might have some other error.  If it's still not working try pasting in a few lines of your jobs list here.  
> **username_0**: The error I was getting was my fault, I fixed it. Now, when I run the parallel command, I get an error from kallisto saying   Error: missing read files Error: file srr3884878.gz exists and is not a directory  It says this for all 17 of my files  
> **username_7**: I suspect you still have an issue with you input file to `Make_job_list_kallisto.sh` or the jobs it's generating, without seeing the input sample ID file list or the job list it's very difficult to debug.  
> **username_7**: Check the diagnostic output of `Make_job_list_kallisto.sh` is sane, I get: `* Input file is: test_for_BS.txt` `* Number of runs: 4` `* Output for GNU parallel saved to: test_job_list.txt`  `Working on 1 of 4 Sample ID: test1, Files ATCACG_L004_R1_001.fastq.gz` `Working on 2 of 4 Sample ID: test2, Files GTCCGC_L004_R1_001.trimmed.fastq.gz`  etc.  
> **username_7**: I think you might have forgotten to give an output dir or sample ID in the first column of your tab delimited input list you're giving to `Make_job_list_kallisto.sh` because it looks like kallisto is missing the variable which would be supplied to `-o` and treating the input fastq.gz file as the output dir, it's then not finding the gz.fastq file as it's trying to use that as the output dir.  
> **username_0**: You were right! I fixed the input file and it seems to have resolved the issue. Many thanks for your fantastic solution and bearing with me while I trudged through this.  
> **username_7**: Glad you got it working!  

---

### Answer by username_1 on 2017-06-29. Score: 5

For generating the index, you should be using one of -i or --index, but not both:
./kallisto index --make-unique -i index.fa.idx index.fa 

You need to point kallisto to the generated index file (usually has a .idx extension), rather than the fasta file:
for file in *.fasta;
  do kallisto quant -i index.fa.idx --single -l 200 -s 0.1 -o "${file}-aligned" "${file}";
done

> **Comments:**
> **username_4**: Why run it sequentially? Shouldn't this be parallelized, i.e. multiprocessed  
> **username_1**: You're asking the wrong person; try adding that as a comment to the question. `Kallisto quant` allows multi-threaded operation with `-t <threads>`.  
> **username_0**: I get the same error when I point to the index file      Error: incompatible indices. Found version 0, expected version 10  
> **username_1**: maybe it's the index command that needs a bit of attention... I'll update the commands to include index generation  
> **username_0**: I managed to get this to work after fiddling with the index generation: ' for f in *.fasta' do kallisto quant -i index --single -l 200 -s 0.1 -o "${file}-aligned" "${file)"; '  But it appears that it's repeatedly analyzing one of the 10 .fasta files I placed in the directory (also I'm not sure how to get that line to appear as a code)  
> **username_5**: @gringer - just an additional comment on multithreading: as of version `0.43.1`, the `-t <threads>` parameter doesn't work should you wish to look at the alignments (which `kallisto quant` can output upon supplying `--pseudobam`. I usually get around it by running several individual `kallisto quant` commands concurrently.  
> **username_6**: @gringer Using `-t` is fine but it’s actually almost certainly much more efficient to process the *files* in parallel rather than in a sequential loop.  
> **username_7**: `-t` is not going to help, it's only useful if you've enabled bootstrapping, otherwise you can set it as high as you want but kallisto will still run single core.  Multi-threaded execution is used only for concurrently running the bootstrapping.  
> **username_7**: If you're using tximport for downstream analysis in DESeq2 rather than Sleuth the bootstrapping won't be used as it's not taking into account quantification uncertainty via the bootstrapping procedure, so you need to consider if this is needed.  

---

