<p>I do this very frequently, using the read file name to identify FAST5 files associated with particular reads. If a FASTQ record includes the channel number and read number (and preferably the runID as well), then I use that information to find the associated FAST5 read.</p>

<p>If reads are called with Albacore, then the <code>sequencing_summary.txt</code> file has additional information. The first column is the exact name of the file that was called, the second column is the read ID of the FASTQ sequence, and the third column is the run ID associated with that read ID. This is much easier to work with, but does require calling reads with Albacore (which, admittedly, is what seems to be producing the best results at the moment).</p>

CC BY-SA 3.0
2017-08-16T05:38:59.320Z
yes this is what I did, basically: 1. extract fast5 to fastq; 2. align the fastq to the reference; 3. get the reads ID that map from the SAM of alignment; 4. grep the mapping reads ID from the fastq; 5. extract the name of the fast5 from the fasta header of each read (last field); 6. find the corresponding fast5 in the original fast5 directory. But I was wondering if I did not miss any software that could do this on one run.
aechchiki
CC BY-SA 3.0
2017-08-16T07:39:08.467Z
Added a note about albacore, which has a direct link between FASTQ read IDs and FAST5 file names
gringer
CC BY-SA 3.0
2017-08-16T08:25:50.843Z
In light of your above comment @AminaEchchiki I have added in compatibility for BAM and SAM files too.
Michael Hall
CC BY-SA 3.0
2017-08-15T18:44:00.690Z
2017-08-16T07:38:17.043Z
2017-08-16T07:38:17.043Z
gringer
<p><strong>NOTICE</strong>:
I have altered my answer slightly from the original as I have turned the original script into a <code>pip</code> installable program (with tests) and have updated the links and code snippets accordingly. The essence of the answer is still exactly the same.</p>

<hr>

<p>This is something I have been meaning to get around to for a while, so thanks for the prompt.</p>

<p>I have <a href="https://github.com/mbhall88/fast5seek" rel="nofollow noreferrer">created a python program called <code>fast5seek</code></a> to do what (I think) you're after.</p>

<p>As you have mentioned this is for educational purposes I have added a tonne of comments to the code too so I think you shouldn't have any issues following it.</p>

<p>The docs on the <a href="https://github.com/mbhall88/fast5seek" rel="nofollow noreferrer">GitHub repo</a> have all the info, but for those reading along at home</p>

<pre><code>pip3 install fast5seek
fast5seek -i /path/to/fast5s -r in.fastq in.bam in.sam -o out.txt
</code></pre>

<p>What it does is read in <code>&lt;in.fastq|in.bam|in.sam&gt;</code> and extract the read id from each header. It then goes through all the fast5 files under <code>/path/to/fast5s</code> and checks whether their read id is in the set of read ids from <code>&lt;in.fastq|in.bam|in.sam&gt;</code>. If it is, the path to the file is written to it's own line in <code>out.txt</code>.</p>

<p>If no output (<code>-o</code>) is given, it will write the output to stdout.</p>

<p>So if you wanted to pipe these paths into another program, you could do something like</p>

<pre><code>mkdir subset_dir/
fast5seek -i /path/to/fast5s/ -r in.fastq | xargs cp -t subset_dir/
</code></pre>

<p>The above example would copy the fast5 files that are found in your fastq/BAM/SAM to <code>subset_dir/</code>.</p>

CC BY-SA 3.0
2017-08-16T11:40:03.693Z
As a caution, the run id should be checked (if that information is available). There's a small possibility that two FASTQ files from different runs will have the same read id.
gringer
CC BY-SA 3.0
2017-08-17T00:56:10.317Z
I have now added this check for fastq files. Although, run_id is not contained within BAM/SAM files (at least not for the ones I have) so this check will only occur on fastq.
Michael Hall
CC BY-SA 4.0
2017-08-16T04:30:09.763Z
2018-11-02T12:22:56.053Z
2018-11-02T12:22:56.053Z
Michael Hall
<p><code>fast5</code> is a variant of <a href="https://support.hdfgroup.org/HDF5/whatishdf5.html" rel="noreferrer"><code>HDF5</code></a> the native format in which raw data from Oxford Nanopore MinION are provided. You can easily extract the reads in fast5 format into a standard fastq format, using for example <a href="https://poretools.readthedocs.io/en/latest/" rel="noreferrer"><code>poretools</code></a>. </p>

<p>Say I have aligned these reads in <code>fastq</code> format to an external reference genome, resulting in a <code>SAM</code> file. Say I have then taken a subset of the  <code>SAM</code> file, according to the <a href="https://samtools.github.io/hts-specs/SAMv1.pdf" rel="noreferrer">bitwise flag</a>, to include only the reads that map to the reference. With the read ID, I can then grep them out from the file containing the reads in <code>fastq</code> format, generating a subset file in <code>fastq</code> format containing only the IDs that have mapped to the reference.  </p>

<p>Now my question is, can we subset reads from the <code>fast5</code> archive according to the list of mapping reads as taken from the file with reads in <code>fastq</code> format? This is for educational purposes, so that we have a smaller starting archive, and the <code>fast5</code> -> <code>fastq</code> extraction takes less cpu time. </p>

CC BY-SA 3.0
2017-08-15T16:18:05.127Z
This seems a nice question to play around if we could have sample data. Could you provide (or a link to) a fast5 and the list of mappings ?
llrs
CC BY-SA 3.0
2017-08-15T18:09:46.967Z
There are some sample reads from my TEDx talk last year [here](https://www.dropbox.com/sh/o7fz7s91865u6vw/AAAsIzqbmrylD6lHl-yOJggXa?dl=0), reads from runs I did during the AGTA conference last year [here](https://www.dropbox.com/sh/0iqbjk7drmpsro0/AACBehmjVL7iABf5U2NVi9Lra?dl=0), and reads from a chimeric read investigation [here](http://www.ebi.ac.uk/ena/data/view/PRJEB20601). The ONT FAST5 "standard" change so frequently that it's difficult to find something that represents what's currently available.
gringer
<nanopore><software-recommendation><fastq><format-conversion><fast5>
How to convert fastq to fast5
# How to convert fastq to fast5

(Asked by: username_0 on 2017-08-15)

`fast5` is a variant of [`HDF5`](https://support.hdfgroup.org/HDF5/whatishdf5.html) the native format in which raw data from Oxford Nanopore MinION are provided. You can easily extract the reads in fast5 format into a standard fastq format, using for example [`poretools`](https://poretools.readthedocs.io/en/latest/).

Say I have aligned these reads in `fastq` format to an external reference genome, resulting in a `SAM` file. Say I have then taken a subset of the `SAM` file, according to the [bitwise flag](https://samtools.github.io/hts-specs/SAMv1.pdf), to include only the reads that map to the reference. With the read ID, I can then grep them out from the file containing the reads in `fastq` format, generating a subset file in `fastq` format containing only the IDs that have mapped to the reference.

Now my question is, can we subset reads from the `fast5` archive according to the list of mapping reads as taken from the file with reads in `fastq` format? This is for educational purposes, so that we have a smaller starting archive, and the `fast5` -> `fastq` extraction takes less cpu time.

**Tags:** nanopore, software-recommendation, fastq, format-conversion, fast5
> **Comments:**
> **username_1**: This seems a nice question to play around if we could have sample data. Could you provide (or a link to) a fast5 and the list of mappings ?  
> **username_2**: There are some sample reads from my TEDx talk last year [here](https://www.dropbox.com/sh/o7fz7s91865u6vw/AAAsIzqbmrylD6lHl-yOJggXa?dl=0), reads from runs I did during the AGTA conference last year [here](https://www.dropbox.com/sh/0iqbjk7drmpsro0/AACBehmjVL7iABf5U2NVi9Lra?dl=0), and reads from a chimeric read investigation [here](http://www.ebi.ac.uk/ena/data/view/PRJEB20601). The ONT FAST5 "standard" change so frequently that it's difficult to find something that represents what's currently available.  

---

## 2 answers

### Answer by username_3 on 2017-08-16. Score: 6 | Accepted answer

NOTICE:
I have altered my answer slightly from the original as I have turned the original script into a pip installable program (with tests) and have updated the links and code snippets accordingly. The essence of the answer is still exactly the same.

This is something I have been meaning to get around to for a while, so thanks for the prompt.
I have created a python program called fast5seek to do what (I think) you're after.
As you have mentioned this is for educational purposes I have added a tonne of comments to the code too so I think you shouldn't have any issues following it.
The docs on the GitHub repo have all the info, but for those reading along at home
pip3 install fast5seek
fast5seek -i /path/to/fast5s -r in.fastq in.bam in.sam -o out.txt

What it does is read in  and extract the read id from each header. It then goes through all the fast5 files under /path/to/fast5s and checks whether their read id is in the set of read ids from . If it is, the path to the file is written to it's own line in out.txt.
If no output (-o) is given, it will write the output to stdout.
So if you wanted to pipe these paths into another program, you could do something like
mkdir subset_dir/
fast5seek -i /path/to/fast5s/ -r in.fastq | xargs cp -t subset_dir/

The above example would copy the fast5 files that are found in your fastq/BAM/SAM to subset_dir/.

> **Comments:**
> **username_2**: As a caution, the run id should be checked (if that information is available). There's a small possibility that two FASTQ files from different runs will have the same read id.  
> **username_3**: I have now added this check for fastq files. Although, run_id is not contained within BAM/SAM files (at least not for the ones I have) so this check will only occur on fastq.  

---

### Answer by username_2 on 2017-08-15. Score: 3

I do this very frequently, using the read file name to identify FAST5 files associated with particular reads. If a FASTQ record includes the channel number and read number (and preferably the runID as well), then I use that information to find the associated FAST5 read.

If reads are called with Albacore, then the `sequencing_summary.txt` file has additional information. The first column is the exact name of the file that was called, the second column is the read ID of the FASTQ sequence, and the third column is the run ID associated with that read ID. This is much easier to work with, but does require calling reads with Albacore (which, admittedly, is what seems to be producing the best results at the moment).

> **Comments:**
> **username_0**: yes this is what I did, basically: 1. extract fast5 to fastq; 2. align the fastq to the reference; 3. get the reads ID that map from the SAM of alignment; 4. grep the mapping reads ID from the fastq; 5. extract the name of the fast5 from the fasta header of each read (last field); 6. find the corresponding fast5 in the original fast5 directory. But I was wondering if I did not miss any software that could do this on one run.  
> **username_2**: Added a note about albacore, which has a direct link between FASTQ read IDs and FAST5 file names  
> **username_3**: In light of your above comment @AminaEchchiki I have added in compatibility for BAM and SAM files too.  

---

