<p>It looks like you're creating too much additional work for yourself. What I'm trying to lead to with my comments is that the file names should help in working out if a file represents single reads or multiple reads, in which case the single reads can be shifted into a separate folder.</p>
<p>Based on the file names that I would expect, a search for files containing '_read_[0-9]' should be sufficient to fish out the single fast5 files:</p>
<pre><code>find . -name '*_read_[0-9]*' &gt; single_reads.txt
mkdir -p single_reads
cat single_reads.txt | while read fname; do mv -i ${fname} single_reads; done
</code></pre>
<p><em>[That <code>single_reads.txt</code> file could be split into multiple files if directory entries get too large.]</em></p>
<p>After which you should have a single folder that contains thousands of single-read fast5 files, and other folders that contain multi-fast5 files, removing the problem of needing to deal with mixed file types.</p>

CC BY-SA 4.0
2023-12-13T21:08:00.113Z
I am gonna try this, thank you! @gringer
Marjan
CC BY-SA 4.0
2023-12-10T05:36:26.217Z
2023-12-14T02:44:44.233Z
2023-12-14T02:44:44.233Z
gringer
<p>I have a dataset that contains thousands of mixed multiple and single fast5 files in a non-homogenous folder structure.
The reads and fast5 files are coming from different labs and some labs gave single fast5 files and some multi fast5 files, without clear mappings. My final goal is to convert all the fast5 files to multi fast5 files.</p>
<p>There are commands in ont_fast5_api which is an interface to HDF5 files of the Oxford Nanopore.
Using this API we can convert single fast5 files and multi fast5 files to each other.
this is the link to this API github page:
<a href="https://github.com/nanoporetech/ont_fast5_api" rel="nofollow noreferrer">https://github.com/nanoporetech/ont_fast5_api</a></p>
<p>So I made a test mixed folder to check if I can use the API without missing reads as a result of conversion. The test folder structure is similar to my original folder structure:</p>
<ul>
<li>mixed
<ul>
<li>Subfolder_A
<ul>
<li>single_fast5 x 4000</li>
</ul>
</li>
<li>Subfolder_B
<ul>
<li>multiple_fast5 (contains 4000 single reads fast5)</li>
</ul>
</li>
<li>single_fast5 x 2</li>
<li>multiple_fast5_1 (contains 4000 single reads fast5)</li>
<li>multiple_fast5_2 (contains 4000 single reads fast5)</li>
<li>multiple_fast5_3 (contains 100 single reads fast5)</li>
</ul>
</li>
</ul>
<p>Here is a screenshot of the structure with a few files from each folder:</p>
<p><img src="https://cdn.discordapp.com/attachments/991841204088098977/1183239459714646037/mixed_tree.png?ex=65879c9e&amp;is=6575279e&amp;hm=0747962eff4b5167c6026c1d7a9612d66983bcab5fb3ce5a3315166b2c19dc5d&amp;" alt="alt text" title="folder structure" /></p>
<p>and I created two other folders outside of the mixed folder as follows:</p>
<ul>
<li><p>single</p>
</li>
<li><p>multi</p>
</li>
</ul>
<p>Here my question is about <em>single_to_multi_fast5</em> and <em>multi_to_single_fast5</em> commands.</p>
<pre><code>test_path=mixed
multi_path=multi
single_path=single
multi_to_single_fast5 -i $test_path/ -s $single_path/ --recursive
</code></pre>
<p>in the $single_path, I expect to see, all of the reads in multi files converted to single files (12100 reads in 12100 single fast5 files), but I see 8100 single files.</p>
<p><img src="https://cdn.discordapp.com/attachments/991841204088098977/1183088499252023406/single.png?ex=65871006&amp;is=65749b06&amp;hm=484c575feb7ddb4effbb6f05bbb91fe64290c7da4d1c25b4339aaffcccd77d50&amp;" alt="alt text" title="single path" />
So I have 4000 missing reads.</p>
<p>Now I convert single read fast5 file to multi:</p>
<pre><code>single_to_multi_fast5 -i $test_path/ -s $multi_path/ --filename_base $output_name --batch_size 1000 --recursive
</code></pre>
<p>in the $multi_path folder, I expect to see that generated multi files contain all my single-read files (4000 + 2), but I see 3006 in the mapping summary.</p>
<p><img src="https://cdn.discordapp.com/attachments/991841204088098977/1183089993523154984/multi.png?ex=6587116b&amp;is=65749c6b&amp;hm=6d78f50b7a5ede6581c7b0205d1db5e74d88397963907ebed457ce6f8f487592&amp;" alt="alt text" title="mutli path" /></p>
<p>So again I missed 996 single reads in my multi files.
Since I made a small subsample of data, I am sure that the files are not overwritten.</p>
<p>Because my final goal is <strong>converting everything to multi</strong> I was going to do the following <strong>pipeline</strong>:</p>
<pre><code>orig_path=original_folder
intermediate_path=intermediate
final_path=final_multi
# converts/collects all multi files in the original folder
multi_to_single_fast5 -i <span class="math-container">$orig_path/ -s $</span>intermediate_path/ --recursive
single_to_multi_fast5 -i <span class="math-container">$intermediate_path/ -s $</span>final_path/ --filename_base $output_name --batch_size 1000 --recursive

# converts/collects all single files in the original folder
single_to_multi_fast5 -i <span class="math-container">$orig_path/ -s $</span>final_path/ --filename_base $output_name --batch_size 1000 --recursive

</code></pre>
<p>However because it seems I get missing reads during the process, I cannot use this pipeline.</p>
<p>At the same time, my original dataset is super huge, I cannot check if individual files are single or multi. It would take ages (I guess).
Is there a solution for solving this problem besides checking every file? Maybe my pipeline is not quite suitable for what I want to do.</p>

CC BY-SA 4.0
2023-12-09T20:20:32.723Z
Can you please show the first few filenames in each folder? e.g. by running the following command: `find mixed -type 'd' | while read dname; do echo "** ${dname} **"; tree -L 1 ${dname} | head; done`
gringer
CC BY-SA 4.0
2023-12-10T02:56:09.893Z
Thank you for your reply! I included the first few filenames in each folder @gringer
Marjan
CC BY-SA 4.0
2023-12-10T05:25:28.453Z
The image seems like it's annotated incorrectly. A file with a channel and read number should be a single fast5 file, not a multi fast5 file.
gringer
CC BY-SA 4.0
2023-12-13T21:06:12.563Z
Yes, the 2nd label (multi fast5 files with 4000 reads) and the 3rd label (single fast5 files) should be swapped. @gringer
Marjan
CC BY-SA 4.0
2023-12-14T02:44:01.150Z
Okay, that's good to know; thanks for the clarification.
gringer
<nanopore><fast5>
single_to_multi_fast5 does not collect all the single files if the input folder contains mixed types of fast5 files
# single_to_multi_fast5 does not collect all the single files if the input folder contains mixed types of fast5 files

(Asked by: username_0 on 2023-12-03)

I have a dataset that contains thousands of mixed multiple and single fast5 files in a non-homogenous folder structure.
The reads and fast5 files are coming from different labs and some labs gave single fast5 files and some multi fast5 files, without clear mappings. My final goal is to convert all the fast5 files to multi fast5 files.
There are commands in ont_fast5_api which is an interface to HDF5 files of the Oxford Nanopore.
Using this API we can convert single fast5 files and multi fast5 files to each other.
this is the link to this API github page:
https://github.com/nanoporetech/ont_fast5_api
So I made a test mixed folder to check if I can use the API without missing reads as a result of conversion. The test folder structure is similar to my original folder structure:

mixed

Subfolder_A

single_fast5 x 4000


Subfolder_B

multiple_fast5 (contains 4000 single reads fast5)


single_fast5 x 2
multiple_fast5_1 (contains 4000 single reads fast5)
multiple_fast5_2 (contains 4000 single reads fast5)
multiple_fast5_3 (contains 100 single reads fast5)



Here is a screenshot of the structure with a few files from each folder:

and I created two other folders outside of the mixed folder as follows:

single

multi


Here my question is about single_to_multi_fast5 and multi_to_single_fast5 commands.
test_path=mixed
multi_path=multi
single_path=single
multi_to_single_fast5 -i $test_path/ -s $single_path/ --recursive

in the $single_path, I expect to see, all of the reads in multi files converted to single files (12100 reads in 12100 single fast5 files), but I see 8100 single files.

So I have 4000 missing reads.
Now I convert single read fast5 file to multi:
single_to_multi_fast5 -i $test_path/ -s $multi_path/ --filename_base $output_name --batch_size 1000 --recursive

in the $multi_path folder, I expect to see that generated multi files contain all my single-read files (4000 + 2), but I see 3006 in the mapping summary.

So again I missed 996 single reads in my multi files.
Since I made a small subsample of data, I am sure that the files are not overwritten.
Because my final goal is converting everything to multi I was going to do the following pipeline:
orig_path=original_folder
intermediate_path=intermediate
final_path=final_multi
# converts/collects all multi files in the original folder
multi_to_single_fast5 -i $orig_path/ -s $intermediate_path/ --recursive
single_to_multi_fast5 -i $intermediate_path/ -s $final_path/ --filename_base $output_name --batch_size 1000 --recursive

# converts/collects all single files in the original folder
single_to_multi_fast5 -i $orig_path/ -s $final_path/ --filename_base $output_name --batch_size 1000 --recursive


However because it seems I get missing reads during the process, I cannot use this pipeline.
At the same time, my original dataset is super huge, I cannot check if individual files are single or multi. It would take ages (I guess).
Is there a solution for solving this problem besides checking every file? Maybe my pipeline is not quite suitable for what I want to do.

**Tags:** nanopore, fast5
> **Comments:**
> **username_1**: Can you please show the first few filenames in each folder? e.g. by running the following command: `find mixed -type 'd' | while read dname; do echo "** ${dname} **"; tree -L 1 ${dname} | head; done`  
> **username_0**: Thank you for your reply! I included the first few filenames in each folder @gringer  
> **username_1**: The image seems like it's annotated incorrectly. A file with a channel and read number should be a single fast5 file, not a multi fast5 file.  
> **username_0**: Yes, the 2nd label (multi fast5 files with 4000 reads) and the 3rd label (single fast5 files) should be swapped. @gringer  
> **username_1**: Okay, that's good to know; thanks for the clarification.  

---

## 1 answers

### Answer by username_1 on 2023-12-10. Score: 1 | Accepted answer

It looks like you're creating too much additional work for yourself. What I'm trying to lead to with my comments is that the file names should help in working out if a file represents single reads or multiple reads, in which case the single reads can be shifted into a separate folder.
Based on the file names that I would expect, a search for files containing '_read_[0-9]' should be sufficient to fish out the single fast5 files:
find . -name '*_read_[0-9]*' > single_reads.txt
mkdir -p single_reads
cat single_reads.txt | while read fname; do mv -i ${fname} single_reads; done

[That single_reads.txt file could be split into multiple files if directory entries get too large.]
After which you should have a single folder that contains thousands of single-read fast5 files, and other folders that contain multi-fast5 files, removing the problem of needing to deal with mixed file types.

> **Comments:**
> **username_0**: I am gonna try this, thank you! @gringer  

---

