<p>Assuming you are on an operating system that supports them, like Linux, you can try using <a href="https://en.wikipedia.org/wiki/Named_pipe" rel="nofollow noreferrer">named pipes</a> in a ramdisk or <a href="https://en.wikipedia.org/wiki/Tmpfs" rel="nofollow noreferrer">tmpfs</a>. Not the most elegant approach, but I don't know if what you are asking for is possible (it may very well be, I just don't know) and this can be a decent workaround.</p>
<p>First, check that you do have a mounted tmpfs. On my system, I have a few:</p>
<pre><code>$ mount | awk '$1==&quot;tmpfs&quot;'
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16278892k,nr_inodes=1048576,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3255776k,nr_inodes=813944,mode=700,uid=1000,gid=1000,inode64)
</code></pre>
<p>These filesystems are actually stored in RAM and should be significantly faster. For instance, on my machine, which has an SSD as the main storage, I still get significant speedup when using <code>/dev/shm</code>:</p>
<pre><code>## SSD
$ time seq 1000000000 &gt; file
real    0m13.687s
user    0m5.623s
sys     0m7.516s

$ time wc -l &lt; file
1000000000

real    0m4.972s
user    0m0.101s
sys     0m2.248s

## tmpfs, RAM
$ time seq 1000000000 &gt; /dev/shm/file

real    0m6.755s
user    0m4.982s
sys     0m1.763s

$ time wc -l &lt; /dev/shm/file
1000000000

real    0m1.024s
user    0m0.120s
sys     0m0.902s
</code></pre>
<p>As you can see, using <code>/dev/shm</code> is significantly faster. And using named pipes can give you a little boost too:</p>
<pre><code>$ time ( seq 1000000000 &gt; /dev/shm/file;  wc -l &lt; /dev/shm/file)
1000000000

real    0m8.341s
user    0m5.277s
sys     0m3.044s

<span class="math-container">$ rm /dev/shm/file &amp;&amp; mkfifo /dev/shm/file 
$</span> time ( seq 1000000000 &gt; /dev/shm/file &amp;  wc -l &lt; /dev/shm/file)
1000000000

real    0m6.564s
user    0m5.426s
sys     0m3.965s
</code></pre>
<p>So, this is essentially a way of doing everything in RAM without needing to code it as doing everything in RAM. You can create these files (the FIFOs, the named pipes) and treat them as files, but since you're creating them in a <code>tmpfs</code> filesystem, they're actually in RAM and by using FIFOs you not only get a further speed jump but also minimise the actual data that need to be stored.</p>

CC BY-SA 4.0
2023-10-16T00:35:57.857Z
This seems like it gets at what I'm trying to accomplish. I will try it! Thank you.
ekofman
CC BY-SA 4.0
2023-10-14T13:33:52.703Z
2023-10-16T09:20:47.210Z
2023-10-16T09:20:47.210Z
terdon
<p>My current job is rate-limited chiefly by the IO overhead of writing reads to their corresponding bam files (they are split up by both cell barcode and contig). The only reason I need to write these mini-split bams is so that I can then use the pysam coverage function to assess read depth at certain locations. I already have all of these reads in memory as a list of AlignedSegments. Is it necessary to write them to a .bam file in order to be able to index and use the coverage function? I feel like there should be a way to have the AlignedSegments be treated as a bam object in memory, possibly even indexed in memory, and then treated as if they were just streamed from a bamfile with an index in order to be able to use the coverage method. Thanks in advance, if this is possible it would increase my processing speed x10. Or, given a list of AlignedSegments, is there a way to get coverage at specific positions across all those AlignedSegments in my list (that would get me what I need).</p>

CC BY-SA 4.0
2023-10-14T02:05:31.577Z
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Community
CC BY-SA 4.0
2023-10-14T13:21:20.633Z
What operating system are you using?
terdon
CC BY-SA 4.0
2023-10-16T00:36:42.953Z
My operating system is linux
ekofman
<coverage><pysam><indexing>
Is it possible to, all in memory without writing a file, given a list of AlignedSegments and a bam header, build a bam, index and get coverage?
# Is it possible to, all in memory without writing a file, given a list of AlignedSegments and a bam header, build a bam, index and get coverage?

(Asked by: username_0 on 2023-10-14)

My current job is rate-limited chiefly by the IO overhead of writing reads to their corresponding bam files (they are split up by both cell barcode and contig). The only reason I need to write these mini-split bams is so that I can then use the pysam coverage function to assess read depth at certain locations. I already have all of these reads in memory as a list of AlignedSegments. Is it necessary to write them to a .bam file in order to be able to index and use the coverage function? I feel like there should be a way to have the AlignedSegments be treated as a bam object in memory, possibly even indexed in memory, and then treated as if they were just streamed from a bamfile with an index in order to be able to use the coverage method. Thanks in advance, if this is possible it would increase my processing speed x10. Or, given a list of AlignedSegments, is there a way to get coverage at specific positions across all those AlignedSegments in my list (that would get me what I need).

**Tags:** coverage, pysam, indexing
> **Comments:**
> **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
> **username_2**: What operating system are you using?  
> **username_0**: My operating system is linux  

---

## 1 answers

### Answer by username_2 on 2023-10-14. Score: 3 | Accepted answer

Assuming you are on an operating system that supports them, like Linux, you can try using named pipes in a ramdisk or tmpfs. Not the most elegant approach, but I don't know if what you are asking for is possible (it may very well be, I just don't know) and this can be a decent workaround.
First, check that you do have a mounted tmpfs. On my system, I have a few:
$ mount | awk '$1=="tmpfs"'
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16278892k,nr_inodes=1048576,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3255776k,nr_inodes=813944,mode=700,uid=1000,gid=1000,inode64)

These filesystems are actually stored in RAM and should be significantly faster. For instance, on my machine, which has an SSD as the main storage, I still get significant speedup when using /dev/shm:
## SSD
$ time seq 1000000000 > file
real    0m13.687s
user    0m5.623s
sys     0m7.516s

$ time wc -l < file
1000000000

real    0m4.972s
user    0m0.101s
sys     0m2.248s

## tmpfs, RAM
$ time seq 1000000000 > /dev/shm/file

real    0m6.755s
user    0m4.982s
sys     0m1.763s

$ time wc -l < /dev/shm/file
1000000000

real    0m1.024s
user    0m0.120s
sys     0m0.902s

As you can see, using /dev/shm is significantly faster. And using named pipes can give you a little boost too:
$ time ( seq 1000000000 > /dev/shm/file;  wc -l < /dev/shm/file)
1000000000

real    0m8.341s
user    0m5.277s
sys     0m3.044s

$ rm /dev/shm/file && mkfifo /dev/shm/file 
$ time ( seq 1000000000 > /dev/shm/file &  wc -l < /dev/shm/file)
1000000000

real    0m6.564s
user    0m5.426s
sys     0m3.965s

So, this is essentially a way of doing everything in RAM without needing to code it as doing everything in RAM. You can create these files (the FIFOs, the named pipes) and treat them as files, but since you're creating them in a tmpfs filesystem, they're actually in RAM and by using FIFOs you not only get a further speed jump but also minimise the actual data that need to be stored.

> **Comments:**
> **username_0**: This seems like it gets at what I'm trying to accomplish. I will try it! Thank you.  

---

