<p><em><strong>Summarising</strong></em></p>
<ol>
<li>@Jesse's comment I think is the right answer, open the database outside parallelisation. What I didn't thoroughly describe (and what was essentially requested) is the architecture, the alignments entering into an entire OOP library and the whole library is parallelised, i.e. every new alignment entering library is on its own CPU. The answer is obvious: instantiate the OOP library with the entire database and thats a key purpose of the design.</li>
<li>The technical answer is <code>joblib</code>. This is built to cache across parallelised archetechture as the <code>joblib</code> team point out <strong><a href="https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache">here</a></strong>. The criticism is that there's parallelisation within parallelisation. The discussion* is as much in the comments as in the response.</li>
<li>@user172818 response is correct too, the algorithm would benefit from C. Its a lot of code to rewrite in <code>C</code>, but sidestepping via Cython is a future development (its a hybrid language connecting C and Python).</li>
</ol>
<p>*, <a href="https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache">https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache</a></p>
<hr />
<p>There was a comment hinting this is Stack Overflow, that's difficult. Ultimately the technical answer (which is Stack Overflow) isn't the correct answer as far as I understand OOP.</p>
<hr />
<p><em><strong>Feedback</strong></em></p>
<p>Just to wrap this, point 1. worked fine, thank you. It means a database sits on each processor in parallelisation, but is imported  once. In truth, it worked nicely and provided cleaner code. Re-writing to instantiate once would affect other stuff so would be more costly than a cost-saving.</p>
<p>Python can be very fast (e.g. numpy), but also really slow (hence it has critiques), extensively parallelising helps bridge the gap.
Python parallelisation has critiques which I strongly, strongly believe are unfounded: I parallelise loads via <code>multiprocessor</code> library no problems.</p>
<p>Caching here, i.e. parallelising across a parallelisation could stretch things too far.</p>
<p><em><strong>Last point</strong></em></p>
<p>Point 3, re-writing in Cython/<code>C</code>, is a greater priority than the testing required for parallelised caching: can't parallelise all data and that code is better as <code>C</code>.</p>

CC BY-SA 4.0
2022-06-29T02:45:50.883Z
2022-07-02T19:46:31.090Z
2022-07-02T19:46:31.090Z
M__
<p>Currently algorithmically developing a calculation for &quot;co-dependent&quot; mutation (quotation appropriate). The algorithm uses chunked biopython alignment objects which are parallelized.</p>
<p><strong>Aim</strong>
One optization is caching a database within parallelization, but I ain't sure whether the parallelization duplicates the cache.</p>
<p><strong>Method</strong>
Each alignment chunk (biopython object) is passed onto a separate CPU and refers back to the original database to obtain its metadata in the final stage of the calculation (dragging metadata around this algorithm makes no sense).</p>
<p>The idea is to cache the database to prevent pulling it in from the harddrive:</p>
<pre><code>from multiprocessor import Pool
from myOOPlibrary import mystuff    
</code></pre>
<p>Facade design, the parallelised index to the alignment chunks enter into the myOOPlibrary:</p>
<pre><code>from cachetools import cached, TTLCache

cache = TTLCache(maxsize=1, ttl=500)

Class mystuff (stuff2):
... # init omitted

    def mymethod(self, ..):
    # to be updated 
    
Class stuff2 ():
... # init omitted

    @cached(cache)
    def getDB(self, ...):
.... #  pull it in (harddrive) and parse to a dictionary
    return metadataDB
</code></pre>
<p><strong>Question</strong>
I'm wondering if caching is the right approach in parallelization, specifically whether this will end up with multiplying the database held in RAM...</p>
<p><strong>Alternatives</strong>
The alternativees to caching are:</p>
<ul>
<li>IO to JSON (not cool)</li>
<li>IO via pickle (binary IO): a JSON will readily<code>pickle</code></li>
</ul>
<p>The possible advantage of the alternativees are the database is written once and accessed by all cores for certain.</p>
<p><strong>Underlying question</strong></p>
<ul>
<li>When caching is the database  cached for each core? If so the RAM overhead becomes an issue because the database is duplicated <code>total CPUs - 1</code> times</li>
<li>Alternatively is the cache shared between CPUs centrally and each CPU refers to this location?</li>
</ul>

CC BY-SA 4.0
2022-06-28T13:40:22.830Z
As currently written, this might be more appropriate for Stack Overflow
Chris_Rands
CC BY-SA 4.0
2022-06-28T14:32:23.137Z
Possibly but parallelization of biopython objects must be fairly common in algorithm/pipeline development. Alot work with biopython here
M__
CC BY-SA 4.0
2022-06-28T15:59:47.633Z
What are you using for the parallelization part?  If it's just the built-in threading module, I don't think anything will be duplicated there, you just have to make sure you don't step on your own toes with any read/write aspects (e.g. use something like Queue objects if needed).  But in that case, why using a caching library at all, versus just a single shared DB object you load once?  (More of the rest of the code might help clarify.)
Jesse
CC BY-SA 4.0
2022-06-28T23:28:30.517Z
Updated, @Jesse comment is cool and closes in on an easy solution.
M__
CC BY-SA 4.0
2022-06-28T23:37:01.973Z
If python code is the bottleneck, rewriting in C/C++ may get you a 50-fold speedup.
user172818
CC BY-SA 4.0
2022-06-28T23:45:20.243Z
Thanks, full Cython coding is the next development (C within Python and compiling into C). You're 100% right @user172818 what I've not described is there's extensive looping and that will be speeded up via C. For numpy there's no gain in Python (which I'm sure you know). Coding in raw C ... lots core-dumps and not quite so enthusiastic.
M__
<python><covid-19><algorithms><mutations>
Caching and parallelization
# Caching and parallelization

(Asked by: username_0 on 2022-06-28)

Currently algorithmically developing a calculation for "co-dependent" mutation (quotation appropriate). The algorithm uses chunked biopython alignment objects which are parallelized.
Aim
One optization is caching a database within parallelization, but I ain't sure whether the parallelization duplicates the cache.
Method
Each alignment chunk (biopython object) is passed onto a separate CPU and refers back to the original database to obtain its metadata in the final stage of the calculation (dragging metadata around this algorithm makes no sense).
The idea is to cache the database to prevent pulling it in from the harddrive:
from multiprocessor import Pool
from myOOPlibrary import mystuff    

Facade design, the parallelised index to the alignment chunks enter into the myOOPlibrary:
from cachetools import cached, TTLCache

cache = TTLCache(maxsize=1, ttl=500)

Class mystuff (stuff2):
... # init omitted

    def mymethod(self, ..):
    # to be updated 
    
Class stuff2 ():
... # init omitted

    @cached(cache)
    def getDB(self, ...):
.... #  pull it in (harddrive) and parse to a dictionary
    return metadataDB

Question
I'm wondering if caching is the right approach in parallelization, specifically whether this will end up with multiplying the database held in RAM...
Alternatives
The alternativees to caching are:

IO to JSON (not cool)
IO via pickle (binary IO): a JSON will readilypickle

The possible advantage of the alternativees are the database is written once and accessed by all cores for certain.
Underlying question

When caching is the database  cached for each core? If so the RAM overhead becomes an issue because the database is duplicated total CPUs - 1 times
Alternatively is the cache shared between CPUs centrally and each CPU refers to this location?

**Tags:** python, covid-19, algorithms, mutations
> **Comments:**
> **username_1**: As currently written, this might be more appropriate for Stack Overflow  
> **username_0**: Possibly but parallelization of biopython objects must be fairly common in algorithm/pipeline development. Alot work with biopython here  
> **username_2**: What are you using for the parallelization part?  If it's just the built-in threading module, I don't think anything will be duplicated there, you just have to make sure you don't step on your own toes with any read/write aspects (e.g. use something like Queue objects if needed).  But in that case, why using a caching library at all, versus just a single shared DB object you load once?  (More of the rest of the code might help clarify.)  
> **username_0**: Updated, @Jesse comment is cool and closes in on an easy solution.  
> **username_3**: If python code is the bottleneck, rewriting in C/C++ may get you a 50-fold speedup.  
> **username_0**: Thanks, full Cython coding is the next development (C within Python and compiling into C). You're 100% right @user172818 what I've not described is there's extensive looping and that will be speeded up via C. For numpy there's no gain in Python (which I'm sure you know). Coding in raw C ... lots core-dumps and not quite so enthusiastic.  

---

## 1 answers

### Answer by username_0 on 2022-06-29. Score: 1 | Accepted answer

***Summarising***

1. @Jesse's comment I think is the right answer, open the database outside parallelisation. What I didn't thoroughly describe (and what was essentially requested) is the architecture, the alignments entering into an entire OOP library and the whole library is parallelised, i.e. every new alignment entering library is on its own CPU. The answer is obvious: instantiate the OOP library with the entire database and thats a key purpose of the design.
2. The technical answer is `joblib`. This is built to cache across parallelised archetechture as the `joblib` team point out **[here](https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache)**. The criticism is that there's parallelisation within parallelisation. The discussion\* is as much in the comments as in the response.
3. @user172818 response is correct too, the algorithm would benefit from C. Its a lot of code to rewrite in `C`, but sidestepping via Cython is a future development (its a hybrid language connecting C and Python).

\*, <https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache>

---

There was a comment hinting this is Stack Overflow, that's difficult. Ultimately the technical answer (which is Stack Overflow) isn't the correct answer as far as I understand OOP.

---

***Feedback***

Just to wrap this, point 1. worked fine, thank you. It means a database sits on each processor in parallelisation, but is imported once. In truth, it worked nicely and provided cleaner code. Re-writing to instantiate once would affect other stuff so would be more costly than a cost-saving.

Python can be very fast (e.g. numpy), but also really slow (hence it has critiques), extensively parallelising helps bridge the gap.
Python parallelisation has critiques which I strongly, strongly believe are unfounded: I parallelise loads via `multiprocessor` library no problems.

Caching here, i.e. parallelising across a parallelisation could stretch things too far.

***Last point***

Point 3, re-writing in Cython/`C`, is a greater priority than the testing required for parallelised caching: can't parallelise all data and that code is better as `C`.


---

