<p>I have a vector of 200 million numbers. It has been running for over 24 hours and still not done. </p>

<pre><code>x &lt;- data   # the target sequence
k &lt;- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
ptm &lt;- proc.time()
y &lt;- convolve(x,k,type="open")
proc.time()-ptm
</code></pre>

<p>How to parallelize the <a href="https://www.rdocumentation.org/packages/stats/versions/3.6.1/topics/convolve" rel="nofollow noreferrer">convolve</a> function?
To parallelize, I tried splitting up the vector into pieces:</p>

<pre><code>&gt; x &lt;- data[1:1e7]
&gt; ptm &lt;- proc.time()
&gt; y &lt;- convolve(x,k,type="open") # smoothed sequence
&gt; proc.time()-ptm
   user  system elapsed 
 40.743   0.206  40.940 
&gt; x &lt;- data[1e7+1:2e7]
&gt; ptm &lt;- proc.time()
&gt; y &lt;- convolve(x,k,type="open") # smoothed sequence
&gt; proc.time()-ptm
   user  system elapsed 
320.193   0.708 320.871 
</code></pre>

<p>However, splitting it up and then adding back the convolutions will be tricky. 
Does anyone have other ideas?</p>

<p>This data is based on biological data. We want to cluster values in the vector based on nearby vector indices.</p>

<p>Reproducible example with 20,000,000 random numbers should finish in about 5.5-6 minutes. But if you try running 200,000,000 locally or on HPC, it will take a long time (my HPC has limit 99 hours and it timed out at 99 hours unfinished).</p>

<pre><code>win &lt;- 100        # window length of gaussian kernel (units acc to orig sequence)
sig &lt;- 5        # std deviation of gaussian
k &lt;- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
l &lt;- 20000000
x &lt;- round(runif(l, 0.0, 1.0),1)   # the target sequence with peaks
t &lt;- seq(0,1-1/length(x),1/length(x))
ptm &lt;- proc.time()
y &lt;- convolve(x,k,type="open")[(win+1):(win+length(x))] # smoothed sequence
proc.time()-ptm
</code></pre>

CC BY-SA 4.0
2019-08-31T15:38:53.633Z
Is the function vectorized? Did you try the same function with less numbers? If you need to make a cluster, why do you need to make the convolution? Perhaps you will get the attention of those that could help you better on stackoverflow.com.
llrs
CC BY-SA 4.0
2019-09-03T13:14:45.453Z
Hi IIrs: The convolve function is a function in R. The input data (x <- data) is a vector. We need to group based on nearby indices in the vector, so we cannot use other clustering methods that were built, for example, clustering gene expression. I can re-post in stackoverflow, but I have heard re-posting the same question is not good (and I am new to the stackoverflow community). I can try re-posting though. Thank you.
user11885521
CC BY-SA 4.0
2019-09-03T15:07:44.577Z
What kind of vector is it? A list or a numeric one? Could you please post a minimal reproducible example (with faked data if needed)?
llrs
CC BY-SA 4.0
2019-09-03T15:59:23.163Z
Hi IIrs, It is a numeric vector. I've troubleshooted and number of decimal places does not affect time to finish. I have edited the post above with a reproducible example with 20,000,000 random numbers. The time to finish with random numbers matches the time to finish with subsetted real data.
user11885521
CC BY-SA 4.0
2019-09-03T18:56:12.743Z
I don't know what is supposed to do but maybe the problem is that there isn't any continuous periodicity to use fft on it??
llrs
<r>
How to run function "convolve" in parallel in R?
# How to run function "convolve" in parallel in R?

(Asked by: username_0 on 2019-08-30)

I have a vector of 200 million numbers. It has been running for over 24 hours and still not done. 
x <- data   # the target sequence
k <- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
ptm <- proc.time()
y <- convolve(x,k,type="open")
proc.time()-ptm

How to parallelize the convolve function?
To parallelize, I tried splitting up the vector into pieces:
> x <- data[1:1e7]
> ptm <- proc.time()
> y <- convolve(x,k,type="open") # smoothed sequence
> proc.time()-ptm
   user  system elapsed 
 40.743   0.206  40.940 
> x <- data[1e7+1:2e7]
> ptm <- proc.time()
> y <- convolve(x,k,type="open") # smoothed sequence
> proc.time()-ptm
   user  system elapsed 
320.193   0.708 320.871 

However, splitting it up and then adding back the convolutions will be tricky. 
Does anyone have other ideas?
This data is based on biological data. We want to cluster values in the vector based on nearby vector indices.
Reproducible example with 20,000,000 random numbers should finish in about 5.5-6 minutes. But if you try running 200,000,000 locally or on HPC, it will take a long time (my HPC has limit 99 hours and it timed out at 99 hours unfinished).
win <- 100        # window length of gaussian kernel (units acc to orig sequence)
sig <- 5        # std deviation of gaussian
k <- exp(-(-win:win)^2/(2*sig^2))/(sig*sqrt(2*pi))  # gaussian kernel
l <- 20000000
x <- round(runif(l, 0.0, 1.0),1)   # the target sequence with peaks
t <- seq(0,1-1/length(x),1/length(x))
ptm <- proc.time()
y <- convolve(x,k,type="open")[(win+1):(win+length(x))] # smoothed sequence
proc.time()-ptm

**Tags:** r
> **Comments:**
> **username_1**: Is the function vectorized? Did you try the same function with less numbers? If you need to make a cluster, why do you need to make the convolution? Perhaps you will get the attention of those that could help you better on stackoverflow.com.  
> **username_0**: Hi IIrs: The convolve function is a function in R. The input data (x <- data) is a vector. We need to group based on nearby indices in the vector, so we cannot use other clustering methods that were built, for example, clustering gene expression. I can re-post in stackoverflow, but I have heard re-posting the same question is not good (and I am new to the stackoverflow community). I can try re-posting though. Thank you.  
> **username_1**: What kind of vector is it? A list or a numeric one? Could you please post a minimal reproducible example (with faked data if needed)?  
> **username_0**: Hi IIrs, It is a numeric vector. I've troubleshooted and number of decimal places does not affect time to finish. I have edited the post above with a reproducible example with 20,000,000 random numbers. The time to finish with random numbers matches the time to finish with subsetted real data.  
> **username_1**: I don't know what is supposed to do but maybe the problem is that there isn't any continuous periodicity to use fft on it??  

---

## 0 answers

