<p>You might want to look at the old programming language <a href="http://www.cs.cmu.edu/~scandal/nesl.html">NESL</a> which took these ideas seriously. The language is based on operations on collections, essentially lists and lists of lists and so forth, but I think trees and graphs were also considered, via tricky encodings. Some of the research done in that project (in the mid 1990s) could help answer your question. The PhD thesis (available as a book) Guy E. Blelloch. <em>Vector Models for Data-Parallel Computing.</em> MIT Press, 1990 may provide some answers. It was some time ago since I looked at it.</p>

<p>Work done on the <em><a href="http://en.wikipedia.org/wiki/Bird-Meertens_Formalism">Bird-Meertens Formalism</a></em> (BMF) falls into this category too, as did the language <a href="http://en.wikipedia.org/wiki/Charity_%28programming_language%29">Charity</a>. From the Charity wikipedia page it says that the language is not Turing complete, but can express Ackermann's function, which means that it's more than primitive recursive. Both BMF and Charity involve operators like folds and scans (catamorphisms, anamorphisms, etc), and they have their roots in Category Theory.</p>

<p>I short, imprecise answer is that you can express quite a lot.</p>

CC BY-SA 2.5
2010-09-30T04:57:14.567Z
NESL isn't a total language.
Per Vognsen
CC BY-SA 2.5
2010-09-30T05:16:40.173Z
I've been superficially aware of NESL for a while but just read one of Blelloch's papers in detail for the first time. Thanks for the tip. NESL is pretty similar to the language I've described above except that, as Per Vognsen noticed, it allows recursion.
Alex Rubinsteyn
CC BY-SA 2.5
2010-09-30T05:31:44.417Z
I'm also interested in Blelloch's choice of primitive operators: map, dist (I believe same as what I called 'fill'), length, array-read, array-write, scan, partition, flatten. Are NESL's primitives "complete", or is there some other operation with a data parallel implementation which can't be efficiently expressed using these?
Alex Rubinsteyn
CC BY-SA 2.5
2010-09-30T06:41:36.287Z
Remove the recursion, then you have a pretty expressive language, especially if you consider folds and so forth. Looking at BMF and the work following it might be more of interest, then. I'm sorry, but I'm not up to date in this area.
Dave Clarke
CC BY-SA 2.5
2010-09-30T04:51:57.287Z
2010-09-30T04:51:57.287Z
Dave Clarke
<p>My other answer pointed out flaws in the language as it stands. In this answer I will give more detail on the issues with the coexistence of folds and unfolds in a total language. Then I will propose a resolution and show that all problems in P (and indeed many more) can be solved in this extended language.</p>

<p>Folding in a total language consumes lists:</p>

<pre><code>fold :: (a -&gt; b -&gt; b) -&gt; b -&gt; List a -&gt; b
</code></pre>

<p>Unfolding in a total language generates streams, which are potentially unbounded:</p>

<pre><code>unfold :: (b -&gt; Maybe (a, b)) -&gt; b -&gt; Stream a
</code></pre>

<p>Unfortunately, lists and streams live in different worlds, so these operators cannot be composed. We need a partial correspondence:</p>

<pre><code>stream :: List a -&gt; Stream a
list :: Int -&gt; Stream a -&gt; List a
</code></pre>

<p>The stream operator embeds a list into a bounded stream. The list function extracts the first n elements (or fewer if the stream terminates earlier) into a list. We thus have the following equation:</p>

<pre><code>for all xs :: List a, xs == list (length xs) (stream xs)
</code></pre>

<p>As an efficiency optimization we can entirely cut out streams as an intermediate data structure:</p>

<pre><code>unfoldList :: Int -&gt; (b -&gt; Maybe (a, b)) -&gt; b -&gt; List a
</code></pre>

<p>I will now sketch a proof that this (with the other operators already implied in the original question) lets us simulate any polynomial-time algorithm.</p>

<p>By definition, a language L is in P when there is a Turing machine M and a polynomial p such that membership of x in L can be decided by running M at most p(n) iterations where n = |x|. By a standard argument, the state of the machine's tape in iteration k can be encoded with a list of length at most 2k + 1, even though the machine's tape is infinite. </p>

<p>The idea is now to represent M's transition as a function from lists to lists. The execution of the machine will be done by unfolding the initial state with the transition function. This generates a stream of lists. The assumption that L is in P means that we need look no further than p(n) elements into the stream. Thus we can compose the unfolding with <code>list p(n)</code> to get a finite list. Finally, we fold over it to check whether the answer to the decision problem was yes or no.</p>

<p>More generally, this shows that whenever we have an algorithm whose termination time can be bounded by a function computable in the language, we can simulate it. This also suggests why something like Ackermann's function can't be simulated: it is its own bound, so there is a chicken and egg problem. </p>

CC BY-SA 2.5
2010-09-30T05:59:11.280Z
2010-09-30T11:08:20.650Z
2010-09-30T11:08:20.650Z
Per Vognsen
<p>It's a very gimped language. Try programming the factorial function:</p>

<pre><code>fact 0 = 1
fact n = n * fact (n-1)
</code></pre>

<p>The problem is that your language only has folds but no unfolds. The natural way of expressing factorial is to compose an unfolding of n into the list [1, 2, ..., n] with the folding that tears it down while multiplying.</p>

<p>Are you really interested in this specific language or in total functional programming in general? It's obvious that your language can at most express polynomial-time algorithms. System F (polymorphic lambda calculus) can express monsters like Ackermann's function with ease. Even if your interest is in polynomial-time algorithms, you frequently need the super-polynomial elbow room to express them naturally.</p>

<p>Edit: As Dave points out, NESL is one of the seminal functional data-parallel programming languages but it's very far from total (it doesn't even try). The APL family had a parallel track of evolution and has a total algebraic subset (avoid the fixed-point operators). If the focus of your question is totalness, David Turner has written some good papers in this area although not specifically on data-parallel programming.</p>

CC BY-SA 2.5
2010-09-30T04:50:54.013Z
Sorry, the lack of unfolding operators is an oversight on my part...I meant to add one but forgot to put it in the post. I'm not interested in this specific language necessarily but rather the expressiveness and limits of data parallel computation.
Alex Rubinsteyn
CC BY-SA 2.5
2010-09-30T04:55:47.107Z
Unfolding is naturally stream-valued, not array-valued, which is a basic problem with corecursion in total strict languages.
Per Vognsen
CC BY-SA 2.5
2010-09-30T04:40:04.970Z
2010-09-30T05:08:10.023Z
2010-09-30T05:08:10.023Z
Per Vognsen
<p>Imagine a functional programming language whose only data types are numerical scalars and arbitrary nestings of arrays. The language lacks any means of unbounded iteration, so the following are disallowed:</p>

<ul>
<li>explicit loops (not much use without side-effects anyhow)</li>
<li>recursion</li>
<li>arbitrary first-class functions (no y-combinator)</li>
</ul>

<p>The language does, however, have:</p>

<ul>
<li>top-level functions</li>
<li>lexically scoped let bindings</li>
<li>branching control flow</li>
<li>common scalar math and logic functions</li>
<li>some simple array constructor like fill(n,x) which creates an n-element array with identical values x</li>
<li>most importantly: a restricted set of higher-order operators which perform parallel structured iteration (such as map, reduce, scan, all-pairs).</li>
</ul>

<p>To be more specific about the data parallel operators: </p>

<ul>
<li>y = map(f,x) => y[i] = f[i] </li>
<li>y = reduce(f, a, x) => y = f(a, f(y[p[0]], f(y[p[1]], ...))) for some permutation p</li>
<li>y = scan(f, a, x) => y[i] = reduce(f, a, y[0...i-1])  </li>
<li>y = allpairs(f, x, y) => y[i,j] = f(x[i], y[j])</li>
</ul>

<p>We could have other operators as well, but to qualify they should have polynomial running time, be implementable under some reasonable model of data parallel computation, and use at most polynomial space. </p>

<p>There are obviously some constructs which can't be expressed in this language, such as:</p>

<pre><code>while f(x) &gt; tol:
    x &lt;- update(x)   
</code></pre>

<p>What can we express in this system? Are we limited only to search problems in FP? Can we capture all polynomial time algorithms? Also, is there some minimal set of operators for this class?</p>

<complexity-classes><functional-programming><dc.parallel-comp>
Which algorithms can be expressed using a total functional language with data parallel operators?
Alex Rubinsteyn
# Which algorithms can be expressed using a total functional language with data parallel operators?

(Asked by: username_0 on 2010-09-30)

Imagine a functional programming language whose only data types are numerical scalars and arbitrary nestings of arrays. The language lacks any means of unbounded iteration, so the following are disallowed:

explicit loops (not much use without side-effects anyhow)
recursion
arbitrary first-class functions (no y-combinator)

The language does, however, have:

top-level functions
lexically scoped let bindings
branching control flow
common scalar math and logic functions
some simple array constructor like fill(n,x) which creates an n-element array with identical values x
most importantly: a restricted set of higher-order operators which perform parallel structured iteration (such as map, reduce, scan, all-pairs).

To be more specific about the data parallel operators: 

y = map(f,x) => y[i] = f[i] 
y = reduce(f, a, x) => y = f(a, f(y[p[0]], f(y[p[1]], ...))) for some permutation p
y = scan(f, a, x) => y[i] = reduce(f, a, y[0...i-1])  
y = allpairs(f, x, y) => y[i,j] = f(x[i], y[j])

We could have other operators as well, but to qualify they should have polynomial running time, be implementable under some reasonable model of data parallel computation, and use at most polynomial space. 
There are obviously some constructs which can't be expressed in this language, such as:
while f(x) > tol:
    x <- update(x)   

What can we express in this system? Are we limited only to search problems in FP? Can we capture all polynomial time algorithms? Also, is there some minimal set of operators for this class?

**Tags:** complexity-classes, functional-programming, dc.parallel-comp


---

## 3 answers

### Answer by username_1 on 2010-09-30. Score: 7

You might want to look at the old programming language [NESL](http://www.cs.cmu.edu/~scandal/nesl.html) which took these ideas seriously. The language is based on operations on collections, essentially lists and lists of lists and so forth, but I think trees and graphs were also considered, via tricky encodings. Some of the research done in that project (in the mid 1990s) could help answer your question. The PhD thesis (available as a book) Guy E. Blelloch. *Vector Models for Data-Parallel Computing.* MIT Press, 1990 may provide some answers. It was some time ago since I looked at it.

Work done on the *[Bird-Meertens Formalism](http://en.wikipedia.org/wiki/Bird-Meertens_Formalism)* (BMF) falls into this category too, as did the language [Charity](http://en.wikipedia.org/wiki/Charity_%28programming_language%29). From the Charity wikipedia page it says that the language is not Turing complete, but can express Ackermann's function, which means that it's more than primitive recursive. Both BMF and Charity involve operators like folds and scans (catamorphisms, anamorphisms, etc), and they have their roots in Category Theory.

I short, imprecise answer is that you can express quite a lot.

> **Comments:**
> **username_2**: NESL isn't a total language.  
> **username_0**: I've been superficially aware of NESL for a while but just read one of Blelloch's papers in detail for the first time. Thanks for the tip. NESL is pretty similar to the language I've described above except that, as username_2 noticed, it allows recursion.  
> **username_0**: I'm also interested in Blelloch's choice of primitive operators: map, dist (I believe same as what I called 'fill'), length, array-read, array-write, scan, partition, flatten. Are NESL's primitives "complete", or is there some other operation with a data parallel implementation which can't be efficiently expressed using these?  
> **username_1**: Remove the recursion, then you have a pretty expressive language, especially if you consider folds and so forth. Looking at BMF and the work following it might be more of interest, then. I'm sorry, but I'm not up to date in this area.  

---

### Answer by username_2 on 2010-09-30. Score: 7

My other answer pointed out flaws in the language as it stands. In this answer I will give more detail on the issues with the coexistence of folds and unfolds in a total language. Then I will propose a resolution and show that all problems in P (and indeed many more) can be solved in this extended language.
Folding in a total language consumes lists:
fold :: (a -> b -> b) -> b -> List a -> b

Unfolding in a total language generates streams, which are potentially unbounded:
unfold :: (b -> Maybe (a, b)) -> b -> Stream a

Unfortunately, lists and streams live in different worlds, so these operators cannot be composed. We need a partial correspondence:
stream :: List a -> Stream a
list :: Int -> Stream a -> List a

The stream operator embeds a list into a bounded stream. The list function extracts the first n elements (or fewer if the stream terminates earlier) into a list. We thus have the following equation:
for all xs :: List a, xs == list (length xs) (stream xs)

As an efficiency optimization we can entirely cut out streams as an intermediate data structure:
unfoldList :: Int -> (b -> Maybe (a, b)) -> b -> List a

I will now sketch a proof that this (with the other operators already implied in the original question) lets us simulate any polynomial-time algorithm.
By definition, a language L is in P when there is a Turing machine M and a polynomial p such that membership of x in L can be decided by running M at most p(n) iterations where n = |x|. By a standard argument, the state of the machine's tape in iteration k can be encoded with a list of length at most 2k + 1, even though the machine's tape is infinite. 
The idea is now to represent M's transition as a function from lists to lists. The execution of the machine will be done by unfolding the initial state with the transition function. This generates a stream of lists. The assumption that L is in P means that we need look no further than p(n) elements into the stream. Thus we can compose the unfolding with list p(n) to get a finite list. Finally, we fold over it to check whether the answer to the decision problem was yes or no.
More generally, this shows that whenever we have an algorithm whose termination time can be bounded by a function computable in the language, we can simulate it. This also suggests why something like Ackermann's function can't be simulated: it is its own bound, so there is a chicken and egg problem.


---

### Answer by username_2 on 2010-09-30. Score: 4

It's a very gimped language. Try programming the factorial function:
fact 0 = 1
fact n = n * fact (n-1)

The problem is that your language only has folds but no unfolds. The natural way of expressing factorial is to compose an unfolding of n into the list [1, 2, ..., n] with the folding that tears it down while multiplying.
Are you really interested in this specific language or in total functional programming in general? It's obvious that your language can at most express polynomial-time algorithms. System F (polymorphic lambda calculus) can express monsters like Ackermann's function with ease. Even if your interest is in polynomial-time algorithms, you frequently need the super-polynomial elbow room to express them naturally.
Edit: As Dave points out, NESL is one of the seminal functional data-parallel programming languages but it's very far from total (it doesn't even try). The APL family had a parallel track of evolution and has a total algebraic subset (avoid the fixed-point operators). If the focus of your question is totalness, David Turner has written some good papers in this area although not specifically on data-parallel programming.

> **Comments:**
> **username_0**: Sorry, the lack of unfolding operators is an oversight on my part...I meant to add one but forgot to put it in the post. I'm not interested in this specific language necessarily but rather the expressiveness and limits of data parallel computation.  
> **username_2**: Unfolding is naturally stream-valued, not array-valued, which is a basic problem with corecursion in total strict languages.  

---

