<p>I don't think one can make a blanket statement on whether primal decomposition methods or dual decomposition methods are more efficient. The efficiency is dictated by the structure of the problem; primal decomposition and dual decomposition are ways of exploiting this structure to develop efficient algorithms.</p>

<p>The key aspect in making a choice between these methods is the kind of parametrization the master brings about on the slaves. Fo e.g., sometimes it is convenient (in terms of theory or practice) to have the parametrization in the objective function of the slave problems instead of having it in the constraints. If the parametrization happens to appear in the constraints (and it has suitable structure), one can try to dualize the slave problem to get in the objective.</p>

<p>In the slides you have linked, the effort seems to be in exploiting the structure for parallelization of the solution of subproblems. For parallelization one would require subproblems that are independent of each other. If there is a natural separability in the constraints, dual variables can be used to parametrize the problem and thereby create such subproblems.</p>

<p><strong>Edit:</strong> I repeat once again that structure is important in determining the choice of the method. I am still not completely clear on the structure afforded by the problems you have in mind. Based on the information given, this is my best attempt at answering your question. </p>

<p>From the examples you have added, the problems appear to be what can be called "nearly separable". In essence, these problems appear to be variants of the problem $\min \{\sum_{i=1}^n f_i(x_i) : (x_1,...,x_n) \in K\}$, where $K$ is a constraint that has a separable structure. For example, the constraint $(x_1,...,x_n) \in K$ may be $\sum_ix_i =b$, where $b$ is a vector of appropriate size. By separable structure I mean that the optimal values of variables $x_1,...,x_n$ can be chosen independently of each other. If one looks at the problem as a whole, the objective functions are decoupled in individual variables, but because of the coupling in $K$ these variables cannot be independently optimized. However because this coupling in $K$ is mild, one can still salvage separability. Dual decomposition lets you separate the variables when they are coupled in the constraints. If the variables were coupled in the objective, one may have to employ primal decomposition (as in Boyd's slides). One may of course try to reformulate problems to bring them in some or the other form that one is comfortable using.</p>

<p>To demonstrate separation by dual decomposition, define the Lagrangian $L(x,\lambda) =\sum_i f_i(x_i) + \lambda^T(\sum_i x_i)$. If $(x,\lambda)$ is a saddle point of $L$, then $x$ solves the original problem. A dual decomposition method would for each value of $\lambda$, minimize $L(x,\lambda)$ over $x$, altering $\lambda$ until a saddle point is found. The minimization in $x$ is easier, in particular separable and parallelizable.</p>

CC BY-SA 3.0
2012-05-03T23:46:04.410Z
Thanks Ankur, I have updated the original post with a couple of examples. Dual decomposition methods are gaining quick popularity in the statistical machine learning literature, and I wonder if there is anything about their structure that makes them a better fit for dual decomposition. I think I have also read S. Boyd making a general statement about dual decomposition methods resulting in more efficient decompositions in practice than primal ones. Perhaps a large number of popular problems exhibit a structure that is more friendly for dual decomposition?
Amelio Vazquez-Reina
CC BY-SA 3.0
2012-05-04T00:04:24.803Z
I think you need to be more specific about the overarching class of structures/optimization problems you are referring to. I don't think something as definitive as "a large number of popular problems exhibit a structure that is more friendly for dual decomposition?" can be said without clarity on the kind of problems you have in mind.
Ankur
CC BY-SA 3.0
2012-05-04T13:52:21.660Z
Thanks @Ankur. [MAP-MRF](http://en.wikipedia.org/wiki/Markov_random_field) problems are relatively common in the bioinformatics, computer vision, and NLP literature. These problems are often formulated as quadratic pseudoboolean optimization problems, and there is a growing trend in the literature to solve these problems via dual decomposition. I have also seen authors pointing out a connection between [belief propagation](http://en.wikipedia.org/wiki/Belief_propagation) and other general [message passing methods](http://en.wikipedia.org/wiki/Variational_message_passing) to dual decomposition.
Amelio Vazquez-Reina
CC BY-SA 3.0
2012-05-05T21:03:46.497Z
I have tried answering your question by taking into account the examples you have cited.
Ankur
CC BY-SA 3.0
2012-05-06T16:35:07.863Z
Thanks @Ankur, this is very helpful. In my update of the original post I expanded more on what type of problems I am referring to: MAP-MRF with **binary and quadratic cost functions**. In other words the cost function is more of the form  argmin$[\sum a_{ij} x_i x_j  + \sum b_k x_k]$ (as opposed to just argmin$[\sum f(x_k)]$ as you wrote). However you are right about the structure of the constraints $K$, they are almost always of the form $\sum_i f(x_i,x_j) = g(x_j)$ (i.e. defining what's known as the **marginal polytope**)
Amelio Vazquez-Reina
CC BY-SA 3.0
2012-05-06T16:42:29.113Z
Has the problem you have written been solved by dual decomposition? Some clever reformulation may be required to put it in a form that is more suitable for dual decomposition.
Ankur
CC BY-SA 3.0
2012-05-06T16:52:46.713Z
The first two references I provided in my original post solve quadratic problems of the form I just wrote. For example, in the second reference I had in the original post, i.e. [(2)](http://people.csail.mit.edu/tommi/papers/SonGloJaa_optbook.pdf), the first equation in the article is quadratic; notice that the second term in the sum is defined on terms of the form $\theta_f (\mathbf{x}_f) = \theta_{ij}(xi, xj)$. These equations arise often in MRFs, where you have a cost function defined on the edges and nodes of a graph.
Amelio Vazquez-Reina
CC BY-SA 3.0
2012-05-03T23:27:03.913Z
2012-05-05T21:03:05.983Z
2012-05-05T21:03:05.983Z
Ankur
<p>I noticed that dual decomposition methods tend to be preferred over primal ones in the large scale optimization literature (here are some examples: <a href="http://www.csd.uoc.gr/~komod/publications/docs/DualDecomposition_PAMI.pdf" rel="nofollow">(1)</a>, <a href="http://people.csail.mit.edu/tommi/papers/SonGloJaa_optbook.pdf" rel="nofollow">(2)</a>). The reason seems to be, from what I understand, that dual decomposition methods often lead to more efficient algorithms.</p>

<p>After reading about decomposition methods on various sources (e.g. see <a href="http://see.stanford.edu/materials/lsocoee364b/08-decomposition_notes.pdf" rel="nofollow">these</a> notes by <a href="http://www.stanford.edu/~boyd/" rel="nofollow">Stephen Boyd</a>), I don't get the intuition of why this would be the case.</p>

<p>Perhaps an important difference is that, as S. Boyd notes in his <a href="http://www.ee.kth.se/~mikaelj/dopt/slides/decomposition_slides.pdf" rel="nofollow">slides</a>, in primal decomposition methods:</p>

<blockquote>
  <p>the master problem manipulates the primal (complicating) variables</p>
</blockquote>

<p>while in the dual decomposition methods, this would not be the case. However, when S. Boyd describes the procedures for primal vs dual methods, the decomposition looks essentially the same:</p>

<pre><code>repeat
   solve the subproblems (presumably on the slaves)
   update complicating variable (presumably on the master)
</code></pre>

<p><strong>So my questions are:</strong></p>

<ul>
<li>Is it true that dual decomposition methods tend to be more efficient in practice than primal methods?  If so, why?</li>
<li>Why does S. Boyd say that in primal decomposition methods, "the master problem manipulates the complicating variables" ? How is this different from the way these variables are handled in a dual decomposition?</li>
</ul>

<h3>Update: Additional background:</h3>

<p>Statistical machine learning problems, such as <a href="http://en.wikipedia.org/wiki/Markov_random_field" rel="nofollow">MAP-MRF</a> estimation, arise often in bioinformatics, computer vision and NLP, and are usually formulated as quadratic pseudoboolean problems (i.e. mathematical programming problems with binary and quadratic cost functions) (e.g. <a href="http://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;ved=0CMMEEBYwAA&amp;url=http://www.robots.ox.ac.uk/~vilem/QPBOPI.pdf&amp;ei=m-OjT4CAOazG6AGBvvSlCQ&amp;usg=AFQjCNEjfI1q1HWd-W_g0iba4dvhw255-Q&amp;sig2=zwLC6j_h8VkpXhxHjoeXyA" rel="nofollow">(3)</a>).  </p>

<p>There seems to be a growing trend in the literature to solve these problems via dual decomposition (as pointed out by the two articles I linked to above: <a href="http://www.csd.uoc.gr/~komod/publications/docs/DualDecomposition_PAMI.pdf" rel="nofollow">(1)</a>, <a href="http://people.csail.mit.edu/tommi/papers/SonGloJaa_optbook.pdf" rel="nofollow">(2)</a>). Moreover, some of these authors have pointed out that there are connections between an important class of inference algorithms, such as <a href="http://en.wikipedia.org/wiki/Belief_propagation" rel="nofollow">belief propagation</a> and other general <a href="http://en.wikipedia.org/wiki/Variational_message_passing" rel="nofollow">message passing methods</a>, to dual decomposition <a href="http://cs.nyu.edu/~dsontag/papers/sontag_phd_thesis.pdf" rel="nofollow">(4)</a>.</p>

<ds.algorithms><optimization><primal-dual><convex-optimization><intuition>
Primal vs dual decomposition methods
Amelio Vazquez-Reina
# Primal vs dual decomposition methods

(Asked by: username_0 on 2012-05-03)

I noticed that dual decomposition methods tend to be preferred over primal ones in the large scale optimization literature (here are some examples: (1), (2)). The reason seems to be, from what I understand, that dual decomposition methods often lead to more efficient algorithms.
After reading about decomposition methods on various sources (e.g. see these notes by Stephen Boyd), I don't get the intuition of why this would be the case.
Perhaps an important difference is that, as S. Boyd notes in his slides, in primal decomposition methods:

the master problem manipulates the primal (complicating) variables

while in the dual decomposition methods, this would not be the case. However, when S. Boyd describes the procedures for primal vs dual methods, the decomposition looks essentially the same:
repeat
   solve the subproblems (presumably on the slaves)
   update complicating variable (presumably on the master)

So my questions are:

Is it true that dual decomposition methods tend to be more efficient in practice than primal methods?  If so, why?
Why does S. Boyd say that in primal decomposition methods, "the master problem manipulates the complicating variables" ? How is this different from the way these variables are handled in a dual decomposition?

Update: Additional background:
Statistical machine learning problems, such as MAP-MRF estimation, arise often in bioinformatics, computer vision and NLP, and are usually formulated as quadratic pseudoboolean problems (i.e. mathematical programming problems with binary and quadratic cost functions) (e.g. (3)).  
There seems to be a growing trend in the literature to solve these problems via dual decomposition (as pointed out by the two articles I linked to above: (1), (2)). Moreover, some of these authors have pointed out that there are connections between an important class of inference algorithms, such as belief propagation and other general message passing methods, to dual decomposition (4).

**Tags:** ds.algorithms, optimization, primal-dual, convex-optimization, intuition

---

## 1 answers

### Answer by username_1 on 2012-05-03. Score: 2

I don't think one can make a blanket statement on whether primal decomposition methods or dual decomposition methods are more efficient. The efficiency is dictated by the structure of the problem; primal decomposition and dual decomposition are ways of exploiting this structure to develop efficient algorithms.

The key aspect in making a choice between these methods is the kind of parametrization the master brings about on the slaves. Fo e.g., sometimes it is convenient (in terms of theory or practice) to have the parametrization in the objective function of the slave problems instead of having it in the constraints. If the parametrization happens to appear in the constraints (and it has suitable structure), one can try to dualize the slave problem to get in the objective.

In the slides you have linked, the effort seems to be in exploiting the structure for parallelization of the solution of subproblems. For parallelization one would require subproblems that are independent of each other. If there is a natural separability in the constraints, dual variables can be used to parametrize the problem and thereby create such subproblems.

**Edit:** I repeat once again that structure is important in determining the choice of the method. I am still not completely clear on the structure afforded by the problems you have in mind. Based on the information given, this is my best attempt at answering your question.

From the examples you have added, the problems appear to be what can be called "nearly separable". In essence, these problems appear to be variants of the problem $\min \{\sum\_{i=1}^n f\_i(x\_i) : (x\_1,...,x\_n) \in K\}$, where $K$ is a constraint that has a separable structure. For example, the constraint $(x\_1,...,x\_n) \in K$ may be $\sum\_ix\_i =b$, where $b$ is a vector of appropriate size. By separable structure I mean that the optimal values of variables $x\_1,...,x\_n$ can be chosen independently of each other. If one looks at the problem as a whole, the objective functions are decoupled in individual variables, but because of the coupling in $K$ these variables cannot be independently optimized. However because this coupling in $K$ is mild, one can still salvage separability. Dual decomposition lets you separate the variables when they are coupled in the constraints. If the variables were coupled in the objective, one may have to employ primal decomposition (as in Boyd's slides). One may of course try to reformulate problems to bring them in some or the other form that one is comfortable using.

To demonstrate separation by dual decomposition, define the Lagrangian $L(x,\lambda) =\sum\_i f\_i(x\_i) + \lambda^T(\sum\_i x\_i)$. If $(x,\lambda)$ is a saddle point of $L$, then $x$ solves the original problem. A dual decomposition method would for each value of $\lambda$, minimize $L(x,\lambda)$ over $x$, altering $\lambda$ until a saddle point is found. The minimization in $x$ is easier, in particular separable and parallelizable.

> **Comments:**
> **username_0**: Thanks Ankur, I have updated the original post with a couple of examples. Dual decomposition methods are gaining quick popularity in the statistical machine learning literature, and I wonder if there is anything about their structure that makes them a better fit for dual decomposition. I think I have also read S. Boyd making a general statement about dual decomposition methods resulting in more efficient decompositions in practice than primal ones. Perhaps a large number of popular problems exhibit a structure that is more friendly for dual decomposition?  
> **username_1**: I think you need to be more specific about the overarching class of structures/optimization problems you are referring to. I don't think something as definitive as "a large number of popular problems exhibit a structure that is more friendly for dual decomposition?" can be said without clarity on the kind of problems you have in mind.  
> **username_0**: Thanks @Ankur. [MAP-MRF](http://en.wikipedia.org/wiki/Markov_random_field) problems are relatively common in the bioinformatics, computer vision, and NLP literature. These problems are often formulated as quadratic pseudoboolean optimization problems, and there is a growing trend in the literature to solve these problems via dual decomposition. I have also seen authors pointing out a connection between [belief propagation](http://en.wikipedia.org/wiki/Belief_propagation) and other general [message passing methods](http://en.wikipedia.org/wiki/Variational_message_passing) to dual decomposition.  
> **username_1**: I have tried answering your question by taking into account the examples you have cited.  
> **username_0**: Thanks @Ankur, this is very helpful. In my update of the original post I expanded more on what type of problems I am referring to: MAP-MRF with **binary and quadratic cost functions**. In other words the cost function is more of the form  argmin$[\sum a_{ij} x_i x_j  + \sum b_k x_k]$ (as opposed to just argmin$[\sum f(x_k)]$ as you wrote). However you are right about the structure of the constraints $K$, they are almost always of the form $\sum_i f(x_i,x_j) = g(x_j)$ (i.e. defining what's known as the **marginal polytope**)  
> **username_1**: Has the problem you have written been solved by dual decomposition? Some clever reformulation may be required to put it in a form that is more suitable for dual decomposition.  
> **username_0**: The first two references I provided in my original post solve quadratic problems of the form I just wrote. For example, in the second reference I had in the original post, i.e. [(2)](http://people.csail.mit.edu/tommi/papers/SonGloJaa_optbook.pdf), the first equation in the article is quadratic; notice that the second term in the sum is defined on terms of the form $\theta_f (\mathbf{x}_f) = \theta_{ij}(xi, xj)$. These equations arise often in MRFs, where you have a cost function defined on the edges and nodes of a graph.  

---

