<p>(This perhaps should be considered a comment.)</p>

<p>If you have solved the HJB equation, it is sufficient to get the optimal solution. So you  do not "have to be concerned with any other optimality conditions," which I believe appears to answer your question.</p>

<p>It appears that you are concerned about the "necessary" component of the theorem. The necessity side of the statement is as follows: if there is an optimal solution, there must exist a solution to the HJB equation.</p>

<p>I have not worked with this particular problem, but the answer in general is that we do not expect to have a differentiable function V. Therefore we do not have a solution to the equation as it is stated. Instead, we need to look at generalised derivatives, and convert the HJB equation into an inequality. In which case, you may get a "viscosity solution." If we extend to use generalised derivatives, it may be possible to prove that a such a solution always exists. Glancing at your proofs, they will not help on the necessity conditions, as you are assuming differentiability.</p>

CC BY-SA 3.0
2017-04-16T00:04:16.940Z
2017-04-16T00:04:16.940Z
Brian Romanchuk
<p>Consider the following differential equation
\begin{align}
\dot x(t)=f(x(t),u(t))
\end{align}
where  $x$ is the state and $u$ the control variable. The solution is given by
\begin{align}
x(t)=x_0 + \int^t_0f(x(s),u(s))ds.
\end{align}
where $x_0:=x(0)$ is the given inital state.</p>

<p>Now consider the following programm
\begin{align}
&amp;V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\
s.t.~&amp;\dot x(t)=f(x(t),u(t))\\
&amp;x(0) = x_0
\end{align}
where $\rho &gt; 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by
\begin{align}
\rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty).
\end{align}</p>

<p>Say I've solved the HJB for $V$. The optimal control is then given by
\begin{align}
u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)].
\end{align}
I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.</p>

<p>The <a href="https://en.wikipedia.org/wiki/Hamilton%E2%80%93Jacobi%E2%80%93Bellman_equation" rel="noreferrer">wiki</a> article says </p>

<blockquote>
  <p>...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum. </p>
</blockquote>

<p>In Bertsekas (2005) <em>Dynamic Programming and Optimal Control</em>, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem. </p>

<p>Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.</p>

<h2>Solution</h2>

<p><strong>I Attempt</strong></p>

<p>I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.</p>

<p>Define the hamiltonian
\begin{align}
H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u)
\end{align}</p>

<p>then we have
\begin{align}
\rho V(x)=\max_u H(x,u,V'(x))
\end{align}</p>

<p>which is
\begin{align}
\rho V(x)= H(x,u^*,V'(x)).
\end{align}</p>

<p>Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix
\begin{align}
x = x^*+\varepsilon q
\end{align}</p>

<p>where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives
\begin{align}
\rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)).
\end{align}</p>

<p>At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition
\begin{align}
\rho V'q = H_x q + H_{V'}V''q.
\end{align}</p>

<p>Now define the adjoint variable with
\begin{align}
\lambda = V'(x).
\end{align}</p>

<p>Differentiate over time
\begin{align}
\dot \lambda = V''\dot x.
\end{align}</p>

<p>and note that
\begin{align}
H_{V'} = f(x,u) = \dot x.
\end{align}</p>

<p>Plug everthing into the foc wich gives
\begin{align}
\rho \lambda = H_x  + \dot \lambda.
\end{align}</p>

<p>That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).</p>

<p>However the transversality condition
\begin{align}
\lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0
\end{align}
is missing.</p>

<p><strong>II Attempt</strong></p>

<p>Define the payoff functional
\begin{align}
J(u):=\int^\infty_0 e^{-\rho t}F(x,u)dt
\end{align}</p>

<p>Note that 
\begin{align}
\int^\infty_0{e^{-\rho t}\lambda[f(x,u) - \dot x]dt} = 0
\end{align}
by definition of $\dot x = f(x,u)$. Add the neutral Term to the payoff funtional
\begin{align}
J(u)&amp;=\int^\infty_0 e^{-\rho t}[F(x,u)+\lambda f(x,u)]dt -  \int^\infty_0{e^{-\rho t}\lambda\dot xdt}\\
&amp;=\int^\infty_0 e^{-\rho t}H(x,u,\lambda) -  \int^\infty_0{e^{-\rho t}\lambda\dot xdt}
\end{align}</p>

<p>Integration by parts of the right term ond the rhs yields
\begin{align}
\int^\infty_0{e^{-\rho t}\lambda\dot xdt} = [e^{-\rho t}\lambda(t)x(t)]^\infty_0 - \int^\infty_0{e^{-\rho t}x(\dot \lambda-\rho\lambda)dt}
\end{align}</p>

<p>Resubstitute that term
\begin{align}
J(u)=\int^\infty_0 e^{-\rho t}[H(x,u,\lambda) + x(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)x(t) +  \lambda(0)x(0)
\end{align}</p>

<p>Define 
\begin{align}
x &amp;= x^*+\varepsilon q\\
u &amp;= u^*+\varepsilon p
\end{align}</p>

<p>which gives 
\begin{align}
J(\varepsilon)=\int^\infty_0 e^{-\rho t}[H(x^*+\varepsilon q,u^*+\varepsilon p,\lambda) + (x^*+\varepsilon q)(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)[x^*(t)+\varepsilon q(t)] +  \lambda(0)x(0)
\end{align}</p>

<p>FOC for maximum $J_\varepsilon = 0$
\begin{align}
J_\varepsilon=\int^\infty_0 e^{-\rho t}[H_x q + H_u p + q(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)q(t) = 0
\end{align}</p>

<p>Since $q$ and $p$ are unconstrained we must have
\begin{align}
H_u &amp;= 0\\
H_x &amp;= \rho\lambda - \dot \lambda\\
\lim_{t\to\infty}e^{-\rho t}\lambda(t) &amp;= 0
\end{align}</p>

CC BY-SA 3.0
2015-07-29T10:00:52.180Z
I am not sure what is asked here: if per Bertsekas solving HJB is _sufficient_, then you do not have to "worry about additional optimality conditions". The "sufficient only" against "necessary and sufficient" would arise in case HJB was not solved -in which case one would say "this does not mean that there is no solution". By the way, your Attempts I and II are valuable content here - the first showing a link bewteen HJB and Optimal Control, the second showing how the Optimal Control FOCs can be derived.
Alecos Papadopoulos
CC BY-SA 3.0
2015-07-29T10:40:11.803Z
Well, I stumbled over "sufficient only" against "necessary and sufficient". If one would have asked me "you solved the HJB, but how do I know that the necessary conditions are met?" I would have had to shrug and reply "don't know". Can we conjecture that solving the HJB implies fulfillment of necessary optimality conditions?
clueless
CC BY-SA 3.0
2015-07-08T22:47:36.043Z
have you identified the necessary and sufficient conditions yet?
Jamzy
CC BY-SA 3.0
2015-07-11T05:04:35.140Z
In what economic context does this come up?
Stan Shunpike
CC BY-SA 3.0
2015-07-11T08:12:34.990Z
Ramsey model for instance http://www.cer.ethz.ch/resec/people/tsteger/Ramsey_Model.pdf
clueless
CC BY-SA 3.0
2015-07-14T13:22:32.647Z
I think that this thread is better suited for http://math.stackexchange.com/ since it is not really linked to econ. A mod may transfer it.
clueless
<mathematical-economics><reference-request><dynamic-programming>
Solving the Hamilton-Jacobi-Bellman equation; necessary and sufficient for optimality?
clueless
# Solving the Hamilton-Jacobi-Bellman equation; necessary and sufficient for optimality?

(Asked by: username_0 on 2015-07-08)

Consider the following differential equation
\begin{align}
\dot x(t)=f(x(t),u(t))
\end{align}
where $x$ is the state and $u$ the control variable. The solution is given by
\begin{align}
x(t)=x\_0 + \int^t\_0f(x(s),u(s))ds.
\end{align}
where $x\_0:=x(0)$ is the given inital state.

Now consider the following programm
\begin{align}
&V(x\_0) := \max\_u \int^\infty\_0 e^{-\rho t}F(x(t),u(t))dt\\
s.t.~&\dot x(t)=f(x(t),u(t))\\
&x(0) = x\_0
\end{align}
where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by
\begin{align}
\rho V(x)=\max\_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty).
\end{align}

Say I've solved the HJB for $V$. The optimal control is then given by
\begin{align}
u^\*=\arg\max\_u [F(x,u) + V'(x)f(x,u)].
\end{align}
I'll get optimal trajectories for the state and control $\{(x^\*(t),u^\*(t)):t\in[0,\infty)\}$.

The [wiki](https://en.wikipedia.org/wiki/Hamilton%E2%80%93Jacobi%E2%80%93Bellman_equation) article says

> ...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) *Dynamic Programming and Optimal Control*, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^\*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

## Solution

**I Attempt**

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian
\begin{align}
H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u)
\end{align}

then we have
\begin{align}
\rho V(x)=\max\_u H(x,u,V'(x))
\end{align}

which is
\begin{align}
\rho V(x)= H(x,u^\*,V'(x)).
\end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim\_{t\to\infty} q(t)=0$. Now fix
\begin{align}
x = x^\*+\varepsilon q
\end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives
\begin{align}
\rho V(x^\*+\varepsilon q)= H(x^\*+\varepsilon q,u^\*,V'(x^\*+\varepsilon q)).
\end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition
\begin{align}
\rho V'q = H\_x q + H\_{V'}V''q.
\end{align}

Now define the adjoint variable with
\begin{align}
\lambda = V'(x).
\end{align}

Differentiate over time
\begin{align}
\dot \lambda = V''\dot x.
\end{align}

and note that
\begin{align}
H\_{V'} = f(x,u) = \dot x.
\end{align}

Plug everthing into the foc wich gives
\begin{align}
\rho \lambda = H\_x + \dot \lambda.
\end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).

However the transversality condition
\begin{align}
\lim\_{t\to\infty} e^{-\rho t}\lambda(t) = 0
\end{align}
is missing.

**II Attempt**

Define the payoff functional
\begin{align}
J(u):=\int^\infty\_0 e^{-\rho t}F(x,u)dt
\end{align}

Note that
\begin{align}
\int^\infty\_0{e^{-\rho t}\lambda[f(x,u) - \dot x]dt} = 0
\end{align}
by definition of $\dot x = f(x,u)$. Add the neutral Term to the payoff funtional
\begin{align}
J(u)&=\int^\infty\_0 e^{-\rho t}[F(x,u)+\lambda f(x,u)]dt - \int^\infty\_0{e^{-\rho t}\lambda\dot xdt}\\
&=\int^\infty\_0 e^{-\rho t}H(x,u,\lambda) - \int^\infty\_0{e^{-\rho t}\lambda\dot xdt}
\end{align}

Integration by parts of the right term ond the rhs yields
\begin{align}
\int^\infty\_0{e^{-\rho t}\lambda\dot xdt} = [e^{-\rho t}\lambda(t)x(t)]^\infty\_0 - \int^\infty\_0{e^{-\rho t}x(\dot \lambda-\rho\lambda)dt}
\end{align}

Resubstitute that term
\begin{align}
J(u)=\int^\infty\_0 e^{-\rho t}[H(x,u,\lambda) + x(\dot \lambda-\rho\lambda)]dt - \lim\_{t\to\infty}e^{-\rho t}\lambda(t)x(t) + \lambda(0)x(0)
\end{align}

Define
\begin{align}
x &= x^\*+\varepsilon q\\
u &= u^\*+\varepsilon p
\end{align}

which gives
\begin{align}
J(\varepsilon)=\int^\infty\_0 e^{-\rho t}[H(x^\*+\varepsilon q,u^\*+\varepsilon p,\lambda) + (x^\*+\varepsilon q)(\dot \lambda-\rho\lambda)]dt - \lim\_{t\to\infty}e^{-\rho t}\lambda(t)[x^\*(t)+\varepsilon q(t)] + \lambda(0)x(0)
\end{align}

FOC for maximum $J\_\varepsilon = 0$
\begin{align}
J\_\varepsilon=\int^\infty\_0 e^{-\rho t}[H\_x q + H\_u p + q(\dot \lambda-\rho\lambda)]dt - \lim\_{t\to\infty}e^{-\rho t}\lambda(t)q(t) = 0
\end{align}

Since $q$ and $p$ are unconstrained we must have
\begin{align}
H\_u &= 0\\
H\_x &= \rho\lambda - \dot \lambda\\
\lim\_{t\to\infty}e^{-\rho t}\lambda(t) &= 0
\end{align}

**Tags:** mathematical-economics, reference-request, dynamic-programming

> **Comments:**
> **username_2**: have you identified the necessary and sufficient conditions yet?  
> **username_3**: In what economic context does this come up?  
> **username_0**: Ramsey model for instance http://www.cer.ethz.ch/resec/people/tsteger/Ramsey_Model.pdf  
> **username_0**: I think that this thread is better suited for http://math.stackexchange.com/ since it is not really linked to econ. A mod may transfer it.  
> **username_1**: I am not sure what is asked here: if per Bertsekas solving HJB is _sufficient_, then you do not have to "worry about additional optimality conditions". The "sufficient only" against "necessary and sufficient" would arise in case HJB was not solved -in which case one would say "this does not mean that there is no solution". By the way, your Attempts I and II are valuable content here - the first showing a link bewteen HJB and Optimal Control, the second showing how the Optimal Control FOCs can be derived.  
> **username_0**: Well, I stumbled over "sufficient only" against "necessary and sufficient". If one would have asked me "you solved the HJB, but how do I know that the necessary conditions are met?" I would have had to shrug and reply "don't know". Can we conjecture that solving the HJB implies fulfillment of necessary optimality conditions?  

---

## 1 answers

### Answer by username_4 on 2017-04-16. Score: 2

(This perhaps should be considered a comment.)

If you have solved the HJB equation, it is sufficient to get the optimal solution. So you do not "have to be concerned with any other optimality conditions," which I believe appears to answer your question.

It appears that you are concerned about the "necessary" component of the theorem. The necessity side of the statement is as follows: if there is an optimal solution, there must exist a solution to the HJB equation.

I have not worked with this particular problem, but the answer in general is that we do not expect to have a differentiable function V. Therefore we do not have a solution to the equation as it is stated. Instead, we need to look at generalised derivatives, and convert the HJB equation into an inequality. In which case, you may get a "viscosity solution." If we extend to use generalised derivatives, it may be possible to prove that a such a solution always exists. Glancing at your proofs, they will not help on the necessity conditions, as you are assuming differentiability.


---

