Q: Understanding of a pendulum with (perpendicular) moving masses I'm currently working on my thesis and am stuck on solving the last part of my work. The problematic system evolves around a rigid pendulum frame with two masses. This pendulum consists of a fixed pivot point O, a massless pendulum (black), two masses ($m_1$ and $m_2$) and two links perpendicular to the pendulum (red). Below a simple sketch of the system is given:

The system works as follows. The pendulum will start swinging due to its initial angle. During this swinging the masses will be moved on their respective links ($m_2$ on AB and $m_1$ on CD). The goal is to keep $m_1$ on the vertical axis as the pendulum is swinging. The mass $m_2$ is moved in the opposite direction to keep the center of mass in the extension of the pendulum (black link).
So what is it that I don't understand? Well to move the masses relative to the pendulum frame a force has to be exerted on the masses. This force will work on the frame in the opposite direction, but since the frame of the pendulum is massless it confuses me. The forces will also cause a moment working around the center of mass?
My current approach this system is to add the two masses together to get a pendulum with a single mass that is a distance lc away from the pivot point. This distance can be calculated by the following equation:
$ lc = l_1 + l_2 \frac{m_2}{m_1+m_2} $.
If one mass is accelerated it will push against the other mass leading to an additional torque component in the simple pendulum equation. Leading to the following equation: $τ= l_cmgsin(θ)+l_1 F_{m2}+(l_1+l_2)F_{m1}$. If this is then divided by $(m_1+m_2)l_c^2$ you will get the equation for angular acceleration.
I hope my explanation was clear and would be very grateful if someone is able to help me understand the underlying physics and if my current approach is the correct way. Thank you in advance.

A: 
This is not the answer of your quation ,but the solution of the your problem.
to solve the problem the position of $~m_2~$ must be constant
starting  with the components of the  position vectors given in $~x~,~y~,z~$ system
$$\vec R_1= \left[ \begin {array}{ccc} \cos \left( \vartheta  \right) &0&\sin
 \left( \vartheta  \right) \\ 0&1&0
\\ -\sin \left( \vartheta  \right) &0&\cos \left( 
\vartheta  \right) \end {array} \right]\,\left[ \begin {array}{c} -x_{{1}}\\ 0
\\  -l_{{1}}-l_{{2}}\end {array} \right] 
$$
and
$$\vec R_2=\left[ \begin {array}{ccc} \cos \left( \vartheta  \right) &0&\sin
 \left( \vartheta  \right) \\ 0&1&0
\\ -\sin \left( \vartheta  \right) &0&\cos \left( 
\vartheta  \right) \end {array} \right]\,\left[ \begin {array}{c} x_{{2}}\\ 0
\\ -l_{{1}}\end {array} \right]
$$
you have 2 degrees of freedom, coordinates $~x_1,\vartheta$
from the requirement that the position of  $~m_1 ~$ remains on the vertical you obtain that
$$R_{2z}=-(l_1+l_2)\quad \Rightarrow\\
\left( l_{{1}}+l_{{2}} \right)  \left( 1-\cos \left( \vartheta 
 \right)  \right) -\sin \left( \vartheta  \right) x_{{1}}
=0\tag 1$$
this is holonomic constraint equation .
from here with EL and Lagrange multiplicator  you can generate the equations for $~\ddot\vartheta~,\ddot x_1~,\lambda$
steady state solution:
steady state  means that all derivatives are zero thus you obtain two equations for the unknows $~\vartheta_s ~$ and $~x_{1s}~$
$$\tan(\vartheta_s)=\frac{x_2}{l_1}\\
x_{1s}\quad \text{arbitrary }$$
the steady state solution for $~x_{1s}~$  you can obtain  with equation (1)
$$x_{1s}=-{\frac { \left( l_{{1}}+l_{{2}} \right) \cos \left( \vartheta _{{s}
} \right) }{\sin \left( \vartheta _{{s}} \right) }}-{\frac {-l_{{1}}-
l_{{2}}}{\sin \left( \vartheta _{{s}} \right) }}
$$
with the steady state solutions you can obtain the torque on the frame $~\tau_F~$ and the force on mass $~m_1~,F_m$
\begin{align*}
&\begin{bmatrix}
  \tau_F \\
  F_{m} \\
\end{bmatrix}=\left[\frac{\partial F_c}{\partial \mathbf{q}}\right]^T\lambda_s= \left[ \begin {array}{c} -\cos \left( \vartheta  \right) x_{{1}}+
\sin \left( \vartheta_s  \right) l_{{1}}+\sin \left( \vartheta_s
 \right) l_{{2}}\\  -\sin \left( \vartheta_s  \right)
\end {array} \right]
\lambda_s\\\\
&\tau_F=  {\frac {m_{{2}}g \left( -{l_{{1}}}^{3}-{l_
{{1}}}^{2}l_{{2}}+{l_{{1}}}^{2}\sqrt {{x_{{2}}}^{2}+{l_{{1}}}^{2}}
+l_{{1}}l_{{2}}\sqrt {{x_{{2}}}^{2}+{l_{{1}}}^{2}}-{x_{{2}}}^{2}
l_{{1}}-{x_{{2}}}^{2}l_{{2}} \right) }{\sqrt {{x_{{2}}}^{2}+{l_{{
1}}}^{2}}x_{{2}}}}
\\
&F_{m}=  {\frac {m_{{2}}gx_{{2}}}{\sqrt {{x_{{2}}}^
{2}+{l_{{1}}}^{2}}}}
\end{align*}
