<p>Here is a more detailed description of what I wrote in the comments. Ideally such a description should come with a ton of graphics illustrating all of the discussed ideas. I did not have the time to produce said images, so my answer is lacking in that respect.</p>
<p><strong>Summary For Implementation Purposes</strong></p>
<p>Let the vertices of the first triangle be <span class="math-container">$v_0, v_1, v_2\in\mathbb{R}^3$</span> and the vertices of the second triangle be <span class="math-container">$w_0, w_1, w_2\in\mathbb{R}^3$</span>. For a unique solution we require that the first triangle is non-degenerate: it must have non-zero area (we cannot have the 3 vertices lying on a single line or in a single point).</p>
<p>Compute <span class="math-container">$e_1 = v_1-v_0, \, e_2 = v_2-v_0$</span> and <span class="math-container">$f_1 = w_1-w_0, \, f_2 = w_2 - w_0$</span>.</p>
<p><em>Option 1</em></p>
<p>Compute <span class="math-container">$e_3 = e_1 \times e_2$</span> and <span class="math-container">$f_3 = f_1 \times f_2$</span> where <span class="math-container">$\times$</span> is the <a href="https://en.wikipedia.org/wiki/Cross_product#Coordinate_notation" rel="nofollow noreferrer">cross product</a>. Then form the <span class="math-container">$3\times 3$</span> matrices <span class="math-container">$E$</span> and <span class="math-container">$F$</span> using the <span class="math-container">$e_i$</span> and <span class="math-container">$f_i$</span> as column vectors:</p>
<p><span class="math-container">$$
\begin{gather}
E = \begin{bmatrix} e_1 &amp; e_2 &amp; e_3 \end{bmatrix}, \quad F = \begin{bmatrix} f_1 &amp; f_2 &amp; f_3\end{bmatrix}.
\end{gather}
$$</span></p>
<p>Finally, compute <span class="math-container">$M = FE^{-1}$</span> where <span class="math-container">$E^{-1}$</span> is the inverse of <span class="math-container">$E$</span>, and compute <span class="math-container">$m_3 = w_0 - Mv_0$</span>. Set <span class="math-container">$M$</span> as the <span class="math-container">$3\times 3$</span> part of your <span class="math-container">$4\times 4$</span> matrix <span class="math-container">$M'$</span> and <span class="math-container">$m_3$</span> as its last column:</p>
<p><span class="math-container">$$
\begin{equation}
M' = \begin{bmatrix} m_0 &amp; m_1 &amp; m_2 &amp; m_3 \\ 0 &amp; 0 &amp; 0 &amp; 1 \end{bmatrix}.
\end{equation}
$$</span></p>
<p><em>Option 2</em></p>
<p>Form the <span class="math-container">$3\times 2$</span> matrices <span class="math-container">$E$</span> and <span class="math-container">$F$</span>:
<span class="math-container">$$
\begin{equation}
E = \begin{bmatrix} e_1 &amp; e_2 \end{bmatrix}, \quad F = \begin{bmatrix} f_1 &amp; f_2 \end{bmatrix}.
\end{equation}
$$</span>
The form of the solution is similar to the one from <em>Option 1</em>:
<span class="math-container">$$
\begin{gather}
M = FE^{+}, \quad m_3 = w_0 - FE^{+}v_0,\quad E^{+} = (E^TE)^{-1}E^T \\
M' = \begin{bmatrix} m_0 &amp; m_1 &amp; m_2 &amp; m_3 \\ 0 &amp; 0 &amp; 0 &amp; 1 \end{bmatrix}.
\end{gather}
$$</span></p>
<p><em>Differences Between Option 1 and Option 2</em></p>
<p>The two approaches (<em>Option 1</em> and <em>Option 2</em>) agree when applied to points <span class="math-container">$p$</span> that lie in the plane of the triangle <span class="math-container">$v_0, v_1, v_2$</span>, however they differ if <span class="math-container">$p$</span> is not in this plane.</p>
<p><em>Option 1</em> rescales space perpendicular to the triangles <span class="math-container">$v_0, v_1, v_2$</span> and <span class="math-container">$w_0, w_1, w_2$</span> according to the ratio <span class="math-container">$r = \frac{area(w_0, w_1, w_2)}{area(v_0,v_1,v_2)}$</span>. For example, if <span class="math-container">$r = 2$</span> (the second triangle has twice larger area than the first) and <span class="math-container">$p$</span> was <span class="math-container">$3$</span> units perpendicularly away from the first triangle's plane, then its transformed version would be <span class="math-container">$3\times r = 3\times 2 = 6$</span> units perpendicularly away from the second triangle.</p>
<p>If <span class="math-container">$p$</span> is not inside the plane in which <span class="math-container">$v_0, v_1, v_2$</span> lies, then <em>Option 2</em> projects <span class="math-container">$p$</span> on this plane, and only then does it transform it. That is, you would always get a transformed point in the plane of the second triangle <span class="math-container">$w_0,w_1,w_2$</span> regardless of what point you feed in, since it would first get projected on the plane of the first triangle. For example if you had a point <span class="math-container">$p$</span> that is <span class="math-container">$3$</span> units perpendicularly away from the plane of the first triangle, then it will end up <span class="math-container">$0$</span> units perpendicularly away from the plane of the second triangle (i.e. it will be in it).</p>
<p><strong>Introduction</strong></p>
<p>The following sections contain a more detailed treatment of the problem along with guidelines on ideas that may be used to derive the final solution.</p>
<p>I'll start with a simplified case for clarity. It turns out that studying <a href="https://en.wikipedia.org/wiki/Affine_transformation#Representation" rel="nofollow noreferrer">affine transformations</a> between <a href="https://en.wikipedia.org/wiki/Tetrahedron" rel="nofollow noreferrer">tetrahedra</a> in 3-D is simpler than studying affine transformations between triangles in 3-D. Studying transformations between triangles is simplest in 2-D for instance, and for <span class="math-container">$n$</span>-D it is simplest to study transformations between <a href="https://en.wikipedia.org/wiki/Simplex" rel="nofollow noreferrer"><span class="math-container">$n$</span>-simplices</a> (a <span class="math-container">$2$</span>-simplex is a triangle, and a <span class="math-container">$3$</span>-simplex is a tetrahedron). This is due to the fact that <a href="https://en.wikipedia.org/wiki/Dimension_(vector_space)" rel="nofollow noreferrer">an <span class="math-container">$n$</span>-dimensional vector space is uniquely represented through exactly <span class="math-container">$n$</span> basis vectors</a>. Consequently, studying transformations between triangles (a 2-D object) in 3-D results in some difficulties compared to studying tetrahedra, since in 3-D the triangle case leaves out one of the basis vectors.</p>
<p><strong>Tetrahedron Case: Formulation</strong></p>
<p>Let <span class="math-container">$v_0,v_1,v_2, v_3 \in\mathbb{R}^3$</span> be the vertices of a non-degenerate tetrahedron <span class="math-container">$\mathcal{T}_{\mathcal{V}}$</span> in 3-D, and let <span class="math-container">$w_0, w_1, w_2, w_3\in\mathbb{R}^3$</span> be the vertices of another (not necessarily non-degenerate) tetrahedron <span class="math-container">$\mathcal{T}_{\mathcal{W}}$</span>. We want an <a href="https://en.wikipedia.org/wiki/Affine_transformation#Representation" rel="nofollow noreferrer">affine</a> (linear + offset) function <span class="math-container">$g:\mathcal{T}_{\mathcal{V}}\to\mathcal{T}_{\mathcal{W}}$</span> that maps the first tetrahedron to the second one. As a special case the corresponding vertices must map to each other:</p>
<p><span class="math-container">$$
\begin{gather}
g(v_0) = w_0, \quad
g(v_1) = w_1, \quad
g(v_2) = w_2, \quad
g(v_3) = w_3.
\end{gather}
$$</span></p>
<p>Since <span class="math-container">$g(p)$</span> is affine and acts on 3-D points, then it follows from the definition that <a href="https://en.wikipedia.org/wiki/Affine_transformation#Representation" rel="nofollow noreferrer">it can be represented by a multiplication with a 3x3 matrix and a translation</a>:</p>
<p><span class="math-container">$$
\begin{equation}
g(p) = Mp + m_3, \quad M = \begin{bmatrix} m_0 &amp; m_1 &amp; m_2 \end{bmatrix}, \quad  m_0, m_1, m_2, m_3 \in\mathbb{R}^3.
\end{equation}
$$</span></p>
<p>That is, we have a 3x3 matrix <span class="math-container">$M$</span> with 3-D column vectors <span class="math-container">$m_0, m_1, m_2$</span> and one translation/offset 3-D vector <span class="math-container">$m_3$</span>. Rewriting the constraints using the above form of <span class="math-container">$g$</span> yields:</p>
<p><span class="math-container">$$
\begin{align}
g(v_0) = w_0 &amp;\iff m_0 v_{0,x} + m_1 v_{0,y} + m_2 v_{0,z} + m_3 = w_0 \\
g(v_1) = w_1 &amp;\iff m_0 v_{1,x} + m_1 v_{1,y} + m_2 v_{1,z} + m_3 = w_1 \\
g(v_2) = w_2 &amp;\iff m_0 v_{2,x} + m_1 v_{2,y} + m_2 v_{2,z} + m_3 = w_2 \\
g(v_3) = w_3 &amp;\iff m_0 v_{3,x} + m_1 v_{3,y} + m_2 v_{3,z} + m_3 = w_3.
\end{align}
$$</span></p>
<p><strong>Tetrahedron Case: Solution</strong></p>
<p>The above is a system with <span class="math-container">$4\times 3=12$</span> unknowns (4 <span class="math-container">$m_i$</span> vectors each with 3 components) and <span class="math-container">$4\times 3=12$</span> equations. From linear algebra we know that <a href="https://en.wikipedia.org/wiki/Rank%E2%80%93nullity_theorem" rel="nofollow noreferrer">as long as the equations are not linearly dependent this system has a unique solution</a>. We can express <span class="math-container">$m_0$</span> from the first equation:</p>
<p><span class="math-container">$$
\begin{equation}
m_3 = w_0 - (m_0 v_{0,x} + m_1 v_{0,y} + m_2 v_{0,z}).
\end{equation}
$$</span></p>
<p>Plugging this expression in the remaining equations reduces the system to 9 unknowns:
<span class="math-container">$$
\begin{align}
m_0 e_{1,x} + m_1 e_{1,y} + m_2 e_{1,z} &amp;= f_1 \\
m_0 e_{2,x} + m_1 e_{2,y} + m_2 e_{2,z} &amp;= f_2 \\
m_0 e_{3,x} + m_1 e_{3,y} + m_2 e_{3,z} &amp;= f_3,
\end{align}
$$</span></p>
<p>where <span class="math-container">$e_1 = v_1-v_0, \, e_2 = v_2-v_0, \, e_3 = v_3-v_0$</span> and <span class="math-container">$f_1 = w_1-w_0, \, f_2 = w_2-w_0, \, f_3 = w_3-w_0$</span>. This can be written compactly in matrix form:</p>
<p><span class="math-container">$$
\begin{gather}
M E = F, \, E = \begin{bmatrix} e_1 &amp; e_2 &amp; e_3 \end{bmatrix}, \, F =  \begin{bmatrix} f_1 &amp; f_2 &amp; f_3 \end{bmatrix}\\ 
\implies \\
M = E^{-1}F, \, m_3 = w_0 - Mv_0.
\end{gather} 
$$</span>
The matrix <a href="https://en.wikipedia.org/wiki/Invertible_matrix#The_invertible_matrix_theorem" rel="nofollow noreferrer"><span class="math-container">$E^{-1}$</span> exists provided that its columns are linearly independent</a>. This is the case when the tetrahedron with vertices <span class="math-container">$v_0, v_1, v_2, v_3$</span> is non-degenerate: the points do not coincide, they do not lie on a single line, they do not lie in a single plane, i.e. <a href="https://en.wikipedia.org/wiki/Determinant#Volume_and_Jacobian_determinant" rel="nofollow noreferrer">it's a proper pyramid with non-zero volume</a> (not a squished one).</p>
<p><a href="https://en.wikipedia.org/wiki/Affine_transformation#Augmented_matrix" rel="nofollow noreferrer">We can form a <span class="math-container">$4\times 4$</span> matrix</a> by using <span class="math-container">$M$</span> as the <span class="math-container">$3\times 3$</span> part, <span class="math-container">$m_3$</span> as the last column, and setting <span class="math-container">$\begin{bmatrix} 0 &amp; 0 &amp; 0 &amp; 1 \end{bmatrix}$</span> in the last row. Then the transformation takes the form:</p>
<p><span class="math-container">$$
\begin{equation}
f(p) = q \iff \begin{bmatrix} m_0 &amp; m_1 &amp; m_2 &amp; m_3 \\ 0 &amp; 0 &amp; 0 &amp; 1 \end{bmatrix}\begin{bmatrix} p \\ 1 \end{bmatrix} = \begin{bmatrix} q \\ 1 \end{bmatrix}
\end{equation}
$$</span></p>
<p>Note also that we could have taken any of the vertices as origin instead of <span class="math-container">$v_0, w_0$</span>:
<span class="math-container">$$
\begin{gather}
m_3 = w_i - (m_0 v_{i,x} + m_1 v_{i,y} + m_2 v_{i,z}) \\
e_j = v_j - v_i, \, f_j = w_j - w_i, \, j\ne i,
\end{gather}
$$</span>
so the choice of <span class="math-container">$v_0,w_0$</span> as origins in the two coordinate systems was arbitrary.</p>
<p><strong>Triangle Case: Formulation</strong></p>
<p>The setting of transforming between two triangles in 3-D is similar to the tetrahedra case, however <span class="math-container">$v_3$</span> and <span class="math-container">$w_3$</span> are not provided. We have:
<span class="math-container">$$
\begin{align}
g(v_0) = w_0 &amp;\iff m_0 v_{0,x} + m_1 v_{0,y} + m_2 v_{0,z} + m_3 = w_0 \\
g(v_1) = w_1 &amp;\iff m_0 v_{1,x} + m_1 v_{1,y} + m_2 v_{1,z} + m_3 = w_1 \\
g(v_2) = w_2 &amp;\iff m_0 v_{2,x} + m_1 v_{2,y} + m_2 v_{2,z} + m_3 = w_2,
\end{align}
$$</span>
which is a system of <span class="math-container">$3\times 3=9$</span> equations, but there are <span class="math-container">$3\times 4 = 12$</span> unknowns. This means that the system is underdetermined <span class="math-container">$\implies$</span> it has infinitely many solutions even if <span class="math-container">$v_0, v_1, v_2$</span> is a non-degenerate triangle (has non-zero area). There are various approaches to reduce the set of solutions to a single one.</p>
<p><strong>Triangle Case: Solution</strong></p>
<p>The first option is to supply a 3rd vector <span class="math-container">$e_3$</span> that is <a href="https://en.wikipedia.org/wiki/Linear_independence" rel="nofollow noreferrer">linearly independent</a> of <span class="math-container">$e_1, e_2$</span> (doesn't lie in the same plane as <span class="math-container">$e_1$</span> and <span class="math-container">$e_2$</span>), and similarly <span class="math-container">$f_3$</span> that isn't necessarily linearly independent of <span class="math-container">$f_1, f_2$</span>. Then this becomes a special case of the tetrahedra transformations with <span class="math-container">$v_3 = v_0 + e_3$</span> and <span class="math-container">$w_3 = w_0 + f_3$</span>. Depending on the direction and length of <span class="math-container">$e_3$</span> and <span class="math-container">$f_3$</span> the space outside of the planes in which the triangles lie gets sheared and scaled differently.</p>
<p>It also becomes geometrically clear at this point why we have <span class="math-container">$12$</span> unknowns and <span class="math-container">$9$</span> equations: the remaining <span class="math-container">$12-9=3$</span> unknowns correspond to this choice of 3-dimensional vectors. One would of course note that we had to specify two vectors: <span class="math-container">$e_3$</span> and <span class="math-container">$f_3$</span>, which constitutes <span class="math-container">$6$</span> free variables instead of the <span class="math-container">$3$</span> pointed out above. The catch is that those <span class="math-container">$6$</span> variables get mixed together through <span class="math-container">$FE^{-1}$</span> in such a way that they could be represented through just <span class="math-container">$3$</span> free variables.</p>
<p>One simple option that guarantees linear independence is <span class="math-container">$e_3 = e_1 \times e_2$</span> and <span class="math-container">$f_3 = f_1 \times f_2$</span> where <span class="math-container">$\times$</span> is the <a href="https://en.wikipedia.org/wiki/Cross_product" rel="nofollow noreferrer">cross product</a>. The geometric implications of this choice are discussed in the <strong>TLDR:</strong> <em>Differences Between Option 1 and Option 2</em> subsection. Another option would be to normalize the vectors: <span class="math-container">$e_3 = \frac{e_1 \times e_2}{\|e_1\times e_2\|}$</span> and <span class="math-container">$f_3 = \frac{f_1 \times f_2}{\|f_1\times f_2\|}$</span>, where <span class="math-container">$\|\cdot\|$</span> is the <a href="https://en.wikipedia.org/wiki/Norm_(mathematics)#Euclidean_norm" rel="nofollow noreferrer">Euclidean norm/length</a> of the vectors. Finally, a choice of the form <span class="math-container">$e_3 = c (e_1\times e_2)$</span>, where <span class="math-container">$c\ne0$</span>, and <span class="math-container">$f_3=0$</span> results in a solution consistent with <em>Option 2</em>.</p>
<p>The difference to <em>Option 1</em> is that points that are perpendicularly <span class="math-container">$X$</span> units away from the plane of the first triangle remain <span class="math-container">$X$</span> units away from the plane of the second triangle after transformation. That is, there is no dependence on the change of triangle area. Thus the case without normalization (i.e. <em>Option 1</em>) assumes the transformation is uniform rescaling with ratio <span class="math-container">$r = \frac{area(w_0,w_1,w_2)}{area(v_0,v_1,v_2)}$</span> -&gt; rotation -&gt; translation, while the normalized case assumes that scale may be applied only perpendicular to the normal of the triangle, followed by a rotation and translation.</p>
<p><strong>Triangle Case: Pseudoinverse Solution</strong></p>
<p>We take a slightly different approach by starting directly from the underdetermined system:
<span class="math-container">$$
\begin{align}
g(v_0) = w_0 &amp;\iff m_0 v_{0,x} + m_1 v_{0,y} + m_2 v_{0,z} + m_3 = w_0 \\
g(v_1) = w_1 &amp;\iff m_0 v_{1,x} + m_1 v_{1,y} + m_2 v_{1,z} + m_3 = w_1 \\
g(v_2) = w_2 &amp;\iff m_0 v_{2,x} + m_1 v_{2,y} + m_2 v_{2,z} + m_3 = w_2.
\end{align}
$$</span>
Rewriting <span class="math-container">$m_3 = w_0 - (m_0 v_{0,x} + m_1 v_{0,y} + m_2 v_{0,z})$</span> and plugging in the remaining two equations yields:
<span class="math-container">$$
\begin{gather}
m_0 e_{1,x} + m_1 e_{1,y} + m_2 e_{1,z} = f_1 \\
m_0 e_{2,x} + m_1 e_{2,y} + m_2 e_{2,z} = f_2 \\
\end{gather}
$$</span>
We can write this more compactly in matrix form:
<span class="math-container">$$
\begin{gather}
M = \begin{bmatrix} m_0 &amp; m_1 &amp; m_2 \end{bmatrix}, \, E=\begin{bmatrix} e_1 &amp; e_2 \end{bmatrix}, \, F=\begin{bmatrix} f_1 &amp; f_2\end{bmatrix} \\
M E = F\\
ME = FI \\
ME = F(E^TE)^{-1}(E^TE) \\
ME = \left(F(E^TE)^{-1}E^T\right)E
\end{gather}
$$</span>
In the above <span class="math-container">$(E^TE)^{-1}$</span> exists provided <span class="math-container">$e_1, e_2$</span> are linearly independent (the triangle <span class="math-container">$v_0,v_1,v_2$</span> is non-degenerate). By identifying the terms on the left- and right-hand sides we come to the conclusion that one possible form of <span class="math-container">$M$</span> is <span class="math-container">$M = F(E^TE)^{-1}E^T$</span>.</p>
<p>The matrix <span class="math-container">$E^{+} = (E^TE)^{-1}E^T$</span> is in fact the <a href="https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse" rel="nofollow noreferrer">Moore-Penrose pseudoinverse</a>. It essentially projects points on the plane of the first triangle. For example, if we have a point <span class="math-container">$p$</span> then <a href="https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse#Projectors" rel="nofollow noreferrer">its projection</a> is <span class="math-container">$p' = EE^+p$</span>. Thus the final solution is:
<span class="math-container">$$
\begin{equation}
M = FE^{+}, \quad m_3 = w_0 - Mv_0.
\end{equation}
$$</span>
Note the similarity to the solution <span class="math-container">$M = FE^{-1}$</span> where <span class="math-container">$E = \begin{bmatrix} e_1 &amp; e_2 &amp; e_3\end{bmatrix}$</span> and <span class="math-container">$F = \begin{bmatrix} f_1 &amp; f_2 &amp; f_3 \end{bmatrix}$</span>. And actually the exact same solution for <span class="math-container">$M$</span> can be recovered by choosing <span class="math-container">$e_3 = c(e_1 \times e_2), c\ne 0$</span>, and <span class="math-container">$f_3 = 0$</span>.</p>
<p>For an efficient application of the Moore-Penrose pseudoinverse <span class="math-container">$E^{+}$</span>  to some vector <span class="math-container">$p$</span> the following can be used:
<span class="math-container">$$
\begin{align}
E^{+}p &amp;= (E^TE)^{-1}E^Tp \\
&amp;= \left(\begin{bmatrix} e_1\cdot e_1 &amp; e_1 \cdot e_2 \\ e_2 \cdot e_1 &amp; e_2 \cdot e_2 \end{bmatrix}\right)^{-1}\begin{bmatrix} e_1^T \\ e_2^T \end{bmatrix}p \\
&amp;= \frac{1}{(e_1\cdot e_1)(e_2 \cdot e_2) - (e_1\cdot e_2)^2}\begin{bmatrix} e_2\cdot e_2 &amp; -e_1\cdot e_2 \\ -e_2 \cdot e_1 &amp; e_1\cdot e_1 \end{bmatrix}\begin{bmatrix} e_1^T \\ e_2^T \end{bmatrix}p \\
&amp;= \frac{1}{(e_1\cdot e_1)(e_2 \cdot e_2) - (e_1\cdot e_2)^2}\begin{bmatrix}(e_2\cdot e_2)(e_1\cdot p) - (e_1\cdot e_2)(e_2\cdot p) \\ (e_1\cdot e_1)(e_2\cdot p) -(e_1\cdot e_2)(e_1\cdot p)\end{bmatrix}.
\end{align}
$$</span>
This results in 5 dot products, 6 scalar multiplications, and 3 subtractions for computing the application of <span class="math-container">$E^{+}$</span> to some vector <span class="math-container">$p$</span>.</p>

CC BY-SA 4.0
2022-08-25T20:38:36.923Z
That is a really complete answer, thanks for taking the time to explain in a concise manner. Great
Kroma
CC BY-SA 4.0
2022-09-17T12:16:39.637Z
Just an update: your solution works perfectly but the practical problem being that option 1 is sensitive to the order of input of the points. ABC != BCA (CCW points) in the final matrix -> and my uvs are wrong in some cases (Please look at my update; but the triangles position is always perfect..) What kind of algo enables to avoid this?
Kroma
CC BY-SA 4.0
2022-09-17T13:41:42.007Z
@Kroma A vertex $v_i$ maps to $w_i$, it is clear that this should depend on the order. As you noted, providing the wrong order has the effect of reordering the uv matching. You cannot avoid specifying which vertices should have which uvs, since this produces different results. You shouldn't expect that an uv ordering of ABC results in the same texturing as BCA, it is simply not true.
lightxbulb
CC BY-SA 4.0
2022-09-17T15:07:43.433Z
I get it, I will add some intelligence there. Thanks a lot :)
Kroma
CC BY-SA 4.0
2022-07-06T12:06:27.567Z
2022-07-06T12:06:27.567Z
lightxbulb
<p>The context openGL instances learning.</p>
<p>I want to transform a reference triangle to another one. So final goal is to have one triangle and the instance being a transformation of that initial triangle.</p>
<p>The idea is to apply a Matrix4x4 transformation on that reference triangle to inject into a 3d rendering pipeline.</p>
<p><strong>The library I use needs a reference triangle and a packed array of matrix 4x4 to tell it the transforms applied to the reference points.</strong></p>
<p>Z is UP (wrong, but it doesn't matter for the calculus logic now, i should still have the shapes right, even on the wrong axis).
Points are packed in a buffer.</p>
<p>So far I have this:</p>
<h3>Reference triangle, column major:</h3>
<pre><code>  array &lt;double, 16&gt; reference_triangle {0, 2, 1, 1,
                                         0, 0, 1, 1,
                                         0, 0, 0, 1,
                                         1, 1, 1, 1};
</code></pre>
<p>x: {0,0,0], y:{2,0,0} and z:{1,1,0}</p>
<h3>Second triangle given by:</h3>
<pre><code>  array &lt;double, 16&gt; to_transf_triangle {5 , 42, 31 ,1,
                                         10, 20, 11 ,1,
                                         0,   0,  0 ,1,
                                         1,   1,  1, 1};
</code></pre>
<p>x: {5,10,0], y:{42,20,0} and z:{31,11,0}</p>
<p><strong>Note:</strong>
Z axis is always 0 so my triangles could be considered 2D in that example (but won't always be the case, so please stick with 3D logic).</p>
<p>The other values are set to one for the transformation.</p>
<h3>Transformation pseudo code:</h3>
<pre><code>temp_transf = inverse(reference_triangle) * to_transf_triangle;
</code></pre>
<p>The final result in 3D is not what I expected: no translation (I expected it somehow because the resulting translation vector is null, I should add it from a point of the transform triangle) and triangles going visually awry.</p>
<p><strong>There must be a missing step.</strong></p>
<p>Now what is the way to transform one triangle given 3 points into another in a matrix 4x4 in a 3D opengl instance context?</p>
<p>I don't need code that much, just the logic.</p>
<p>Not even sure my method is a good starting point.</p>
<p>Thanks,</p>
<p>Update:</p>
<p>Wrong uvs depending on the point input order, but triangle shapes of that plane are perfect:
<a href="https://i.sstatic.net/Fqkk6.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Fqkk6.jpg" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2022-07-02T11:50:58.650Z
Let $v_0, v_1, v_2$ be the 3 original vertices. Compute: $e_1 = v_1-v_0, e_2=v_2-v_0, e_3= e_1 \times e_2$. Let $w_0, w_1, w_2$ be the second triangle's vertices, compute $f_1 = w_1-w_0, f_2 = w_2-w_0, f_3 = f_1\times f_2$. Form the 3x3 matrix $E=\begin{bmatrix} e_1 & e_2 & e_3 \end{bmatrix}$ with columns $e_1,e_2,e_3$, similarly $F=\begin{bmatrix} f_1 & f_2 & f_3 \end{bmatrix}$. Now use $q = w_0 + F E^{-1} (p-v_0)$ to transform from the space of the first to the space of the second triangle, e.g. if you set $p=v_1$ then you will get $q=w_1$.
lightxbulb
CC BY-SA 4.0
2022-07-02T11:55:48.693Z
Alternatively you can use $q=w_0 + F(E^TE)^{-1}E^T(p-v_0)$ where $E=\begin{bmatrix} e_1 & e_2 \end{bmatrix}$ and $F=\begin{bmatrix} f_1 & f_2 \end{bmatrix}$. The difference to the first approach is that if $p$ does not lie in the plane of the first triangle, then it will first be projected there and then be transformed.
lightxbulb
CC BY-SA 4.0
2022-07-02T15:33:41.197Z
The idea of computing a matrix transform that takes a given triangle and generates a specific output triangle is valid. But that computation will *not* look like "put the positions of the input triangle into the rows of a matrix" or whatever else you're doing here. It's not going to look anything like that. You are defining a change in coordinate systems, so you're going to have to compute the coordinate system change from the source to the destination.
Nicol Bolas
CC BY-SA 4.0
2022-07-02T17:47:19.843Z
@NicolBolas: yes you are right, I saw some random videos on Youtube that were telling to do the way I showed initially. I am currently implementing it into code.
Kroma
CC BY-SA 4.0
2022-07-02T17:48:12.290Z
@lightxbulb: if I understood, I then use the q vector to multiply each point vector? that will give me each component of my matrix3x3, that I will finally concatenate in my matrix 4x4, then add my translation component?
Kroma
CC BY-SA 4.0
2022-07-02T18:06:34.787Z
No. If you want a 4x4 matrix from the first formulation then the 3x3 part is $FE^{-1}$ and the translation (4th column) is $w_0-FE^{-1}v_0$. If you want to use the second formulation then the 3x3 part is $F(E^TE)^{-1}E^T$ and the translation is $w_0-F(E^TE)^{-1}E^Tv_0$. Note that for performance reasons you can analytically compute $E^TE=\begin{bmatrix} e_1 \cdot e_1 & e_1 \cdot e_2 \\ e_2 \cdot e_1 & e_2\cdot e_2\end{bmatrix}$  and its inverse (2x2 inverse).
lightxbulb
CC BY-SA 4.0
2022-07-02T20:14:40.313Z
@lightxbulb: it works  :). Thanks a lot. May I ask you what T means in E^T (I just implemented the first version)? Translation? And, if possible, how is this calculation called in the litterature, so that I get better in this field by reading? Again, thanks a lot.
Kroma
CC BY-SA 4.0
2022-07-02T21:36:11.600Z
$M^T$ means $M$ transposed. The $FE^{-1}$ part is called change of basis. Specifically you go from global coordinates in $E$ to local (barycentric) coordinates through $E^{-1}$ and then those get mapped to global coordinates wrt another basis  $F$. The $-v_0$ and $w_0$ makes it a change of coordinate system (basis + origin). The first approach has missing information regarding whether scale is applied along the axis perpendicular to the triangle, it makes the assumption that the change in scale along this axis is the same as the change in area since $|a\times b| = parallelogram\_area(a,b)$.
lightxbulb
CC BY-SA 4.0
2022-07-02T21:42:33.160Z
The second approach deals away with the arbitrary choice of scale along the axis perpendicular to the triangle, by requiring the point to be transformed $p$, to be in the plane defined by the triangle. If $p$ is not on the triangle, then $E^{+} = (E^TE)^{-1}E^T$ projects it on the triangle and then finds its barycentric coordinates. $E^{+}$ is known as the Moore-Penrose pseudoinverse. If $p$ is on the triangle then $E^{+}$ simply finds its barycentric coordinates. This means that the two approaches agree for points in the plane of the initial triangle, but differ otherwise.
lightxbulb
CC BY-SA 4.0
2022-07-03T05:21:34.750Z
@lightxbulb: thanks a lot, you gave me plenty of information and helped me learn something useful. +1
Kroma
<opengl><3d><transformations><matrix><performance>
Convert one triangle to another in matrix4x4?
Kroma
# Convert one triangle to another in matrix4x4?

(Asked by: username_0 on 2022-07-02)

The context openGL instances learning.

I want to transform a reference triangle to another one. So final goal is to have one triangle and the instance being a transformation of that initial triangle.

The idea is to apply a Matrix4x4 transformation on that reference triangle to inject into a 3d rendering pipeline.

**The library I use needs a reference triangle and a packed array of matrix 4x4 to tell it the transforms applied to the reference points.**

Z is UP (wrong, but it doesn't matter for the calculus logic now, i should still have the shapes right, even on the wrong axis).
Points are packed in a buffer.

So far I have this:

### Reference triangle, column major:

```
  array <double, 16> reference_triangle {0, 2, 1, 1,
                                         0, 0, 1, 1,
                                         0, 0, 0, 1,
                                         1, 1, 1, 1};

```

x: {0,0,0], y:{2,0,0} and z:{1,1,0}

### Second triangle given by:

```
  array <double, 16> to_transf_triangle {5 , 42, 31 ,1,
                                         10, 20, 11 ,1,
                                         0,   0,  0 ,1,
                                         1,   1,  1, 1};

```

x: {5,10,0], y:{42,20,0} and z:{31,11,0}

**Note:**
Z axis is always 0 so my triangles could be considered 2D in that example (but won't always be the case, so please stick with 3D logic).

The other values are set to one for the transformation.

### Transformation pseudo code:

```
temp_transf = inverse(reference_triangle) * to_transf_triangle;

```

The final result in 3D is not what I expected: no translation (I expected it somehow because the resulting translation vector is null, I should add it from a point of the transform triangle) and triangles going visually awry.

**There must be a missing step.**

Now what is the way to transform one triangle given 3 points into another in a matrix 4x4 in a 3D opengl instance context?

I don't need code that much, just the logic.

Not even sure my method is a good starting point.

Thanks,

Update:

Wrong uvs depending on the point input order, but triangle shapes of that plane are perfect:
[![enter image description here](https://i.sstatic.net/Fqkk6.jpg)](https://i.sstatic.net/Fqkk6.jpg)

**Tags:** opengl, 3d, transformations, matrix, performance

> **Comments:**
> **username_1**: Let $v_0, v_1, v_2$ be the 3 original vertices. Compute: $e_1 = v_1-v_0, e_2=v_2-v_0, e_3= e_1 \times e_2$. Let $w_0, w_1, w_2$ be the second triangle's vertices, compute $f_1 = w_1-w_0, f_2 = w_2-w_0, f_3 = f_1\times f_2$. Form the 3x3 matrix $E=\begin{bmatrix} e_1 & e_2 & e_3 \end{bmatrix}$ with columns $e_1,e_2,e_3$, similarly $F=\begin{bmatrix} f_1 & f_2 & f_3 \end{bmatrix}$. Now use $q = w_0 + F E^{-1} (p-v_0)$ to transform from the space of the first to the space of the second triangle, e.g. if you set $p=v_1$ then you will get $q=w_1$.  
> **username_1**: Alternatively you can use $q=w_0 + F(E^TE)^{-1}E^T(p-v_0)$ where $E=\begin{bmatrix} e_1 & e_2 \end{bmatrix}$ and $F=\begin{bmatrix} f_1 & f_2 \end{bmatrix}$. The difference to the first approach is that if $p$ does not lie in the plane of the first triangle, then it will first be projected there and then be transformed.  
> **username_2**: The idea of computing a matrix transform that takes a given triangle and generates a specific output triangle is valid. But that computation will *not* look like "put the positions of the input triangle into the rows of a matrix" or whatever else you're doing here. It's not going to look anything like that. You are defining a change in coordinate systems, so you're going to have to compute the coordinate system change from the source to the destination.  
> **username_0**: @username_2: yes you are right, I saw some random videos on Youtube that were telling to do the way I showed initially. I am currently implementing it into code.  
> **username_0**: @username_1: if I understood, I then use the q vector to multiply each point vector? that will give me each component of my matrix3x3, that I will finally concatenate in my matrix 4x4, then add my translation component?  
> **username_1**: No. If you want a 4x4 matrix from the first formulation then the 3x3 part is $FE^{-1}$ and the translation (4th column) is $w_0-FE^{-1}v_0$. If you want to use the second formulation then the 3x3 part is $F(E^TE)^{-1}E^T$ and the translation is $w_0-F(E^TE)^{-1}E^Tv_0$. Note that for performance reasons you can analytically compute $E^TE=\begin{bmatrix} e_1 \cdot e_1 & e_1 \cdot e_2 \\ e_2 \cdot e_1 & e_2\cdot e_2\end{bmatrix}$  and its inverse (2x2 inverse).  
> **username_0**: @username_1: it works  :). Thanks a lot. May I ask you what T means in E^T (I just implemented the first version)? Translation? And, if possible, how is this calculation called in the litterature, so that I get better in this field by reading? Again, thanks a lot.  
> **username_1**: $M^T$ means $M$ transposed. The $FE^{-1}$ part is called change of basis. Specifically you go from global coordinates in $E$ to local (barycentric) coordinates through $E^{-1}$ and then those get mapped to global coordinates wrt another basis  $F$. The $-v_0$ and $w_0$ makes it a change of coordinate system (basis + origin). The first approach has missing information regarding whether scale is applied along the axis perpendicular to the triangle, it makes the assumption that the change in scale along this axis is the same as the change in area since $|a\times b| = parallelogram\_area(a,b)$.  
> **username_1**: The second approach deals away with the arbitrary choice of scale along the axis perpendicular to the triangle, by requiring the point to be transformed $p$, to be in the plane defined by the triangle. If $p$ is not on the triangle, then $E^{+} = (E^TE)^{-1}E^T$ projects it on the triangle and then finds its barycentric coordinates. $E^{+}$ is known as the Moore-Penrose pseudoinverse. If $p$ is on the triangle then $E^{+}$ simply finds its barycentric coordinates. This means that the two approaches agree for points in the plane of the initial triangle, but differ otherwise.  
> **username_0**: @username_1: thanks a lot, you gave me plenty of information and helped me learn something useful. +1  

---

## 1 answers

### Answer by username_1 on 2022-07-06. Score: 0 | Accepted answer

Here is a more detailed description of what I wrote in the comments. Ideally such a description should come with a ton of graphics illustrating all of the discussed ideas. I did not have the time to produce said images, so my answer is lacking in that respect.

**Summary For Implementation Purposes**

Let the vertices of the first triangle be $v\_0, v\_1, v\_2\in\mathbb{R}^3$ and the vertices of the second triangle be $w\_0, w\_1, w\_2\in\mathbb{R}^3$. For a unique solution we require that the first triangle is non-degenerate: it must have non-zero area (we cannot have the 3 vertices lying on a single line or in a single point).

Compute $e\_1 = v\_1-v\_0, \, e\_2 = v\_2-v\_0$ and $f\_1 = w\_1-w\_0, \, f\_2 = w\_2 - w\_0$.

*Option 1*

Compute $e\_3 = e\_1 \times e\_2$ and $f\_3 = f\_1 \times f\_2$ where $\times$ is the [cross product](https://en.wikipedia.org/wiki/Cross_product#Coordinate_notation). Then form the $3\times 3$ matrices $E$ and $F$ using the $e\_i$ and $f\_i$ as column vectors:

$$
\begin{gather}
E = \begin{bmatrix} e\_1 & e\_2 & e\_3 \end{bmatrix}, \quad F = \begin{bmatrix} f\_1 & f\_2 & f\_3\end{bmatrix}.
\end{gather}
$$

Finally, compute $M = FE^{-1}$ where $E^{-1}$ is the inverse of $E$, and compute $m\_3 = w\_0 - Mv\_0$. Set $M$ as the $3\times 3$ part of your $4\times 4$ matrix $M'$ and $m\_3$ as its last column:

$$
\begin{equation}
M' = \begin{bmatrix} m\_0 & m\_1 & m\_2 & m\_3 \\ 0 & 0 & 0 & 1 \end{bmatrix}.
\end{equation}
$$

*Option 2*

Form the $3\times 2$ matrices $E$ and $F$:
$$
\begin{equation}
E = \begin{bmatrix} e\_1 & e\_2 \end{bmatrix}, \quad F = \begin{bmatrix} f\_1 & f\_2 \end{bmatrix}.
\end{equation}
$$
The form of the solution is similar to the one from *Option 1*:
$$
\begin{gather}
M = FE^{+}, \quad m\_3 = w\_0 - FE^{+}v\_0,\quad E^{+} = (E^TE)^{-1}E^T \\
M' = \begin{bmatrix} m\_0 & m\_1 & m\_2 & m\_3 \\ 0 & 0 & 0 & 1 \end{bmatrix}.
\end{gather}
$$

*Differences Between Option 1 and Option 2*

The two approaches (*Option 1* and *Option 2*) agree when applied to points $p$ that lie in the plane of the triangle $v\_0, v\_1, v\_2$, however they differ if $p$ is not in this plane.

*Option 1* rescales space perpendicular to the triangles $v\_0, v\_1, v\_2$ and $w\_0, w\_1, w\_2$ according to the ratio $r = \frac{area(w\_0, w\_1, w\_2)}{area(v\_0,v\_1,v\_2)}$. For example, if $r = 2$ (the second triangle has twice larger area than the first) and $p$ was $3$ units perpendicularly away from the first triangle's plane, then its transformed version would be $3\times r = 3\times 2 = 6$ units perpendicularly away from the second triangle.

If $p$ is not inside the plane in which $v\_0, v\_1, v\_2$ lies, then *Option 2* projects $p$ on this plane, and only then does it transform it. That is, you would always get a transformed point in the plane of the second triangle $w\_0,w\_1,w\_2$ regardless of what point you feed in, since it would first get projected on the plane of the first triangle. For example if you had a point $p$ that is $3$ units perpendicularly away from the plane of the first triangle, then it will end up $0$ units perpendicularly away from the plane of the second triangle (i.e. it will be in it).

**Introduction**

The following sections contain a more detailed treatment of the problem along with guidelines on ideas that may be used to derive the final solution.

I'll start with a simplified case for clarity. It turns out that studying [affine transformations](https://en.wikipedia.org/wiki/Affine_transformation#Representation) between [tetrahedra](https://en.wikipedia.org/wiki/Tetrahedron) in 3-D is simpler than studying affine transformations between triangles in 3-D. Studying transformations between triangles is simplest in 2-D for instance, and for $n$-D it is simplest to study transformations between [$n$-simplices](https://en.wikipedia.org/wiki/Simplex) (a $2$-simplex is a triangle, and a $3$-simplex is a tetrahedron). This is due to the fact that [an $n$-dimensional vector space is uniquely represented through exactly $n$ basis vectors](https://en.wikipedia.org/wiki/Dimension_(vector_space)). Consequently, studying transformations between triangles (a 2-D object) in 3-D results in some difficulties compared to studying tetrahedra, since in 3-D the triangle case leaves out one of the basis vectors.

**Tetrahedron Case: Formulation**

Let $v\_0,v\_1,v\_2, v\_3 \in\mathbb{R}^3$ be the vertices of a non-degenerate tetrahedron $\mathcal{T}\_{\mathcal{V}}$ in 3-D, and let $w\_0, w\_1, w\_2, w\_3\in\mathbb{R}^3$ be the vertices of another (not necessarily non-degenerate) tetrahedron $\mathcal{T}\_{\mathcal{W}}$. We want an [affine](https://en.wikipedia.org/wiki/Affine_transformation#Representation) (linear + offset) function $g:\mathcal{T}\_{\mathcal{V}}\to\mathcal{T}\_{\mathcal{W}}$ that maps the first tetrahedron to the second one. As a special case the corresponding vertices must map to each other:

$$
\begin{gather}
g(v\_0) = w\_0, \quad
g(v\_1) = w\_1, \quad
g(v\_2) = w\_2, \quad
g(v\_3) = w\_3.
\end{gather}
$$

Since $g(p)$ is affine and acts on 3-D points, then it follows from the definition that [it can be represented by a multiplication with a 3x3 matrix and a translation](https://en.wikipedia.org/wiki/Affine_transformation#Representation):

$$
\begin{equation}
g(p) = Mp + m\_3, \quad M = \begin{bmatrix} m\_0 & m\_1 & m\_2 \end{bmatrix}, \quad m\_0, m\_1, m\_2, m\_3 \in\mathbb{R}^3.
\end{equation}
$$

That is, we have a 3x3 matrix $M$ with 3-D column vectors $m\_0, m\_1, m\_2$ and one translation/offset 3-D vector $m\_3$. Rewriting the constraints using the above form of $g$ yields:

$$
\begin{align}
g(v\_0) = w\_0 &\iff m\_0 v\_{0,x} + m\_1 v\_{0,y} + m\_2 v\_{0,z} + m\_3 = w\_0 \\
g(v\_1) = w\_1 &\iff m\_0 v\_{1,x} + m\_1 v\_{1,y} + m\_2 v\_{1,z} + m\_3 = w\_1 \\
g(v\_2) = w\_2 &\iff m\_0 v\_{2,x} + m\_1 v\_{2,y} + m\_2 v\_{2,z} + m\_3 = w\_2 \\
g(v\_3) = w\_3 &\iff m\_0 v\_{3,x} + m\_1 v\_{3,y} + m\_2 v\_{3,z} + m\_3 = w\_3.
\end{align}
$$

**Tetrahedron Case: Solution**

The above is a system with $4\times 3=12$ unknowns (4 $m\_i$ vectors each with 3 components) and $4\times 3=12$ equations. From linear algebra we know that [as long as the equations are not linearly dependent this system has a unique solution](https://en.wikipedia.org/wiki/Rank%E2%80%93nullity_theorem). We can express $m\_0$ from the first equation:

$$
\begin{equation}
m\_3 = w\_0 - (m\_0 v\_{0,x} + m\_1 v\_{0,y} + m\_2 v\_{0,z}).
\end{equation}
$$

Plugging this expression in the remaining equations reduces the system to 9 unknowns:
$$
\begin{align}
m\_0 e\_{1,x} + m\_1 e\_{1,y} + m\_2 e\_{1,z} &= f\_1 \\
m\_0 e\_{2,x} + m\_1 e\_{2,y} + m\_2 e\_{2,z} &= f\_2 \\
m\_0 e\_{3,x} + m\_1 e\_{3,y} + m\_2 e\_{3,z} &= f\_3,
\end{align}
$$

where $e\_1 = v\_1-v\_0, \, e\_2 = v\_2-v\_0, \, e\_3 = v\_3-v\_0$ and $f\_1 = w\_1-w\_0, \, f\_2 = w\_2-w\_0, \, f\_3 = w\_3-w\_0$. This can be written compactly in matrix form:

$$
\begin{gather}
M E = F, \, E = \begin{bmatrix} e\_1 & e\_2 & e\_3 \end{bmatrix}, \, F = \begin{bmatrix} f\_1 & f\_2 & f\_3 \end{bmatrix}\\
\implies \\
M = E^{-1}F, \, m\_3 = w\_0 - Mv\_0.
\end{gather}
$$
The matrix [$E^{-1}$ exists provided that its columns are linearly independent](https://en.wikipedia.org/wiki/Invertible_matrix#The_invertible_matrix_theorem). This is the case when the tetrahedron with vertices $v\_0, v\_1, v\_2, v\_3$ is non-degenerate: the points do not coincide, they do not lie on a single line, they do not lie in a single plane, i.e. [it's a proper pyramid with non-zero volume](https://en.wikipedia.org/wiki/Determinant#Volume_and_Jacobian_determinant) (not a squished one).

[We can form a $4\times 4$ matrix](https://en.wikipedia.org/wiki/Affine_transformation#Augmented_matrix) by using $M$ as the $3\times 3$ part, $m\_3$ as the last column, and setting $\begin{bmatrix} 0 & 0 & 0 & 1 \end{bmatrix}$ in the last row. Then the transformation takes the form:

$$
\begin{equation}
f(p) = q \iff \begin{bmatrix} m\_0 & m\_1 & m\_2 & m\_3 \\ 0 & 0 & 0 & 1 \end{bmatrix}\begin{bmatrix} p \\ 1 \end{bmatrix} = \begin{bmatrix} q \\ 1 \end{bmatrix}
\end{equation}
$$

Note also that we could have taken any of the vertices as origin instead of $v\_0, w\_0$:
$$
\begin{gather}
m\_3 = w\_i - (m\_0 v\_{i,x} + m\_1 v\_{i,y} + m\_2 v\_{i,z}) \\
e\_j = v\_j - v\_i, \, f\_j = w\_j - w\_i, \, j\ne i,
\end{gather}
$$
so the choice of $v\_0,w\_0$ as origins in the two coordinate systems was arbitrary.

**Triangle Case: Formulation**

The setting of transforming between two triangles in 3-D is similar to the tetrahedra case, however $v\_3$ and $w\_3$ are not provided. We have:
$$
\begin{align}
g(v\_0) = w\_0 &\iff m\_0 v\_{0,x} + m\_1 v\_{0,y} + m\_2 v\_{0,z} + m\_3 = w\_0 \\
g(v\_1) = w\_1 &\iff m\_0 v\_{1,x} + m\_1 v\_{1,y} + m\_2 v\_{1,z} + m\_3 = w\_1 \\
g(v\_2) = w\_2 &\iff m\_0 v\_{2,x} + m\_1 v\_{2,y} + m\_2 v\_{2,z} + m\_3 = w\_2,
\end{align}
$$
which is a system of $3\times 3=9$ equations, but there are $3\times 4 = 12$ unknowns. This means that the system is underdetermined $\implies$ it has infinitely many solutions even if $v\_0, v\_1, v\_2$ is a non-degenerate triangle (has non-zero area). There are various approaches to reduce the set of solutions to a single one.

**Triangle Case: Solution**

The first option is to supply a 3rd vector $e\_3$ that is [linearly independent](https://en.wikipedia.org/wiki/Linear_independence) of $e\_1, e\_2$ (doesn't lie in the same plane as $e\_1$ and $e\_2$), and similarly $f\_3$ that isn't necessarily linearly independent of $f\_1, f\_2$. Then this becomes a special case of the tetrahedra transformations with $v\_3 = v\_0 + e\_3$ and $w\_3 = w\_0 + f\_3$. Depending on the direction and length of $e\_3$ and $f\_3$ the space outside of the planes in which the triangles lie gets sheared and scaled differently.

It also becomes geometrically clear at this point why we have $12$ unknowns and $9$ equations: the remaining $12-9=3$ unknowns correspond to this choice of 3-dimensional vectors. One would of course note that we had to specify two vectors: $e\_3$ and $f\_3$, which constitutes $6$ free variables instead of the $3$ pointed out above. The catch is that those $6$ variables get mixed together through $FE^{-1}$ in such a way that they could be represented through just $3$ free variables.

One simple option that guarantees linear independence is $e\_3 = e\_1 \times e\_2$ and $f\_3 = f\_1 \times f\_2$ where $\times$ is the [cross product](https://en.wikipedia.org/wiki/Cross_product). The geometric implications of this choice are discussed in the **TLDR:** *Differences Between Option 1 and Option 2* subsection. Another option would be to normalize the vectors: $e\_3 = \frac{e\_1 \times e\_2}{\|e\_1\times e\_2\|}$ and $f\_3 = \frac{f\_1 \times f\_2}{\|f\_1\times f\_2\|}$, where $\|\cdot\|$ is the [Euclidean norm/length](https://en.wikipedia.org/wiki/Norm_(mathematics)#Euclidean_norm) of the vectors. Finally, a choice of the form $e\_3 = c (e\_1\times e\_2)$, where $c\ne0$, and $f\_3=0$ results in a solution consistent with *Option 2*.

The difference to *Option 1* is that points that are perpendicularly $X$ units away from the plane of the first triangle remain $X$ units away from the plane of the second triangle after transformation. That is, there is no dependence on the change of triangle area. Thus the case without normalization (i.e. *Option 1*) assumes the transformation is uniform rescaling with ratio $r = \frac{area(w\_0,w\_1,w\_2)}{area(v\_0,v\_1,v\_2)}$ -> rotation -> translation, while the normalized case assumes that scale may be applied only perpendicular to the normal of the triangle, followed by a rotation and translation.

**Triangle Case: Pseudoinverse Solution**

We take a slightly different approach by starting directly from the underdetermined system:
$$
\begin{align}
g(v\_0) = w\_0 &\iff m\_0 v\_{0,x} + m\_1 v\_{0,y} + m\_2 v\_{0,z} + m\_3 = w\_0 \\
g(v\_1) = w\_1 &\iff m\_0 v\_{1,x} + m\_1 v\_{1,y} + m\_2 v\_{1,z} + m\_3 = w\_1 \\
g(v\_2) = w\_2 &\iff m\_0 v\_{2,x} + m\_1 v\_{2,y} + m\_2 v\_{2,z} + m\_3 = w\_2.
\end{align}
$$
Rewriting $m\_3 = w\_0 - (m\_0 v\_{0,x} + m\_1 v\_{0,y} + m\_2 v\_{0,z})$ and plugging in the remaining two equations yields:
$$
\begin{gather}
m\_0 e\_{1,x} + m\_1 e\_{1,y} + m\_2 e\_{1,z} = f\_1 \\
m\_0 e\_{2,x} + m\_1 e\_{2,y} + m\_2 e\_{2,z} = f\_2 \\
\end{gather}
$$
We can write this more compactly in matrix form:
$$
\begin{gather}
M = \begin{bmatrix} m\_0 & m\_1 & m\_2 \end{bmatrix}, \, E=\begin{bmatrix} e\_1 & e\_2 \end{bmatrix}, \, F=\begin{bmatrix} f\_1 & f\_2\end{bmatrix} \\
M E = F\\
ME = FI \\
ME = F(E^TE)^{-1}(E^TE) \\
ME = \left(F(E^TE)^{-1}E^T\right)E
\end{gather}
$$
In the above $(E^TE)^{-1}$ exists provided $e\_1, e\_2$ are linearly independent (the triangle $v\_0,v\_1,v\_2$ is non-degenerate). By identifying the terms on the left- and right-hand sides we come to the conclusion that one possible form of $M$ is $M = F(E^TE)^{-1}E^T$.

The matrix $E^{+} = (E^TE)^{-1}E^T$ is in fact the [Moore-Penrose pseudoinverse](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse). It essentially projects points on the plane of the first triangle. For example, if we have a point $p$ then [its projection](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse#Projectors) is $p' = EE^+p$. Thus the final solution is:
$$
\begin{equation}
M = FE^{+}, \quad m\_3 = w\_0 - Mv\_0.
\end{equation}
$$
Note the similarity to the solution $M = FE^{-1}$ where $E = \begin{bmatrix} e\_1 & e\_2 & e\_3\end{bmatrix}$ and $F = \begin{bmatrix} f\_1 & f\_2 & f\_3 \end{bmatrix}$. And actually the exact same solution for $M$ can be recovered by choosing $e\_3 = c(e\_1 \times e\_2), c\ne 0$, and $f\_3 = 0$.

For an efficient application of the Moore-Penrose pseudoinverse $E^{+}$ to some vector $p$ the following can be used:
$$
\begin{align}
E^{+}p &= (E^TE)^{-1}E^Tp \\
&= \left(\begin{bmatrix} e\_1\cdot e\_1 & e\_1 \cdot e\_2 \\ e\_2 \cdot e\_1 & e\_2 \cdot e\_2 \end{bmatrix}\right)^{-1}\begin{bmatrix} e\_1^T \\ e\_2^T \end{bmatrix}p \\
&= \frac{1}{(e\_1\cdot e\_1)(e\_2 \cdot e\_2) - (e\_1\cdot e\_2)^2}\begin{bmatrix} e\_2\cdot e\_2 & -e\_1\cdot e\_2 \\ -e\_2 \cdot e\_1 & e\_1\cdot e\_1 \end{bmatrix}\begin{bmatrix} e\_1^T \\ e\_2^T \end{bmatrix}p \\
&= \frac{1}{(e\_1\cdot e\_1)(e\_2 \cdot e\_2) - (e\_1\cdot e\_2)^2}\begin{bmatrix}(e\_2\cdot e\_2)(e\_1\cdot p) - (e\_1\cdot e\_2)(e\_2\cdot p) \\ (e\_1\cdot e\_1)(e\_2\cdot p) -(e\_1\cdot e\_2)(e\_1\cdot p)\end{bmatrix}.
\end{align}
$$
This results in 5 dot products, 6 scalar multiplications, and 3 subtractions for computing the application of $E^{+}$ to some vector $p$.

> **Comments:**
> **username_0**: That is a really complete answer, thanks for taking the time to explain in a concise manner. Great  
> **username_0**: Just an update: your solution works perfectly but the practical problem being that option 1 is sensitive to the order of input of the points. ABC != BCA (CCW points) in the final matrix -> and my uvs are wrong in some cases (Please look at my update; but the triangles position is always perfect..) What kind of algo enables to avoid this?  
> **username_1**: @username_0 A vertex $v_i$ maps to $w_i$, it is clear that this should depend on the order. As you noted, providing the wrong order has the effect of reordering the uv matching. You cannot avoid specifying which vertices should have which uvs, since this produces different results. You shouldn't expect that an uv ordering of ABC results in the same texturing as BCA, it is simply not true.  
> **username_0**: I get it, I will add some intelligence there. Thanks a lot :)  

---

