<blockquote>
  <p>then we dot product that normal with one of the 3 vertices</p>
</blockquote>

<p>That's incorrect. It's not difficult to construct a scenario where this will result in an incorrect determination of facing.</p>

<p>Imagine a triangle that is nearly edge-on with the view, but is still facing the view. Now, imagine that this triangle has a normal that, when edge-on, is pointed slightly away from the view. Remember: vertex normals on a surface that is supposed to be curved represent the normal at that portion of the curved surface. And therefore they don't have to be aligned with the face normal.</p>

<p>In any case, the math for determining the facing of a triangle happens in <em>window</em> space, not clip-space. It's pretty simple: you get the direction from the first position to the second and the direction from the second to the third. And you take the cross-product of them. If the Z component of the cross product is positive, then the triangle is counter-clockwise, relative to the window. So this doesn't even require the entire cross product; just the Z component (and just the sign of it at that).</p>

CC BY-SA 4.0
2020-02-03T18:57:50.773Z
actually I'm using about the face normal for the dot product, not the normals that go along with the vertices. But perhaps I'm misunderstanding your point? What do you mean by edge-on?
Harry Kane
CC BY-SA 4.0
2020-02-03T20:36:35.293Z
@HarryKane: "*What do you mean by edge-on?*" Hold your hand in front of your face, with the palm facing you. Now rotate your wrist 90 degrees. That's edge-on.
Nicol Bolas
CC BY-SA 4.0
2020-02-03T14:41:39.860Z
2020-02-03T14:41:39.860Z
Nicol Bolas
<p>It's actually fairly simple (and in essence 'equivalent' to what you said above). If you represent triangle ABC's <em>projected</em> XY coordinates as a matrix...</p>

<p><span class="math-container">$$M_{ABC} =   \begin{bmatrix}
    X_A &amp; X_B &amp; X_C \\
    Y_A &amp; Y_B &amp; Y_C \\
    1 &amp; 1 &amp; 1 \\
    \end{bmatrix}
$$</span></p>

<p>...then the sign of the determinant e.g. <span class="math-container">$X_A (Y_B - Y_C)+X_B(Y_C-Ya)+X_C(Y_A-Y_B)$</span> will allow you to decide how to cull. 
Further if the determinant is zero, then the on-screen triangle has zero area and can clearly be removed.</p>

<p>Another reason to use this notation is that the adjoint of <span class="math-container">$M_{ABC}$</span>  gives you equations for the edges.</p>

CC BY-SA 4.0
2020-02-03T14:21:59.703Z
Can you expand on the math behind it? Why does the sign of determinant tell you which face the triangle is facing?
Harry Kane
CC BY-SA 4.0
2020-02-04T10:48:51.400Z
The determinant of that matrix is proportional to the *signed* area of the triangle (can't recall the exact relationship and am a bit busy to plug in the values at present - sorry). In the meantime,  I will try to give some thought as to an explanation.
Simon F
CC BY-SA 4.0
2021-02-06T17:57:17.310Z
Positive is front facing, Negative is back facing...by default. Also, if you stare at the determinant of a 3x3 matrix for to long it starts to look an awful lot like the scalar triple product. In fact it is 1/scalar triple product.  (edit because I inadvertently wrote vector triple...which of course it is not)
pmw1234
CC BY-SA 4.0
2020-02-03T09:51:12.430Z
2020-02-03T09:51:12.430Z
Simon F
<p><a href="https://www.khronos.org/registry/OpenGL/specs/gl/glspec46.core.pdf" rel="nofollow noreferrer">The OpenGL spec</a> section 14.6.1 spells it out</p>
<blockquote>
<h2>14.6.1 Basic Polygon Rasterization</h2>
<p>The first step of polygon rasterization is to determine if the polygon is back-facing
or front-facing. This determination is made based on the sign of the (clipped or
unclipped) polygon’s area computed in window coordinates. One way to compute
this area is</p>
<p><a href="https://i.sstatic.net/tFs8W.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tFs8W.png" alt="enter image description here" /></a></p>
<p>where  = 1 when the clip control origin is <code>LOWER_LEFT</code> and  = −1 when the
origin is <code>UPPER_LEFT</code>, <em>x<sup>i</sup><sub>w</sub></em> and <em>y<sup>i</sup><sub>w</sub></em> are the <em>x</em> and <em>y</em> window coordinates of the <em>i</em>th
vertex of the <em>n</em>-vertex polygon (vertices are numbered starting at zero for purposes
of this computation) and <em>i</em> ⊕ 1 is (<em>i</em> + 1) mod <em>n</em>. The interpretation of the sign of
this value is controlled with</p>
<pre><code>void FrontFace( enum dir );
</code></pre>
<p>Setting <em>dir</em> to <code>CCW</code> (corresponding to counter-clockwise orientation of the projected polygon in window coordinates) uses <em>a</em> as computed above. Setting <em>dir</em> to
<code>CW</code> (corresponding to clockwise orientation) indicates that the sign of <em>a</em> should be
reversed prior to use. Front face determination requires one bit of state, and is
initially set to <code>CCW</code>.</p>
<p>If the sign of <em>a</em> (including the possible reversal of this sign as determined by
<strong>FrontFace</strong>) is positive, the polygon is front-facing; otherwise, it is back-facing.
This determination is used in conjunction with the <strong>CullFace</strong> enable bit and mode
value to decide whether or not a particular polygon is rasterized. The <strong>CullFace</strong>
mode is set by calling</p>
<pre><code>void CullFace( enum mode );
</code></pre>
<p><em>mode</em> must be <code>FRONT</code>, <code>BACK</code> or <code>FRONT_AND_BACK</code>. Culling is enabled or disabled
by calling <strong>Enable</strong> or <strong>Disable</strong> with target <code>CULL_FACE</code>. Front-facing polygons are
rasterized if either culling is disabled or the <strong>CullFace</strong> mode is <code>BACK</code> while back facing polygons are rasterized only if either culling is disabled or the <strong>CullFace</strong>
mode is <code>FRONT</code>. The initial setting of the <strong>CullFace</strong> mode is <code>BACK</code>. Initially, culling
is disabled</p>
</blockquote>

CC BY-SA 4.0
2021-02-01T15:24:17.770Z
2021-02-01T15:24:17.770Z
gman
<p>When I was writing a software rasterizer for my university class, we did culling in  eye coordinates. Essentially, we use cross product to get a normal from the 3 vertices (in eye coordinates), then we dot product that normal with one of the 3 vertices (since we are in eye coordinate, that vertex is essentially a vector from eye [0, 0, 0,.1] to triangle surface). If the dot product is > 0, meaning the normal and the eye vector point in the same direction, we cull that triangle.</p>

<p>OpenGL however doesn't do culling in eye space, since culling is after vertex shader. So all they have are vertices in clip space. So how do they know which triangle to do culling? Our triangles, when fed into the graphic pipeline, are always in CCW order. How do some turn into CW order?</p>

CC BY-SA 4.0
2021-02-06T16:39:55.817Z
Graphics API's don't "know" which triangles to cull, we the developers tell them with 1) Winding direction...CW or CCW and 2) Which face to cull .. Front face or Back Face. OpenGL has these value preset to CCW, back face. And that little extra info is all that the API needs (using the math outlined below) to figure it out.
pmw1234
<opengl>
How does graphics api like OpenGL determine which triangle is back face to cull?
Harry Kane
# How does graphics api like OpenGL determine which triangle is back face to cull?

(Asked by: username_0 on 2020-02-02)

When I was writing a software rasterizer for my university class, we did culling in eye coordinates. Essentially, we use cross product to get a normal from the 3 vertices (in eye coordinates), then we dot product that normal with one of the 3 vertices (since we are in eye coordinate, that vertex is essentially a vector from eye [0, 0, 0,.1] to triangle surface). If the dot product is > 0, meaning the normal and the eye vector point in the same direction, we cull that triangle.

OpenGL however doesn't do culling in eye space, since culling is after vertex shader. So all they have are vertices in clip space. So how do they know which triangle to do culling? Our triangles, when fed into the graphic pipeline, are always in CCW order. How do some turn into CW order?

**Tags:** opengl

> **Comments:**
> **username_1**: Graphics API's don't "know" which triangles to cull, we the developers tell them with 1) Winding direction...CW or CCW and 2) Which face to cull .. Front face or Back Face. OpenGL has these value preset to CCW, back face. And that little extra info is all that the API needs (using the math outlined below) to figure it out.  

---

## 3 answers

### Answer by username_2 on 2020-02-03. Score: 1

> then we dot product that normal with one of the 3 vertices

That's incorrect. It's not difficult to construct a scenario where this will result in an incorrect determination of facing.

Imagine a triangle that is nearly edge-on with the view, but is still facing the view. Now, imagine that this triangle has a normal that, when edge-on, is pointed slightly away from the view. Remember: vertex normals on a surface that is supposed to be curved represent the normal at that portion of the curved surface. And therefore they don't have to be aligned with the face normal.

In any case, the math for determining the facing of a triangle happens in *window* space, not clip-space. It's pretty simple: you get the direction from the first position to the second and the direction from the second to the third. And you take the cross-product of them. If the Z component of the cross product is positive, then the triangle is counter-clockwise, relative to the window. So this doesn't even require the entire cross product; just the Z component (and just the sign of it at that).

> **Comments:**
> **username_0**: actually I'm using about the face normal for the dot product, not the normals that go along with the vertices. But perhaps I'm misunderstanding your point? What do you mean by edge-on?  
> **username_2**: @username_0: "*What do you mean by edge-on?*" Hold your hand in front of your face, with the palm facing you. Now rotate your wrist 90 degrees. That's edge-on.  

---

### Answer by username_3 on 2020-02-03. Score: 1

It's actually fairly simple (and in essence 'equivalent' to what you said above). If you represent triangle ABC's *projected* XY coordinates as a matrix...

$$M\_{ABC} = \begin{bmatrix}
X\_A & X\_B & X\_C \\
Y\_A & Y\_B & Y\_C \\
1 & 1 & 1 \\
\end{bmatrix}
$$

...then the sign of the determinant e.g. $X\_A (Y\_B - Y\_C)+X\_B(Y\_C-Ya)+X\_C(Y\_A-Y\_B)$ will allow you to decide how to cull.
Further if the determinant is zero, then the on-screen triangle has zero area and can clearly be removed.

Another reason to use this notation is that the adjoint of $M\_{ABC}$ gives you equations for the edges.

> **Comments:**
> **username_0**: Can you expand on the math behind it? Why does the sign of determinant tell you which face the triangle is facing?  
> **username_3**: The determinant of that matrix is proportional to the *signed* area of the triangle (can't recall the exact relationship and am a bit busy to plug in the values at present - sorry). In the meantime,  I will try to give some thought as to an explanation.  
> **username_1**: Positive is front facing, Negative is back facing...by default. Also, if you stare at the determinant of a 3x3 matrix for to long it starts to look an awful lot like the scalar triple product. In fact it is 1/scalar triple product.  (edit because I inadvertently wrote vector triple...which of course it is not)  

---

### Answer by username_4 on 2021-02-01. Score: 1

[The OpenGL spec](https://www.khronos.org/registry/OpenGL/specs/gl/glspec46.core.pdf) section 14.6.1 spells it out

> ## 14.6.1 Basic Polygon Rasterization
>
> The first step of polygon rasterization is to determine if the polygon is back-facing
> or front-facing. This determination is made based on the sign of the (clipped or
> unclipped) polygon’s area computed in window coordinates. One way to compute
> this area is
>
> [![enter image description here](https://i.sstatic.net/tFs8W.png)](https://i.sstatic.net/tFs8W.png)
>
> where = 1 when the clip control origin is `LOWER_LEFT` and = −1 when the
> origin is `UPPER_LEFT`, *xiw* and *yiw* are the *x* and *y* window coordinates of the *i*th
> vertex of the *n*-vertex polygon (vertices are numbered starting at zero for purposes
> of this computation) and *i* ⊕ 1 is (*i* + 1) mod *n*. The interpretation of the sign of
> this value is controlled with
>
> ```
> void FrontFace( enum dir );
>
> ```
>
> Setting *dir* to `CCW` (corresponding to counter-clockwise orientation of the projected polygon in window coordinates) uses *a* as computed above. Setting *dir* to
> `CW` (corresponding to clockwise orientation) indicates that the sign of *a* should be
> reversed prior to use. Front face determination requires one bit of state, and is
> initially set to `CCW`.
>
> If the sign of *a* (including the possible reversal of this sign as determined by
> **FrontFace**) is positive, the polygon is front-facing; otherwise, it is back-facing.
> This determination is used in conjunction with the **CullFace** enable bit and mode
> value to decide whether or not a particular polygon is rasterized. The **CullFace**
> mode is set by calling
>
> ```
> void CullFace( enum mode );
>
> ```
>
> *mode* must be `FRONT`, `BACK` or `FRONT_AND_BACK`. Culling is enabled or disabled
> by calling **Enable** or **Disable** with target `CULL_FACE`. Front-facing polygons are
> rasterized if either culling is disabled or the **CullFace** mode is `BACK` while back facing polygons are rasterized only if either culling is disabled or the **CullFace**
> mode is `FRONT`. The initial setting of the **CullFace** mode is `BACK`. Initially, culling
> is disabled


---

