<p>If you're interested in vertex normals specifically, there's an easy answer even for non-planar polygons that avoids the question of defining what the exact surface is: for each vertex, calculate the normal of the plane formed by the two edges entering and leaving that vertex.</p>
<p>More formally, given vertices <span class="math-container">$\mathbf{v}_1, \mathbf{v}_2, \ldots \mathbf{v}_n$</span> with counterclockwise winding, define the normal at the <span class="math-container">$i$</span>th vertex as:
<span class="math-container">$$
\mathbf{n}_i = (\mathbf{v}_i - \mathbf{v}_{i-1}) \times (\mathbf{v}_{i+1} - \mathbf{v}_i)
$$</span>
(where the indices wrap around).</p>
<p>You can then proceed to accumulate normals calculated this way from all the faces that share a given vertex, as usual for smooth shading.</p>
<p>As has been discussed, there are multiple ways to define a smooth surface corresponding to a non-planar polygon, but <em>any</em> reasonable way of defining such a surface must converge to the normals as defined here near the vertices, or else the surface can not both be smooth (locally flat) and meet the straight-line edges between the vertices.</p>
<p>A caveat to this approach, though, is that it won't define a normal for collinear vertices (where the entering and exiting edges are parallel), since the cross product goes to zero there. If this is a problem, it might work to patch up such vertices by interpolating normals to them from the surrounding non-collinear vertices.</p>

CC BY-SA 4.0
2021-03-19T17:44:26.877Z
Thank you so much, this is perfect! Could I ask if there's any literature describing this method so I can read up more on why this works?
Lenny White
CC BY-SA 4.0
2021-03-19T18:02:46.350Z
Or generally I would like to find more on this topic, but I don't know to google it.
Lenny White
CC BY-SA 4.0
2021-03-19T18:10:18.740Z
Sorry, I don't really know if there's any literature on this; I'm just going off geometric intuition.
Nathan Reed
CC BY-SA 4.0
2021-03-19T18:24:28.103Z
I see, appreciate it anyway! Can I ask "*or else the surface can not both be smooth (locally flat) and meet the straight-line edges between the vertices*". This is a bit too advanced for me. Could you explain what this means in simpler terms?
Lenny White
CC BY-SA 4.0
2021-03-19T22:27:42.147Z
If the surface is smooth, then when you zoom in and look at any very small region of it, that region should appear fairly flat (this is basically what it means to be smooth). And the surface must conform to the given vertices and edges, whatever else it does in the middle. So, if you zoom in on a small region near an edge, the surface there must run parallel to the edge so that it can meet the edge. And if you zoom in on a vertex, the surface has to be parallel to both edges around that vertex. That completely determines the orientation of the surface in that small region.
Nathan Reed
CC BY-SA 4.0
2021-03-19T17:27:05.687Z
2021-03-19T17:27:05.687Z
Nathan Reed
<p>for smooth shading you usually store the normal vectors per vertex and not per face (flat shading). The normal vector on a specific point on a face will then be calculated by interpolating the normal vectors of the vertices (corners) of the face.</p>
<p><strong>For example when having a triangle:</strong>
you would use an interpolation which uses the barycentric coordinates of three normal vectors like discribed <a href="http://15462.courses.cs.cmu.edu/fall2015/lecture/texture/slide_019" rel="nofollow noreferrer">here</a>.</p>
<p>For non planar polygons, of cause you don't have a specific defined surface.
Therefore it depend on you, which kind of interpolation you are using. But it should interpolate with respect to the corners distance.</p>
<p>And don't forget to normalize the vector at the end!</p>

CC BY-SA 4.0
2021-03-18T18:38:48.033Z
Unfortunately this was not what I was wondering about. It's given that we need to calculate the vertex normals and then these are interpolated across the triangle's surface. The question is about how to calculate the vertex normals in the first place. This involves taking the normalized average of all the faces it belongs to, if the all faces are triangles. However what I'm wondering about is what are the ways to handle the case where the faces are non-planar polygons with more than 3 vertices.
Lenny White
CC BY-SA 4.0
2021-03-19T14:59:48.203Z
Maybe you can subdivide the non planar quads/whatever into triangles and use your approach? Or doing something like mesh relaxation that will converge to nice planar results?
Felipe Gutierrez
CC BY-SA 4.0
2021-03-19T16:51:01.683Z
@FelipeGutierrez Making the polygons planar wouldn't work for me unfortunately. I'm working on a simple 3d modeling application. And I need to be able to have non-planar polygons. Triangulating would work, but at first glance at least it seems like a lot of headache to implement this if I need to be able to work with a dynamic mesh.
Lenny White
CC BY-SA 4.0
2021-03-18T16:49:44.920Z
2021-03-18T16:49:44.920Z
Thomas
<p>If I have a mesh consisting of polygons that are not necessarily triangles and not necessarily planar. As answered in the previous <a href="https://computergraphics.stackexchange.com/questions/10750/why-do-3d-modeling-packages-use-a-single-normal-per-polygon-and-how-is-this-viab/10751?noredirect=1#comment16864_10751">question</a> I asked, there's no correct answer to calculating normals for such polygons. So I'm wondering instead no matter how wrong, what are some of the strategies used in 3d modeling to calculate the vertex normals for such a mesh, <em>specifically for smooth shading</em>?</p>

CC BY-SA 4.0
2021-03-18T22:30:54.820Z
What is the surface of such a non-planar "polygon"? There are infinitely many surfaces that can pass through the vertices of such a "polygon". If you define the surface then you can compute the normal based on the partial derivatives.
lightxbulb
CC BY-SA 4.0
2021-03-18T23:08:59.693Z
@lightxbulb I would like to further specify that I'm referring to simple polygons. To my knowledge in 3d modeling the non-planar polygon surface is visualized by triangulating the polygon. So I guess the surface is that of individual triangles the polygon consists of.
Lenny White
CC BY-SA 4.0
2021-03-18T23:18:17.867Z
I unfortunately can't use derivatives to calculate the normals. I need to calculate the surface normals on GPU using triangle interpolation (with OpenGL), and for that I need to use vertex normals.
Lenny White
CC BY-SA 4.0
2021-03-19T08:11:03.300Z
There are more than one way to triangulate a polygon. When it is non-planar the different triangulations result in different surfaces. So first you have to specify what a non-planar polygon means (e.g. specify its surface). Only then can one argue about normals.
lightxbulb
CC BY-SA 4.0
2021-03-19T15:46:02.363Z
@LennyWhite: Q&A sites are not appropriate for questions of the form "give me a list of all the ways to do X". We generally prefer questions that are more specific in nature.
Nicol Bolas
CC BY-SA 4.0
2021-03-19T16:02:29.500Z
@NicolBolas I strongly disagree with you in this case.  In the previous question I referenced, I essentially asked for the right way to do this. There's no one right way. So then it seems logical to me to ask for all the possible ways to do this.
Lenny White
CC BY-SA 4.0
2021-03-19T16:11:42.267Z
@lightxbulb Why can't we just consider one specific case of triangulation? Say I have triangulated the polygon in some specific way. How do I calculate normals for that specific case of triangulated polygon is what I'm wondering.
Lenny White
CC BY-SA 4.0
2021-03-19T16:25:38.420Z
@LennyWhite: "*How do I calculate normals for that specific case of triangulated polygon is what I'm wondering.*" If you already have triangulated it, then there are no "non-planar polygons" anymore. So your question seems to have answered itself.
Nicol Bolas
CC BY-SA 4.0
2021-03-19T16:46:21.023Z
@NicolBolas I guess it has. So the answer then would be simply, if you have a polygonal mesh, you first triangulate it to define the surface, and then calculate normals for the triangular mesh. I guess when asking the question I thought maybe there were some alternative ways to handle this without triangulating the polygons in first place.
Lenny White
CC BY-SA 4.0
2021-03-19T16:46:26.210Z
For instance would this be a completely incorrect approach. For each vertex iterate CW/CCW through all the edges it belongs to, then for each two edges (`i` and `(i+1)%numEdges`) define a triangle. Lastly calculate the vertex normal as the normalized average of all those triangles.
Lenny White
CC BY-SA 4.0
2021-03-19T16:58:09.497Z
@LennyWhite: "*if you have a polygonal mesh*" No. If I had a mesh containing non-planar polygons, I would go back to the person who gave me that mesh and tell them to *fix the mesh*. Tell them to figure out what they really intended for the mesh's surface. If the mesh came from an algorithm, I'd fix the bug in the algorithm that caused it to produce non-planar polygons. Non-planar polygons are a *bug*. You don't live with them or work around them; you *fix them*.
Nicol Bolas
CC BY-SA 4.0
2021-03-19T17:05:21.147Z
@NicolBolas In this case the non-planar polygons are produced by end-user who wants to model in 3d modeling application, and create a polygon by connecting a bunch of points. I have no control over if the resulting polygons are going to be planar or not.
Lenny White
CC BY-SA 4.0
2021-03-19T19:23:58.007Z
Connecting a bunch of points is not enough to define a non-planar surface uniquely. A specific approach there cannot be considered more correct in the absence of additional constraints. It's an ill-posed problem. Note that the surface of a "non-planar polygon" can also be a smooth curved surface and not a triangulation. If you are given just a point cloud and you want to reconstruct a surface from that, there are various approaches, but they are beyond the scope of your current question.
lightxbulb
CC BY-SA 4.0
2021-03-19T19:51:43.060Z
@lightxbulb But what's wrong with calculating the vertex normals the way Nathan suggested. Why try to define the polygon surface in the first place if you can just use the edges around a given vertex to calculate its(vertex's) normal.
Lenny White
CC BY-SA 4.0
2021-03-19T20:40:02.243Z
Because what Nathan suggested results in incorrect normals for almost any surface. There are many different algorithms for that working under different assumptions: https://www.researchgate.net/publication/220067106_A_comparison_of_algorithms_for_vertex_normal_computation There may be semi-plausible like the ones you'd use for "smooth triangles", but if you already have the surface why not calculate the correct ones? You have to define a surface either way if you want to draw anything. You can't just throw a "non-planar polygon" at a rendering API and except reasonable results.
lightxbulb
CC BY-SA 4.0
2021-03-20T15:09:07.737Z
@lightxbulb I understand. Appreciate the article. Didn't know there was research done on this topic.
Lenny White
<opengl><shading><polygon>
How to calculate vertex normals on a mesh with non-planar polygons
Lenny White
# How to calculate vertex normals on a mesh with non-planar polygons

(Asked by: username_0 on 2021-03-18)

If I have a mesh consisting of polygons that are not necessarily triangles and not necessarily planar. As answered in the previous [question](https://computergraphics.stackexchange.com/questions/10750/why-do-3d-modeling-packages-use-a-single-normal-per-polygon-and-how-is-this-viab/10751?noredirect=1#comment16864_10751) I asked, there's no correct answer to calculating normals for such polygons. So I'm wondering instead no matter how wrong, what are some of the strategies used in 3d modeling to calculate the vertex normals for such a mesh, *specifically for smooth shading*?

**Tags:** opengl, shading, polygon

> **Comments:**
> **username_1**: What is the surface of such a non-planar "polygon"? There are infinitely many surfaces that can pass through the vertices of such a "polygon". If you define the surface then you can compute the normal based on the partial derivatives.  
> **username_0**: @username_1 I would like to further specify that I'm referring to simple polygons. To my knowledge in 3d modeling the non-planar polygon surface is visualized by triangulating the polygon. So I guess the surface is that of individual triangles the polygon consists of.  
> **username_0**: I unfortunately can't use derivatives to calculate the normals. I need to calculate the surface normals on GPU using triangle interpolation (with OpenGL), and for that I need to use vertex normals.  
> **username_1**: There are more than one way to triangulate a polygon. When it is non-planar the different triangulations result in different surfaces. So first you have to specify what a non-planar polygon means (e.g. specify its surface). Only then can one argue about normals.  
> **username_2**: @username_0: Q&A sites are not appropriate for questions of the form "give me a list of all the ways to do X". We generally prefer questions that are more specific in nature.  
> **username_0**: @username_2 I strongly disagree with you in this case.  In the previous question I referenced, I essentially asked for the right way to do this. There's no one right way. So then it seems logical to me to ask for all the possible ways to do this.  
> **username_0**: @username_1 Why can't we just consider one specific case of triangulation? Say I have triangulated the polygon in some specific way. How do I calculate normals for that specific case of triangulated polygon is what I'm wondering.  
> **username_2**: @username_0: "*How do I calculate normals for that specific case of triangulated polygon is what I'm wondering.*" If you already have triangulated it, then there are no "non-planar polygons" anymore. So your question seems to have answered itself.  
> **username_0**: @username_2 I guess it has. So the answer then would be simply, if you have a polygonal mesh, you first triangulate it to define the surface, and then calculate normals for the triangular mesh. I guess when asking the question I thought maybe there were some alternative ways to handle this without triangulating the polygons in first place.  
> **username_0**: For instance would this be a completely incorrect approach. For each vertex iterate CW/CCW through all the edges it belongs to, then for each two edges (`i` and `(i+1)%numEdges`) define a triangle. Lastly calculate the vertex normal as the normalized average of all those triangles.  
> **username_2**: @username_0: "*if you have a polygonal mesh*" No. If I had a mesh containing non-planar polygons, I would go back to the person who gave me that mesh and tell them to *fix the mesh*. Tell them to figure out what they really intended for the mesh's surface. If the mesh came from an algorithm, I'd fix the bug in the algorithm that caused it to produce non-planar polygons. Non-planar polygons are a *bug*. You don't live with them or work around them; you *fix them*.  
> **username_0**: @username_2 In this case the non-planar polygons are produced by end-user who wants to model in 3d modeling application, and create a polygon by connecting a bunch of points. I have no control over if the resulting polygons are going to be planar or not.  
> **username_1**: Connecting a bunch of points is not enough to define a non-planar surface uniquely. A specific approach there cannot be considered more correct in the absence of additional constraints. It's an ill-posed problem. Note that the surface of a "non-planar polygon" can also be a smooth curved surface and not a triangulation. If you are given just a point cloud and you want to reconstruct a surface from that, there are various approaches, but they are beyond the scope of your current question.  
> **username_0**: @username_1 But what's wrong with calculating the vertex normals the way username_3 suggested. Why try to define the polygon surface in the first place if you can just use the edges around a given vertex to calculate its(vertex's) normal.  
> **username_1**: Because what username_3 suggested results in incorrect normals for almost any surface. There are many different algorithms for that working under different assumptions: https://www.researchgate.net/publication/220067106_A_comparison_of_algorithms_for_vertex_normal_computation There may be semi-plausible like the ones you'd use for "smooth triangles", but if you already have the surface why not calculate the correct ones? You have to define a surface either way if you want to draw anything. You can't just throw a "non-planar polygon" at a rendering API and except reasonable results.  
> **username_0**: @username_1 I understand. Appreciate the article. Didn't know there was research done on this topic.  

---

## 2 answers

### Answer by username_3 on 2021-03-19. Score: 3 | Accepted answer

If you're interested in vertex normals specifically, there's an easy answer even for non-planar polygons that avoids the question of defining what the exact surface is: for each vertex, calculate the normal of the plane formed by the two edges entering and leaving that vertex.

More formally, given vertices $\mathbf{v}\_1, \mathbf{v}\_2, \ldots \mathbf{v}\_n$ with counterclockwise winding, define the normal at the $i$th vertex as:
$$
\mathbf{n}\_i = (\mathbf{v}\_i - \mathbf{v}\_{i-1}) \times (\mathbf{v}\_{i+1} - \mathbf{v}\_i)
$$
(where the indices wrap around).

You can then proceed to accumulate normals calculated this way from all the faces that share a given vertex, as usual for smooth shading.

As has been discussed, there are multiple ways to define a smooth surface corresponding to a non-planar polygon, but *any* reasonable way of defining such a surface must converge to the normals as defined here near the vertices, or else the surface can not both be smooth (locally flat) and meet the straight-line edges between the vertices.

A caveat to this approach, though, is that it won't define a normal for collinear vertices (where the entering and exiting edges are parallel), since the cross product goes to zero there. If this is a problem, it might work to patch up such vertices by interpolating normals to them from the surrounding non-collinear vertices.

> **Comments:**
> **username_0**: Thank you so much, this is perfect! Could I ask if there's any literature describing this method so I can read up more on why this works?  
> **username_0**: Or generally I would like to find more on this topic, but I don't know to google it.  
> **username_3**: Sorry, I don't really know if there's any literature on this; I'm just going off geometric intuition.  
> **username_0**: I see, appreciate it anyway! Can I ask "*or else the surface can not both be smooth (locally flat) and meet the straight-line edges between the vertices*". This is a bit too advanced for me. Could you explain what this means in simpler terms?  
> **username_3**: If the surface is smooth, then when you zoom in and look at any very small region of it, that region should appear fairly flat (this is basically what it means to be smooth). And the surface must conform to the given vertices and edges, whatever else it does in the middle. So, if you zoom in on a small region near an edge, the surface there must run parallel to the edge so that it can meet the edge. And if you zoom in on a vertex, the surface has to be parallel to both edges around that vertex. That completely determines the orientation of the surface in that small region.  

---

### Answer by username_4 on 2021-03-18. Score: 0

for smooth shading you usually store the normal vectors per vertex and not per face (flat shading). The normal vector on a specific point on a face will then be calculated by interpolating the normal vectors of the vertices (corners) of the face.

**For example when having a triangle:**
you would use an interpolation which uses the barycentric coordinates of three normal vectors like discribed [here](http://15462.courses.cs.cmu.edu/fall2015/lecture/texture/slide_019).

For non planar polygons, of cause you don't have a specific defined surface.
Therefore it depend on you, which kind of interpolation you are using. But it should interpolate with respect to the corners distance.

And don't forget to normalize the vector at the end!

> **Comments:**
> **username_0**: Unfortunately this was not what I was wondering about. It's given that we need to calculate the vertex normals and then these are interpolated across the triangle's surface. The question is about how to calculate the vertex normals in the first place. This involves taking the normalized average of all the faces it belongs to, if the all faces are triangles. However what I'm wondering about is what are the ways to handle the case where the faces are non-planar polygons with more than 3 vertices.  
> **username_5**: Maybe you can subdivide the non planar quads/whatever into triangles and use your approach? Or doing something like mesh relaxation that will converge to nice planar results?  
> **username_0**: @username_5 Making the polygons planar wouldn't work for me unfortunately. I'm working on a simple 3d modeling application. And I need to be able to have non-planar polygons. Triangulating would work, but at first glance at least it seems like a lot of headache to implement this if I need to be able to work with a dynamic mesh.  

---

