<p>I'm working on a simple 3d modeling application. I'm using 1 meter as unit of measurement. When working with small triangles where an edge length is around 1 centimeter (0.01 units) or less, I started to encounter issues with floating point precision (I'm using 4 bytes float data type). More specifically with the algorithm for calculating barycentric coordinates (which is used in ray triangle intersection). In rough terms the algorithm calculates dot products of edge vectors and then multiplies the said dot products with each other, which results in very small values.</p>
<p>From what I understand Maya and some other modeling software use 1 cm as measurement unit. However not all it seems, Blender for example uses 1 m as measurement unit (?)</p>
<p>Would it make sense to switch to 1 cm to avoid these precision issues? I would still get the precision issues at around 0.01 units, but this would then correspond to 0.1 mm which is more then enough for me.</p>

CC BY-SA 4.0
2023-07-12T09:23:05.890Z
Are these calculations be applied to variables calculated by a perspective projection matrix? If so, check the near and far planes of this matrix. If the near plane is too close to the camera and the far plane is too far away, the interpolated values will lose precision...
Thomas
<mesh><model><float>
Floating point precision in 3d modeling software
Lenny White
# Floating point precision in 3d modeling software

(Asked by: username_0 on 2023-07-09)

I'm working on a simple 3d modeling application. I'm using 1 meter as unit of measurement. When working with small triangles where an edge length is around 1 centimeter (0.01 units) or less, I started to encounter issues with floating point precision (I'm using 4 bytes float data type). More specifically with the algorithm for calculating barycentric coordinates (which is used in ray triangle intersection). In rough terms the algorithm calculates dot products of edge vectors and then multiplies the said dot products with each other, which results in very small values.

From what I understand Maya and some other modeling software use 1 cm as measurement unit. However not all it seems, Blender for example uses 1 m as measurement unit (?)

Would it make sense to switch to 1 cm to avoid these precision issues? I would still get the precision issues at around 0.01 units, but this would then correspond to 0.1 mm which is more then enough for me.

**Tags:** mesh, model, float
> **Comments:**
> **username_1**: Are these calculations be applied to variables calculated by a perspective projection matrix? If so, check the near and far planes of this matrix. If the near plane is too close to the camera and the far plane is too far away, the interpolated values will lose precision...  

---

## 0 answers

