<blockquote>
<p>One problem is that the points that are behind or on the camera plane are distorted due to the projection matrix.</p>
</blockquote>
<p>Because of this, you need to consider the box not as 8 points, but 12 line segments. Then, you can <em>clip</em> these line segments individually — shorten them so that you keep only the parts which lie within the visible region where projection works as desired. Those shortened lines can then be projected and drawn normally.</p>
<p>This is how GPUs rasterize arbitrary shapes with an arbitrary perspective projection, except that they do the clipping <em>after</em> projection, in the homogeneous (XYZW) coordinate system.</p>
<blockquote>
<p>It is quite easy to imagine a new image which is further away and then zooming in to this image to match the original one. But why cant we do the same thing mathematically?</p>
</blockquote>
<p>Zooming and moving the camera <strong>cannot</strong> compensate for each other, except in the special case where all points of the scene are at the same depth (e.g. taking a photograph of a flat document). The projections are irreconcilably different. Since your goal is to map boxes accurately onto an existing image, you cannot move the camera.</p>
<p>(If you did find a compensating transform, it would be equivalent to un-projecting and re-projecting with a new perspective projection, so you'd have your original problem over again.)</p>
<blockquote>
<p>Why does Zooming in does not work with Projection Matrix but works in Real Life?</p>
</blockquote>
<p>If you try zooming in and moving back simultaneously in real life, you will find you’ve created the <a href="https://en.wikipedia.org/wiki/Dolly_zoom" rel="nofollow noreferrer">dolly zoom</a>.</p>

CC BY-SA 4.0
2024-08-06T12:08:00.073Z
That explanation is quite helpful, thank you very much. Also, the dolly zoom is proving exactly what I mathematically computed, so thanks for sharing that.

My solution was to only take the boxes that have at least one face fully on the correct side of the image plane, and then shorten the edges of the ones on the wrong side such that all of them are on the correct side.
Yusuf Ziya Güleray
CC BY-SA 4.0
2024-08-06T14:40:24.167Z
@YusufZiyaGüleray That will work only as long as the camera's axis is perpendicular to a face of the box.
Kevin Reid
CC BY-SA 4.0
2024-08-07T15:35:50.823Z
It does not necessarily have to be perpendicular. If at least one face is fully on the correct side, then it means every corner of the box has a connected corner that is on the correct side. Then it only becomes moving the points on the behind side by moving it along the edge that connect it to its front neighbor.

Of course in case it is not perpendicular, the box will become an irregular hexahedron, but I am fine with it.
Yusuf Ziya Güleray
CC BY-SA 4.0
2024-08-05T17:34:29.657Z
2024-08-05T17:34:29.657Z
Kevin Reid
<p>My goal is to project 3D bounding boxes to an image. For those who are not familiar with what 3 Dimensional Bounding Boxes are, they are basically boxes defined by eight corners encapsulating an object of interest. And for visualization purposes, I need to project it on an image.</p>
<p>One problem is that the points that are behind or on the camera plane are distorted due to the projection matrix. And normally filtering them out is quite easy and practical but in the case of 3D bounding boxes, it is possible that a bounding box has some of its points behind and some of its points front of the camera plane. And this boxes needs to be visualized.</p>
<p>My solution was to:</p>
<ol>
<li>Move the camera x meters behind in the extrinsic matrix so that when filtering happens all the originally intersecting bounding boxes are moved completely in front of camera.</li>
<li>Do the filtering</li>
<li>Use a new focal length or an additional correction matrix to compensate the movement of the camera such that it would look as the same with the original pose and focal lengths.</li>
</ol>
<p>But I couldn't find a focal length that was independent from the distance to object that can fix the issue.</p>
<p>So my question is: It is quite easy to imagine a new image which is further away and then zooming in to this image to match the original one. But why cant we do the same thing mathematically?</p>

CC BY-SA 4.0
2024-08-05T12:54:54.623Z
Selam.  1) You can't divide by zero.  You either have to "bite the bullet" and do clipping prior to projection.
2) "works in real life".   You CAN'T see things that are level with your eye's lens.
Simon F
<projections><camera-matrix>
Why does Zooming in does not work with Projection Matrix but works in Real Life?
Yusuf Ziya Güleray
# Why does Zooming in does not work with Projection Matrix but works in Real Life?

(Asked by: username_0 on 2024-08-05)

My goal is to project 3D bounding boxes to an image. For those who are not familiar with what 3 Dimensional Bounding Boxes are, they are basically boxes defined by eight corners encapsulating an object of interest. And for visualization purposes, I need to project it on an image.

One problem is that the points that are behind or on the camera plane are distorted due to the projection matrix. And normally filtering them out is quite easy and practical but in the case of 3D bounding boxes, it is possible that a bounding box has some of its points behind and some of its points front of the camera plane. And this boxes needs to be visualized.

My solution was to:

1. Move the camera x meters behind in the extrinsic matrix so that when filtering happens all the originally intersecting bounding boxes are moved completely in front of camera.
2. Do the filtering
3. Use a new focal length or an additional correction matrix to compensate the movement of the camera such that it would look as the same with the original pose and focal lengths.

But I couldn't find a focal length that was independent from the distance to object that can fix the issue.

So my question is: It is quite easy to imagine a new image which is further away and then zooming in to this image to match the original one. But why cant we do the same thing mathematically?

**Tags:** projections, camera-matrix

> **Comments:**
> **username_1**: Selam.  1) You can't divide by zero.  You either have to "bite the bullet" and do clipping prior to projection. 2) "works in real life".   You CAN'T see things that are level with your eye's lens.  

---

## 1 answers

### Answer by username_2 on 2024-08-05. Score: 0 | Accepted answer

> One problem is that the points that are behind or on the camera plane are distorted due to the projection matrix.

Because of this, you need to consider the box not as 8 points, but 12 line segments. Then, you can *clip* these line segments individually — shorten them so that you keep only the parts which lie within the visible region where projection works as desired. Those shortened lines can then be projected and drawn normally.

This is how GPUs rasterize arbitrary shapes with an arbitrary perspective projection, except that they do the clipping *after* projection, in the homogeneous (XYZW) coordinate system.

> It is quite easy to imagine a new image which is further away and then zooming in to this image to match the original one. But why cant we do the same thing mathematically?

Zooming and moving the camera **cannot** compensate for each other, except in the special case where all points of the scene are at the same depth (e.g. taking a photograph of a flat document). The projections are irreconcilably different. Since your goal is to map boxes accurately onto an existing image, you cannot move the camera.

(If you did find a compensating transform, it would be equivalent to un-projecting and re-projecting with a new perspective projection, so you'd have your original problem over again.)

> Why does Zooming in does not work with Projection Matrix but works in Real Life?

If you try zooming in and moving back simultaneously in real life, you will find you’ve created the [dolly zoom](https://en.wikipedia.org/wiki/Dolly_zoom).

> **Comments:**
> **username_0**: That explanation is quite helpful, thank you very much. Also, the dolly zoom is proving exactly what I mathematically computed, so thanks for sharing that.  My solution was to only take the boxes that have at least one face fully on the correct side of the image plane, and then shorten the edges of the ones on the wrong side such that all of them are on the correct side.  
> **username_2**: @username_0 That will work only as long as the camera's axis is perpendicular to a face of the box.  
> **username_0**: It does not necessarily have to be perpendicular. If at least one face is fully on the correct side, then it means every corner of the box has a connected corner that is on the correct side. Then it only becomes moving the points on the behind side by moving it along the edge that connect it to its front neighbor.  Of course in case it is not perpendicular, the box will become an irregular hexahedron, but I am fine with it.  

---

