<p>I'm trying to use <a href="https://github.com/NVlabs/instant-ngp/tree/master" rel="nofollow noreferrer">Instant NGP</a> on a synthetic image dataset. I have image sequences of a moving object with the following information:</p>
<ul>
<li>I have the object's xyz and the camera's xyz positions in a void.</li>
<li>The object can rotate, and I have it's yaw, pitch, and roll values.</li>
<li>The object is always centered in the images.</li>
</ul>
<p>To use Instant NGP, I need to provide <a href="https://github.com/NVlabs/instant-ngp/issues/61" rel="nofollow noreferrer">camera-to-world transforms</a> for each camera, with the camera rotated to look at the object which is centered around [0,0,0] and is in a yaw = pitch = roll = 0 state. I'm a 2D vision person and I can't quite wrap my head around this, and my latest implmentation (based on <a href="https://github.com/wingedrasengan927/Image-formation-and-camera-calibration/tree/master" rel="nofollow noreferrer">this</a>) is close but not right.</p>
<p>My current approach is:</p>
<ul>
<li>Centre the object at [0,0,0]</li>
<li>Move the camera to the correct relative position (camera - object)</li>
<li>Rotate the camera's position about the origin to account for the rotation of the object (roll then yaw)</li>
<li>Rotate the camera's look angle to be pointing at the origin (roll, then yaw, then pitch)</li>
<li>Rotate a constant pi/2 roll offset to point the image up.</li>
</ul>
<p>But there's a lot of signs, angles, and axes to keep track of and I'm getting something wrong. It is entirely possible my definitions used above are wrong due to inconsistent axes labelling.</p>
<p>Any advice on how to do this, or the correct terminology for me to search for, would be appreciated. I'm working in python if there are any helper libraries I should be using.</p>
<p>EDIT TO ADD MORE DETAIL (approach above exapnded):
I tried to get it working on 4 known points (a close pass over the object (in front, directly above, behind), and one from far above.) These are pointing in roughly the right direction but not exactly centered, but I'm happy with the relative positions between object and camera.</p>
<p><a href="https://i.sstatic.net/9Nhla.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9Nhla.jpg" alt="Cameras used to develop code against, yellow box is an inset view" /></a></p>
<p>This is an example sequence of views. Again the relative positions seem correct given the object's orientation, but the cameras are mostly looking in the wrong directions:</p>
<p><a href="https://i.sstatic.net/VLw5V.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VLw5V.jpg" alt="Example image sequence" /></a>
<a href="https://i.sstatic.net/Yvg2V.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Yvg2V.jpg" alt="Cameras pointing in the wrong direction" /></a></p>

CC BY-SA 4.0
2023-08-18T14:55:22.253Z
It sounds like the camera is in a fixed position and the object is also in a fixed position, is this true? If this is true you can use a fairly standard "LookAt" function to generate the view matrix.
pmw1234
CC BY-SA 4.0
2023-08-18T19:17:53.930Z
Welcome to computer graphics stack exchange =). You said, you are close to the solution. Please discribe how your solution works (which rotation order...) and what is different to the correct solution? Maybe some pictures will help to understand where your code fail and how to fix it. Which of your 4 points work? And which don't?
Thomas
CC BY-SA 4.0
2023-08-19T16:11:57.657Z
I remember instant-npg / nerfstudio / original nerf and opencv has pretty different coordinate frame settings (like which is the 'upward-pointing' axis and which is the forward one). I was once bothered by this, and I think the coordinate frame conversion of different models is going to help: https://github.com/NVlabs/instant-ngp/issues/658
Enigmatisms
CC BY-SA 4.0
2023-08-21T09:20:00.980Z
Thanks all - post edited to add more detail and example images.

pmw1234 - each "camera" is static, but there are images taken from around the scene (see images). A LookAt function sounds like exactly what I need, I'll search and try.

Thomas - added some detail, although my axis labelling is likely wrong/inconsistent which is part of my problem!

Enigmatisms - thanks, this looks very useful. Hopefully combining this and a LookAt function I can find a solution.
Thom
CC BY-SA 4.0
2023-08-21T11:00:37.097Z
Update: [this post](https://stackoverflow.com/questions/56782580/what-is-the-lookat-matrix-multiplication-function-in-python-for-a-dragon-superbi) was extremely useful, I think I've got a working solution now. Will confirm and update.
Thom
<3d><camera-matrix>
Rotating a set of cameras to look at a rotated object
Thom
# Rotating a set of cameras to look at a rotated object

(Asked by: username_0 on 2023-08-18)

I'm trying to use [Instant NGP](https://github.com/NVlabs/instant-ngp/tree/master) on a synthetic image dataset. I have image sequences of a moving object with the following information:

* I have the object's xyz and the camera's xyz positions in a void.
* The object can rotate, and I have it's yaw, pitch, and roll values.
* The object is always centered in the images.

To use Instant NGP, I need to provide [camera-to-world transforms](https://github.com/NVlabs/instant-ngp/issues/61) for each camera, with the camera rotated to look at the object which is centered around [0,0,0] and is in a yaw = pitch = roll = 0 state. I'm a 2D vision person and I can't quite wrap my head around this, and my latest implmentation (based on [this](https://github.com/wingedrasengan927/Image-formation-and-camera-calibration/tree/master)) is close but not right.

My current approach is:

* Centre the object at [0,0,0]
* Move the camera to the correct relative position (camera - object)
* Rotate the camera's position about the origin to account for the rotation of the object (roll then yaw)
* Rotate the camera's look angle to be pointing at the origin (roll, then yaw, then pitch)
* Rotate a constant pi/2 roll offset to point the image up.

But there's a lot of signs, angles, and axes to keep track of and I'm getting something wrong. It is entirely possible my definitions used above are wrong due to inconsistent axes labelling.

Any advice on how to do this, or the correct terminology for me to search for, would be appreciated. I'm working in python if there are any helper libraries I should be using.

EDIT TO ADD MORE DETAIL (approach above exapnded):
I tried to get it working on 4 known points (a close pass over the object (in front, directly above, behind), and one from far above.) These are pointing in roughly the right direction but not exactly centered, but I'm happy with the relative positions between object and camera.

[![Cameras used to develop code against, yellow box is an inset view](https://i.sstatic.net/9Nhla.jpg)](https://i.sstatic.net/9Nhla.jpg)

This is an example sequence of views. Again the relative positions seem correct given the object's orientation, but the cameras are mostly looking in the wrong directions:

[![Example image sequence](https://i.sstatic.net/VLw5V.jpg)](https://i.sstatic.net/VLw5V.jpg)
[![Cameras pointing in the wrong direction](https://i.sstatic.net/Yvg2V.jpg)](https://i.sstatic.net/Yvg2V.jpg)

**Tags:** 3d, camera-matrix

> **Comments:**
> **username_1**: It sounds like the camera is in a fixed position and the object is also in a fixed position, is this true? If this is true you can use a fairly standard "LookAt" function to generate the view matrix.  
> **username_2**: Welcome to computer graphics stack exchange =). You said, you are close to the solution. Please discribe how your solution works (which rotation order...) and what is different to the correct solution? Maybe some pictures will help to understand where your code fail and how to fix it. Which of your 4 points work? And which don't?  
> **username_3**: I remember instant-npg / nerfstudio / original nerf and opencv has pretty different coordinate frame settings (like which is the 'upward-pointing' axis and which is the forward one). I was once bothered by this, and I think the coordinate frame conversion of different models is going to help: https://github.com/NVlabs/instant-ngp/issues/658  
> **username_0**: Thanks all - post edited to add more detail and example images.  username_1 - each "camera" is static, but there are images taken from around the scene (see images). A LookAt function sounds like exactly what I need, I'll search and try.  username_2 - added some detail, although my axis labelling is likely wrong/inconsistent which is part of my problem!  username_3 - thanks, this looks very useful. Hopefully combining this and a LookAt function I can find a solution.  
> **username_0**: Update: [this post](https://stackoverflow.com/questions/56782580/what-is-the-lookat-matrix-multiplication-function-in-python-for-a-dragon-superbi) was extremely useful, I think I've got a working solution now. Will confirm and update.  

---

## 0 answers

