<p>If the iPhone had LiDAR sensors, it would be possible to generate a 3D point cloud that you could theoretically use to furnish a 3D AR reconstruction of your surroundings. E.g. you could create an AR video chat whereby you see a "hologram" projection of the person you are speaking within your 3D environment. </p>

<blockquote>
  <p>However, given that LiDAR has yet to be released on iPhone, do you know of an alternative transformation algorithm that may be employed on a single video stream for consistent video depth estimation? </p>
</blockquote>

<p>I came across <a href="https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/" rel="nofollow noreferrer">this SIGGRAPH 2020 video depth estimation project</a> which seems to do this pretty reliably. I think this question ultimately falls with the class of SLAM problems. The SIGGRAPH paper seems to be very promising, but ideally, this would be done in realtime (or with a small time lag), which doesn't seem plausible; this is a <a href="https://www.youtube.com/watch?v=5Tia2oblJAg&amp;fbclid=IwAR3xaG82vvatIeTQ9wVn0O-R_sbZoqN_e6qGwmUdgHleNPsY5ZGyHOkuBzk" rel="nofollow noreferrer">demo video</a> in case you're curious.</p>

<p>Alternatively, I thought it might be possible using two cameras simultaneously, e.g. the wide and normal, to correct for the camera FOV and optic distortion in each and then use the 3D stereoscopic transform in OpenCV knowing the optical properties of the setup (distance between the cameras, focal length, etc.) to generate a 3D image of the scene with almost as much precision as LiDAR. This <a href="https://becominghuman.ai/stereo-3d-reconstruction-with-opencv-using-an-iphone-camera-part-i-c013907d1ab5" rel="nofollow noreferrer">Medium article</a> on Stereo 3D reconstruction seems to suggest that such a configuration is possible, although I suppose the small baseline distance between the cameras relative to the object might affect the outcome. I think <a href="https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/capturing_photos_with_depth" rel="nofollow noreferrer">depth map</a> in Apple's AVFoundation is certainly an alternative; however, the TrueDepth camera system was only introduced with the iPhoneX.</p>

<p>Thanks in advance! I would appreciate it if this could be demonstrated by example :) I cross-posted <a href="https://computergraphics.stackexchange.com/questions/9867/can-you-use-opencv-stereo-3d-reconstruction-with-iphone-cameras-to-approximate-l">this question</a> on the Computer Graphics exchange.</p>

CC BY-SA 4.0
2020-05-20T17:30:45.233Z
True, the tiny distance between the cameras will strongly reduce 3D reconstruction precision. Once a feature point appears on the very same location in both pictures there is no way to do proper stereo reconstruction for it. I think the critical distance will be quite small.
Isolin
CC BY-SA 4.0
2020-05-20T17:53:36.220Z
@Isolin Are you aware of an alternative, pre-TrueDepth, i.e. for models prior to iPhoneX?
Sergio Charles
CC BY-SA 4.0
2020-05-20T20:32:41.683Z
No, I have "traditional" photogrammetry experience, but honestly I was never interested in Apple products. If you assume your scene to be static, you can fallback to a progressive approach. At least for close objects the stereo could help a bit. But if stereo would be the ultimate solution to go, no one would ever think of integrating other technologies like structured light or LiDAR :)
Isolin
CC BY-SA 4.0
2020-05-20T21:58:44.030Z
@Isolin Well said! Yes, I suppose a Stereo solution would be limited to close objects.
Sergio Charles
CC BY-SA 4.0
2020-06-01T17:28:01.423Z
Estimating depth from monocular images and videos is a quite hot topic. I id not keep an eye on it for a while but if you are looking for similar projects then just https://www.google.com/search?q=depth+from+monocular or https://www.google.com/search?q=depth+from+monocular+video. There are many with code and benchmarks available.
datjko
<3d><transformations><geometry><projections><stereo-rendering>
Consistent Video Depth Estimation from Monocular Lens to approximate LiDAR?
Sergio Charles
# Consistent Video Depth Estimation from Monocular Lens to approximate LiDAR?

(Asked by: username_0 on 2020-05-20)

If the iPhone had LiDAR sensors, it would be possible to generate a 3D point cloud that you could theoretically use to furnish a 3D AR reconstruction of your surroundings. E.g. you could create an AR video chat whereby you see a "hologram" projection of the person you are speaking within your 3D environment.

> However, given that LiDAR has yet to be released on iPhone, do you know of an alternative transformation algorithm that may be employed on a single video stream for consistent video depth estimation?

I came across [this SIGGRAPH 2020 video depth estimation project](https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/) which seems to do this pretty reliably. I think this question ultimately falls with the class of SLAM problems. The SIGGRAPH paper seems to be very promising, but ideally, this would be done in realtime (or with a small time lag), which doesn't seem plausible; this is a [demo video](https://www.youtube.com/watch?v=5Tia2oblJAg&fbclid=IwAR3xaG82vvatIeTQ9wVn0O-R_sbZoqN_e6qGwmUdgHleNPsY5ZGyHOkuBzk) in case you're curious.

Alternatively, I thought it might be possible using two cameras simultaneously, e.g. the wide and normal, to correct for the camera FOV and optic distortion in each and then use the 3D stereoscopic transform in OpenCV knowing the optical properties of the setup (distance between the cameras, focal length, etc.) to generate a 3D image of the scene with almost as much precision as LiDAR. This [Medium article](https://becominghuman.ai/stereo-3d-reconstruction-with-opencv-using-an-iphone-camera-part-i-c013907d1ab5) on Stereo 3D reconstruction seems to suggest that such a configuration is possible, although I suppose the small baseline distance between the cameras relative to the object might affect the outcome. I think [depth map](https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/capturing_photos_with_depth) in Apple's AVFoundation is certainly an alternative; however, the TrueDepth camera system was only introduced with the iPhoneX.

Thanks in advance! I would appreciate it if this could be demonstrated by example :) I cross-posted [this question](https://computergraphics.stackexchange.com/questions/9867/can-you-use-opencv-stereo-3d-reconstruction-with-iphone-cameras-to-approximate-l) on the Computer Graphics exchange.

**Tags:** 3d, transformations, geometry, projections, stereo-rendering

> **Comments:**
> **username_1**: True, the tiny distance between the cameras will strongly reduce 3D reconstruction precision. Once a feature point appears on the very same location in both pictures there is no way to do proper stereo reconstruction for it. I think the critical distance will be quite small.  
> **username_0**: @username_1 Are you aware of an alternative, pre-TrueDepth, i.e. for models prior to iPhoneX?  
> **username_1**: No, I have "traditional" photogrammetry experience, but honestly I was never interested in Apple products. If you assume your scene to be static, you can fallback to a progressive approach. At least for close objects the stereo could help a bit. But if stereo would be the ultimate solution to go, no one would ever think of integrating other technologies like structured light or LiDAR :)  
> **username_0**: @username_1 Well said! Yes, I suppose a Stereo solution would be limited to close objects.  
> **username_2**: Estimating depth from monocular images and videos is a quite hot topic. I id not keep an eye on it for a while but if you are looking for similar projects then just https://www.google.com/search?q=depth+from+monocular or https://www.google.com/search?q=depth+from+monocular+video. There are many with code and benchmarks available.  

---

## 0 answers

