<p>You can calculate the world position of the pixel on near plane quite easily by first defining Normalized Device Coordinates (NDC) for the point and then transforming the NDC back to the world space. You can calculate NDC for your point as follows:
$$v=[2*(x+0.5)/width-1, 1-2*(y+0.5)/height, 0, 1]$$
I'm using 0 here for the z-component assuming near-plane in NDC is 0, but it could be something else as well, so check with your projection matrix what values comes out at the near-plane. This vector must be then transformed to clip-space by multiplying by near-plane distance:
$$v'=np*v$$
Now that you have clip-space coordinates, you just need to transform this back to world space with the standard clip$\rightarrow$world transformation matrix:
$$v''=M*v'$$</p>

CC BY-SA 3.0
2017-01-11T15:00:42.003Z
Thank you, but a little confused.
"check with your projection matrix what values comes out at the near-plane" I'm no quite sure how I would do that...
And in your first equation, isn't the first and second component of v the x and y in "clip space"?
remi000
CC BY-SA 3.0
2017-01-11T15:14:31.830Z
If you transform a point $[0, 0, np]$ from camera space to NDC, what's the value you get for z-coordinate. And to your second question, no, they are in NDC. You have to multiply by w (i.e. near-plane distance in your case) to get them into clip-space
JarkkoL
CC BY-SA 3.0
2017-01-11T14:45:57.720Z
2017-01-11T14:45:57.720Z
JarkkoL
<p>I'm having some trouble trying to figure out how to go from a coordinate on the screen, to 3D world coordinates on a plane.</p>

<p>My case is as seen on the picture further down. World origin is in the center with the grid representing the X-Y plane.<br>
I want to be able to click/touch on the screen, and transform the screen coordinates to the correct 3D world coordinates on the X-Y plane.</p>

<p>I first tried to figure out the maths myself by simple geometry, but got stuck...<br>
I then tried the function <code>glm::unProject()</code> after checking out <a href="https://stackoverflow.com/questions/9901453/using-glms-unproject">this post</a>.</p>

<p>The screen has dimensions <code>mWindowWidth</code> and <code>mWindowHeight</code>.<br>
<code>x</code>, <code>y</code> are the screen coordinates where origin is in the top left corner. Positive x to the right, positive y down.</p>

<p><code>projM</code> is the projection matrix.</p>

<p>Third parameter is the model matrix, which I believe in my case only needs to be an identity matrix.</p>

<p>I get the world coordinates X and Y (Z is already known as it is on the plane) by using these parameters in <code>glm::unProject</code>:</p>

<pre><code>glm::vec3 posVec = glm::unProject(
   glm::vec3(x, float(mWindowHeight) - y, 1.0f), 
   glm::mat4(1.0f), 
   projM, 
   glm::vec4(0.0f, 0.0f, float(mWindowWidth), float(mWindowHeight))
);
</code></pre>

<p>What happens is that if I have screen coordinates near world origin, the world coordinates seems correct. But If I use screen coordinates near the edges and corners of the screen, I get a certain offset (see image below).</p>

<p>Where is this offset coming from, and what am I misunderstanding?</p>

<p><a href="https://i.sstatic.net/Gc5Mu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Gc5Mu.png" alt="enter image description here"></a></p>

<p><strong>EDIT:</strong></p>

<p>As mentioned in the comments, the second parameter should be the "model-view" matrix. I tried to switch <code>glm::mat4(1.0f)</code> with <code>viewM</code> (view matrix). The effect is still the same, if not with even more offset.</p>

<p>I also tried to get the <code>z</code> buffer depth value by using:
(Got it from this <a href="http://nehe.gamedev.net/article/using_gluunproject/16013/" rel="nofollow noreferrer">tutorial</a>)</p>

<pre><code>float z;
glReadPixels(int(x), int(mWindowHeight) - int(y), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &amp;z);
</code></pre>

<p>This results in very small values of X and Y world coordinates, and essentially where ever I touch, the resulting coordinates are near the origin.</p>

CC BY-SA 3.0
2017-01-10T01:45:11.073Z
Does it work better if the plane is facing the camera? I don't think you're giving it the correct Z. I don't know that particular unproject function but I suspect it should be Z from your depth buffer which is certainly not constant with the plane orientation shown.
Olivier
CC BY-SA 3.0
2017-01-10T05:52:19.800Z
I get a similar result with the plane facing the camera.. I'm not sure how to get that z value from the depth buffer. But a value different from 1 just seem to make it worse.
remi000
CC BY-SA 3.0
2017-01-10T10:40:57.370Z
You are not passing the view matrix. In the StackOverflow link you posted, it states that you should be passing the ModelView matrix on the 'model' parameter. That makes sense, since the function cannot guess which view matrix you are using and it needs to know that to reverse all the transformations.
Samu
CC BY-SA 3.0
2017-01-10T10:44:55.610Z
Ok, yes good point. I think I assumed since the function takes in the window dimensions it had the parameters needed for the "view". But this makes more sence, I didn't understand why the model matrix was a parameter. Will try this later, thanks.
remi000
CC BY-SA 3.0
2017-01-10T19:48:13.203Z
I tried with the view matrix. See edits.
remi000
CC BY-SA 3.0
2017-01-11T12:23:32.477Z
Does the plane really have a model matrix equal to identity (i.e. no scales, etc)? Also, glm::unproject returns a vector in object coordinates (model space). How are you using that returned vector when determining where it is in the world?
Samu
CC BY-SA 3.0
2017-01-11T12:27:00.147Z
The plane is just to visualize the xy world plane (made up of lines).
I take the x and y value from the returned vector and create a translation matrix with 0.0 as z translation. No scale and rotation.
That gives me the model matrix which is sent to the shader with ViewM and ProjM.
remi000
<raytracing><projections><glm>
Screen to World coordinates glm::unProject
remi000
# Screen to World coordinates glm::unProject

(Asked by: username_0 on 2017-01-09)

I'm having some trouble trying to figure out how to go from a coordinate on the screen, to 3D world coordinates on a plane.

My case is as seen on the picture further down. World origin is in the center with the grid representing the X-Y plane.  
I want to be able to click/touch on the screen, and transform the screen coordinates to the correct 3D world coordinates on the X-Y plane.

I first tried to figure out the maths myself by simple geometry, but got stuck...  
I then tried the function `glm::unProject()` after checking out [this post](https://stackoverflow.com/questions/9901453/using-glms-unproject).

The screen has dimensions `mWindowWidth` and `mWindowHeight`.  
`x`, `y` are the screen coordinates where origin is in the top left corner. Positive x to the right, positive y down.

`projM` is the projection matrix.

Third parameter is the model matrix, which I believe in my case only needs to be an identity matrix.

I get the world coordinates X and Y (Z is already known as it is on the plane) by using these parameters in `glm::unProject`:

```
glm::vec3 posVec = glm::unProject(
   glm::vec3(x, float(mWindowHeight) - y, 1.0f), 
   glm::mat4(1.0f), 
   projM, 
   glm::vec4(0.0f, 0.0f, float(mWindowWidth), float(mWindowHeight))
);

```

What happens is that if I have screen coordinates near world origin, the world coordinates seems correct. But If I use screen coordinates near the edges and corners of the screen, I get a certain offset (see image below).

Where is this offset coming from, and what am I misunderstanding?

[![enter image description here](https://i.sstatic.net/Gc5Mu.png)](https://i.sstatic.net/Gc5Mu.png)

**EDIT:**

As mentioned in the comments, the second parameter should be the "model-view" matrix. I tried to switch `glm::mat4(1.0f)` with `viewM` (view matrix). The effect is still the same, if not with even more offset.

I also tried to get the `z` buffer depth value by using:
(Got it from this [tutorial](http://nehe.gamedev.net/article/using_gluunproject/16013/))

```
float z;
glReadPixels(int(x), int(mWindowHeight) - int(y), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &z);

```

This results in very small values of X and Y world coordinates, and essentially where ever I touch, the resulting coordinates are near the origin.

**Tags:** raytracing, projections, glm

> **Comments:**
> **username_1**: Does it work better if the plane is facing the camera? I don't think you're giving it the correct Z. I don't know that particular unproject function but I suspect it should be Z from your depth buffer which is certainly not constant with the plane orientation shown.  
> **username_0**: I get a similar result with the plane facing the camera.. I'm not sure how to get that z value from the depth buffer. But a value different from 1 just seem to make it worse.  
> **username_2**: You are not passing the view matrix. In the StackOverflow link you posted, it states that you should be passing the ModelView matrix on the 'model' parameter. That makes sense, since the function cannot guess which view matrix you are using and it needs to know that to reverse all the transformations.  
> **username_0**: Ok, yes good point. I think I assumed since the function takes in the window dimensions it had the parameters needed for the "view". But this makes more sence, I didn't understand why the model matrix was a parameter. Will try this later, thanks.  
> **username_0**: I tried with the view matrix. See edits.  
> **username_2**: Does the plane really have a model matrix equal to identity (i.e. no scales, etc)? Also, glm::unproject returns a vector in object coordinates (model space). How are you using that returned vector when determining where it is in the world?  
> **username_0**: The plane is just to visualize the xy world plane (made up of lines). I take the x and y value from the returned vector and create a translation matrix with 0.0 as z translation. No scale and rotation. That gives me the model matrix which is sent to the shader with ViewM and ProjM.  

---

## 1 answers

### Answer by username_3 on 2017-01-11. Score: 1

You can calculate the world position of the pixel on near plane quite easily by first defining Normalized Device Coordinates (NDC) for the point and then transforming the NDC back to the world space. You can calculate NDC for your point as follows:
$$v=[2\*(x+0.5)/width-1, 1-2\*(y+0.5)/height, 0, 1]$$
I'm using 0 here for the z-component assuming near-plane in NDC is 0, but it could be something else as well, so check with your projection matrix what values comes out at the near-plane. This vector must be then transformed to clip-space by multiplying by near-plane distance:
$$v'=np\*v$$
Now that you have clip-space coordinates, you just need to transform this back to world space with the standard clip$\rightarrow$world transformation matrix:
$$v''=M\*v'$$

> **Comments:**
> **username_0**: Thank you, but a little confused. "check with your projection matrix what values comes out at the near-plane" I'm no quite sure how I would do that... And in your first equation, isn't the first and second component of v the x and y in "clip space"?  
> **username_3**: If you transform a point $[0, 0, np]$ from camera space to NDC, what's the value you get for z-coordinate. And to your second question, no, they are in NDC. You have to multiply by w (i.e. near-plane distance in your case) to get them into clip-space  

---

