<ol>
<li><p><em>The best results</em> strongly depend on your use case. They also depend on what effect you want to achieve. Sobel is just an edge detection filter: the edges will depend on the input signal, choosing that input signal is up to you.</p>

<p>Here you are using the color image as an input, and the filter rightfully detects faint edges in the blue gradient, while the edges of the cube get interrupted where its color is too close from the background color.</p>

<p>Since I assume your program is also responsible for drawing the cube, you have access to other information that you can feed to your Sobel filter. For example depth and normals are good candidates for edge detection. The albedo before lighting could be used to. Test with different inputs and decide which ones to use depending on the results you get.</p></li>
<li><p>Regarding your question about how to combine the edge information, I suggest you filter <code>g</code> a little before using it. Then you can use it to interpolate between the original color and the wanted edge color.</p>

<p>For example you could try something like this:</p></li>
</ol>

<pre class="lang-c prettyprint-override"><code>    float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));

    // Try different values and see what happens
    g = smoothstep(0.4, 0.6, g);

    vec3 edgeColor = vec3(1., 0., 0.2);
    color = vec4(mix(diffuse, edgeColor, g), 1.);
</code></pre>

<hr>

<h3>Addendum</h3>

<p>To use depth or normals, you would need to save them in a texture if that's not done already. When you create the frame buffer for your regular rendering pass, you can attach various textures to it (see <a href="https://www.opengl.org/sdk/docs/man/html/glFramebufferTexture.xhtml" rel="noreferrer"><code>glFramebufferTexture2D</code></a>) and write other information to them than just the color of the scene.</p>

<p>If you attach a depth texture (with <code>GL_DEPTH_ATTACHMENT</code>), it will be used automatically for depth. If you attach one or several color textures (with <code>GL_COLOR_ATTACHMENTi</code>), you can write to them by declaring several outputs to your fragment shaders (this used to be done with <code>gl_FragData</code>; either way, see <a href="https://www.opengl.org/sdk/docs/man/html/glDrawBuffers.xhtml" rel="noreferrer"><code>glDrawBuffers</code></a>).</p>

<p>For more information on the topic, look up <em>"Multi Render Target"</em> (MRT).</p>

CC BY-SA 3.0
2016-06-19T14:25:37.927Z
Nice information, thanks Julien. One more question: How could I use the depth or normal information in my fragment shader? I'm still quite new with GLSL and therefore have no idea how to include the values in the Sobel algorithm.
enne87
CC BY-SA 3.0
2016-06-20T00:12:16.130Z
When you create the framebuffer for your rendering pass, you can attach more than one texture to it. If you attach a depth texture, it will be used automatically for depth. It you attach another color texture you can write into it separately (see https://www.opengl.org/sdk/docs/man/html/glDrawBuffers.xhtml), a feature called "Multi Render Target" (MRT).
Julien Guertault
CC BY-SA 3.0
2016-06-21T02:52:35.430Z
@enne87: Happy to help. :)
Julien Guertault
CC BY-SA 3.0
2016-06-19T13:10:38.510Z
2016-06-22T12:27:08.380Z
2016-06-22T12:27:08.380Z
Julien Guertault
<p>With respect to <a href="https://computergraphics.stackexchange.com/questions/2450/opengl-detection-of-edges">this topic </a> I've successfully implemented the <a href="https://en.wikipedia.org/wiki/Sobel_operator" rel="noreferrer">Sobel Edge Detection</a> filter in GLSL. Here is the fragment shader code of the filter:</p>

<pre><code>#version 330 core
in vec2 TexCoords;
out vec4 color;

uniform sampler2D screenTexture;

mat3 sx = mat3( 
    1.0, 2.0, 1.0, 
    0.0, 0.0, 0.0, 
   -1.0, -2.0, -1.0 
);
mat3 sy = mat3( 
    1.0, 0.0, -1.0, 
    2.0, 0.0, -2.0, 
    1.0, 0.0, -1.0 
);

void main()
{
    vec3 diffuse = texture(screenTexture, TexCoords.st).rgb;
    mat3 I;
    for (int i=0; i&lt;3; i++) {
        for (int j=0; j&lt;3; j++) {
            vec3 sample  = texelFetch(screenTexture, ivec2(gl_FragCoord) + ivec2(i-1,j-1), 0 ).rgb;
            I[i][j] = length(sample); 
    }
}

float gx = dot(sx[0], I[0]) + dot(sx[1], I[1]) + dot(sx[2], I[2]); 
float gy = dot(sy[0], I[0]) + dot(sy[1], I[1]) + dot(sy[2], I[2]);

float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));
color = vec4(diffuse - vec3(g), 1.0);
} 
</code></pre>

<p>And here is the result of a cube with Sobel edge detection:</p>

<p><a href="https://i.sstatic.net/PawMD.png" rel="noreferrer"><img src="https://i.sstatic.net/PawMD.png" alt="Cube with Sobel edge detection filter"></a></p>

<p>If you enlarge the picture, you will see that there is a lot of "noise" produced by Sobel: 
There are gray horizontal stripes all over the scene due to the blue/white gradient. Futhermore, the light cones produce an unwanted pattern on the cube. The black edges on the left of the cube also seem to fade because of the light cone on the left half of the cube.</p>

<p>So I read <a href="https://www.objc.io/issues/21-camera-and-photos/gpu-accelerated-machine-vision/" rel="noreferrer">this article</a> which stated that one should grayscale the image first and use a Gaussian blur filter to make the edges more apparent. On the bottom of the article, there is also the canny edge detection filter which seem to produce better results.</p>

<p>Now I have two questions:</p>

<ol>
<li><p>Are the following steps correct to produce the best possible edge detection results:</p>

<ul>
<li>Grayscale</li>
<li>Gaussian Blur</li>
<li>Sobel/Canny edge detection</li>
</ul></li>
<li><p>If yes, how would I merge the original image with the processed image? I mean, after processing the steps stated above, I get an image that is either completely black with white edges or vice verca. How would I put the edges on my original image/texture?</p></li>
</ol>

<p>Thanks for your help!</p>

CC BY-SA 3.0
2016-06-19T21:44:11.327Z
I wonder if you might get the results you want by utilizing OpenGL's [edge flag](https://www.opengl.org/sdk/docs/man2/xhtml/glEdgeFlagPointer.xml) in some way. It looks like only edges between vertices with the edge flag on are drawn when in line mode. There's [a description here](http://www.songho.ca/opengl/gl_vertexarray.html). (I've not used it myself or I'd post an example.)
user1118321
<opengl><edge-detection>
OpenGL GLSL - Sobel Edge Detection Filter
enne87
# OpenGL GLSL - Sobel Edge Detection Filter

(Asked by: username_0 on 2016-06-19)

With respect to [this topic](https://computergraphics.stackexchange.com/questions/2450/opengl-detection-of-edges)  I've successfully implemented the [Sobel Edge Detection](https://en.wikipedia.org/wiki/Sobel_operator) filter in GLSL. Here is the fragment shader code of the filter:

```
#version 330 core
in vec2 TexCoords;
out vec4 color;

uniform sampler2D screenTexture;

mat3 sx = mat3( 
    1.0, 2.0, 1.0, 
    0.0, 0.0, 0.0, 
   -1.0, -2.0, -1.0 
);
mat3 sy = mat3( 
    1.0, 0.0, -1.0, 
    2.0, 0.0, -2.0, 
    1.0, 0.0, -1.0 
);

void main()
{
    vec3 diffuse = texture(screenTexture, TexCoords.st).rgb;
    mat3 I;
    for (int i=0; i<3; i++) {
        for (int j=0; j<3; j++) {
            vec3 sample  = texelFetch(screenTexture, ivec2(gl_FragCoord) + ivec2(i-1,j-1), 0 ).rgb;
            I[i][j] = length(sample); 
    }
}

float gx = dot(sx[0], I[0]) + dot(sx[1], I[1]) + dot(sx[2], I[2]); 
float gy = dot(sy[0], I[0]) + dot(sy[1], I[1]) + dot(sy[2], I[2]);

float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));
color = vec4(diffuse - vec3(g), 1.0);
} 

```

And here is the result of a cube with Sobel edge detection:

[![Cube with Sobel edge detection filter](https://i.sstatic.net/PawMD.png)](https://i.sstatic.net/PawMD.png)

If you enlarge the picture, you will see that there is a lot of "noise" produced by Sobel:
There are gray horizontal stripes all over the scene due to the blue/white gradient. Futhermore, the light cones produce an unwanted pattern on the cube. The black edges on the left of the cube also seem to fade because of the light cone on the left half of the cube.

So I read [this article](https://www.objc.io/issues/21-camera-and-photos/gpu-accelerated-machine-vision/) which stated that one should grayscale the image first and use a Gaussian blur filter to make the edges more apparent. On the bottom of the article, there is also the canny edge detection filter which seem to produce better results.

Now I have two questions:

1. Are the following steps correct to produce the best possible edge detection results:

   * Grayscale
   * Gaussian Blur
   * Sobel/Canny edge detection
2. If yes, how would I merge the original image with the processed image? I mean, after processing the steps stated above, I get an image that is either completely black with white edges or vice verca. How would I put the edges on my original image/texture?

Thanks for your help!

**Tags:** opengl, edge-detection

> **Comments:**
> **username_1**: I wonder if you might get the results you want by utilizing OpenGL's [edge flag](https://www.opengl.org/sdk/docs/man2/xhtml/glEdgeFlagPointer.xml) in some way. It looks like only edges between vertices with the edge flag on are drawn when in line mode. There's [a description here](http://www.songho.ca/opengl/gl_vertexarray.html). (I've not used it myself or I'd post an example.)  

---

## 1 answers

### Answer by username_2 on 2016-06-19. Score: 11 | Accepted answer

1. *The best results* strongly depend on your use case. They also depend on what effect you want to achieve. Sobel is just an edge detection filter: the edges will depend on the input signal, choosing that input signal is up to you.

   Here you are using the color image as an input, and the filter rightfully detects faint edges in the blue gradient, while the edges of the cube get interrupted where its color is too close from the background color.

   Since I assume your program is also responsible for drawing the cube, you have access to other information that you can feed to your Sobel filter. For example depth and normals are good candidates for edge detection. The albedo before lighting could be used to. Test with different inputs and decide which ones to use depending on the results you get.
2. Regarding your question about how to combine the edge information, I suggest you filter `g` a little before using it. Then you can use it to interpolate between the original color and the wanted edge color.

   For example you could try something like this:

```c
    float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));

    // Try different values and see what happens
    g = smoothstep(0.4, 0.6, g);

    vec3 edgeColor = vec3(1., 0., 0.2);
    color = vec4(mix(diffuse, edgeColor, g), 1.);

```

---

### Addendum

To use depth or normals, you would need to save them in a texture if that's not done already. When you create the frame buffer for your regular rendering pass, you can attach various textures to it (see [`glFramebufferTexture2D`](https://www.opengl.org/sdk/docs/man/html/glFramebufferTexture.xhtml)) and write other information to them than just the color of the scene.

If you attach a depth texture (with `GL_DEPTH_ATTACHMENT`), it will be used automatically for depth. If you attach one or several color textures (with `GL_COLOR_ATTACHMENTi`), you can write to them by declaring several outputs to your fragment shaders (this used to be done with `gl_FragData`; either way, see [`glDrawBuffers`](https://www.opengl.org/sdk/docs/man/html/glDrawBuffers.xhtml)).

For more information on the topic, look up *"Multi Render Target"* (MRT).

> **Comments:**
> **username_0**: Nice information, thanks username_2. One more question: How could I use the depth or normal information in my fragment shader? I'm still quite new with GLSL and therefore have no idea how to include the values in the Sobel algorithm.  
> **username_2**: When you create the framebuffer for your rendering pass, you can attach more than one texture to it. If you attach a depth texture, it will be used automatically for depth. It you attach another color texture you can write into it separately (see https://www.opengl.org/sdk/docs/man/html/glDrawBuffers.xhtml), a feature called "Multi Render Target" (MRT).  
> **username_2**: @username_0: Happy to help. :)  

---

