<p>I can't give you a &quot;better&quot; approach here, but I think that the approach to</p>
<blockquote>
<p>project each individual brush image onto the surface of the object</p>
</blockquote>
<p>is not really a performance issue if you implement that using the GPU. Especially in an editor environment, where not much is going on in comparison to a computer game. All you basically want to do is to render a single image per frame onto a limited set of textures. I can give you a draft of how I would approach the subject if I needed this feature. However, since I haven't implemented it myself yet, there might be some complications and problems I haven't thought of so let me know in the comments if something doesn't work as expected.</p>
<ol>
<li><p>Find the affected surfaces</p>
<ul>
<li>Use the corners of your brush texture and generate a reduced brush projection frustum</li>
<li>Check all objects against it and find the closest surfaces -&gt; If you already use some kind of culling for your scene, reuse the already culled object list, since the brush can only &quot;see&quot; what the camera sees</li>
<li>Identify the subsurfaces/triangles affected by the brush</li>
</ul>
</li>
<li><p>Gather all affected textures. You should know for each surface that is affected, which texture it possesses. Optimally, your whole object has just a single UV texture.</p>
</li>
<li><p>Create a shader program to render to your texture.</p>
<ul>
<li>be careful, the vertex shader has to calculate the normalized UV coordinates and pass them down the pipeline as the vertex position, since they are the position in your texture. Check what additional information you need to pass down the pipeline.</li>
<li>In the fragment shader, overwrite each pixel that is under the brush. You might need to use blending here for non-rectangular brushes.</li>
<li>The brush position is a little bit tricky. You need it in the textures UV space in the fragment shader and I am not sure if you can calculate it in the vertex shader. So you might need to do that in the fragment shader or alternatively in a geometry shader. The problem I see is that the angle in which the brush projection hits the surface can't be calculated in the vertex shader and it is necessary to determine the covered area. The geometry shader has enough information to calculate the angle and the covered area. In the fragment shader, you can't calculate the angle either, but there you can simply check if and where it is located in the brush projection frustum.</li>
</ul>
</li>
<li><p>Now simply render to your textures.</p>
<ul>
<li>Bind your brush texture</li>
<li>For each affected texture
<ul>
<li>Bind the texture as render target</li>
<li>Render all affected objects and let the GPU sort out which pixels need to be overwritten.</li>
</ul>
</li>
</ul>
</li>
<li><p>(Optional) If you want to save your modifications, you need to read back the textures from the GPU and save it.</p>
</li>
</ol>
<p>Since the main work is done by the GPU, it should be quite fast. However, I can see some problems with the described approach when the brush hits multiple objects/textures that are partially overlapping. Sorting out which pixel of the brush affects which texture/object can get messy. There might be some stencil/depth buffer tricks one can apply, but I think this is a topic for another question.</p>

CC BY-SA 4.0
2020-09-19T15:09:11.783Z
Thanks! It's nice to know that this texture projection approach is viable for texture painting!
Lenny White
CC BY-SA 4.0
2020-09-18T08:18:03.573Z
2020-09-18T08:18:03.573Z
wychmaster
<p>As long as no pixel on the texture is used twice, you can render the geometry and display the uv-coordinates (texture coordinates). Usually they are a combination of red and green (2d).
When using a brush, you only need to read out the uv-coordinates you've hit and color the texture at position (uv) in the desired color.</p>
<p><strong>Hint:</strong>
paint onto a second texture, so you'll not destroy the original texture color.</p>
<p><strong>Drawbacks:</strong>
Sometimes the texture resolution of different triangles within a geometry vary. So at some faces it will look nice, and on others it will be pixelated.</p>
<p>As said before, if the mesh (geometry) uses the same part of a texture twice (two faces aiming at the same texture coordinates), and you draw on one of them, the other will be painted as well.</p>

CC BY-SA 4.0
2020-09-18T08:30:15.993Z
to avaid the problem of two faces having the same texture-coordinates, you can re-uv map the geometry while loading the mesh and store both (the original and the new calculated uv-coordinates) into the vertex buffer object (VBO). While rendering the model, use the original uv-coordinates to render the original color and use the new uv-coordinates to add the changes.
Thomas
CC BY-SA 4.0
2020-09-18T08:32:28.823Z
2nd hint: the texture resolution can be higher than the original texture, because the uv-coordinates are between 0 and 1 in both dimensions
Thomas
CC BY-SA 4.0
2020-09-19T15:17:01.030Z
Thanks! Could you clarify what you mean by "uv coordinates you've hit"? It's way too vague. Are you talking about uv coordinates I get after generating brush projection frustum?
Lenny White
CC BY-SA 4.0
2020-09-18T08:13:41.680Z
2020-09-18T08:13:41.680Z
Thomas
<p>You could do it in different ways.</p>
<p>But the most logical way for me is that there are points sampled along the path, then splines/benzier curves are used to make it smooth (additonal points can be interpolated), thereafter it is projected and saved on the texture. So no &quot;every&quot; point has to be sampled.</p>

CC BY-SA 4.0
2020-09-17T00:19:37.630Z
2020-09-17T00:19:37.630Z
the1bird
<p>I have searched this topic for very long time and have found basically no information on it.</p>
<p>If we have a 3d model with a uv layout. What is the basic technique for 3d painting on said model with a brush? A brush stroke consists of the brush pixel data/image laid out in succession. So do we project each individual brush image onto the surface of the object? (Although in actuality to do this we would be projecting the object onto the image space instead). I have managed to project textures onto objects before. And projecting each individual brush image definitely seems feasible, but I'm not sure if it's practical from performance standpoint and whether there're better techniques.</p>
<p><a href="https://i.sstatic.net/sSfgM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sSfgM.png" alt="enter image description here" /></a></p>

<opengl><texture><image-processing><uv-mapping>
How to paint onto 3d models
Lenny White
# How to paint onto 3d models

(Asked by: username_0 on 2020-09-16)

I have searched this topic for very long time and have found basically no information on it.

If we have a 3d model with a uv layout. What is the basic technique for 3d painting on said model with a brush? A brush stroke consists of the brush pixel data/image laid out in succession. So do we project each individual brush image onto the surface of the object? (Although in actuality to do this we would be projecting the object onto the image space instead). I have managed to project textures onto objects before. And projecting each individual brush image definitely seems feasible, but I'm not sure if it's practical from performance standpoint and whether there're better techniques.

[![enter image description here](https://i.sstatic.net/sSfgM.png)](https://i.sstatic.net/sSfgM.png)

**Tags:** opengl, texture, image-processing, uv-mapping


---

## 3 answers

### Answer by username_1 on 2020-09-18. Score: 1 | Accepted answer

I can't give you a "better" approach here, but I think that the approach to

> project each individual brush image onto the surface of the object

is not really a performance issue if you implement that using the GPU. Especially in an editor environment, where not much is going on in comparison to a computer game. All you basically want to do is to render a single image per frame onto a limited set of textures. I can give you a draft of how I would approach the subject if I needed this feature. However, since I haven't implemented it myself yet, there might be some complications and problems I haven't thought of so let me know in the comments if something doesn't work as expected.

1. Find the affected surfaces

   * Use the corners of your brush texture and generate a reduced brush projection frustum
   * Check all objects against it and find the closest surfaces -> If you already use some kind of culling for your scene, reuse the already culled object list, since the brush can only "see" what the camera sees
   * Identify the subsurfaces/triangles affected by the brush
2. Gather all affected textures. You should know for each surface that is affected, which texture it possesses. Optimally, your whole object has just a single UV texture.
3. Create a shader program to render to your texture.

   * be careful, the vertex shader has to calculate the normalized UV coordinates and pass them down the pipeline as the vertex position, since they are the position in your texture. Check what additional information you need to pass down the pipeline.
   * In the fragment shader, overwrite each pixel that is under the brush. You might need to use blending here for non-rectangular brushes.
   * The brush position is a little bit tricky. You need it in the textures UV space in the fragment shader and I am not sure if you can calculate it in the vertex shader. So you might need to do that in the fragment shader or alternatively in a geometry shader. The problem I see is that the angle in which the brush projection hits the surface can't be calculated in the vertex shader and it is necessary to determine the covered area. The geometry shader has enough information to calculate the angle and the covered area. In the fragment shader, you can't calculate the angle either, but there you can simply check if and where it is located in the brush projection frustum.
4. Now simply render to your textures.

   * Bind your brush texture
   * For each affected texture
     + Bind the texture as render target
     + Render all affected objects and let the GPU sort out which pixels need to be overwritten.
5. (Optional) If you want to save your modifications, you need to read back the textures from the GPU and save it.

Since the main work is done by the GPU, it should be quite fast. However, I can see some problems with the described approach when the brush hits multiple objects/textures that are partially overlapping. Sorting out which pixel of the brush affects which texture/object can get messy. There might be some stencil/depth buffer tricks one can apply, but I think this is a topic for another question.

> **Comments:**
> **username_0**: Thanks! It's nice to know that this texture projection approach is viable for texture painting!  

---

### Answer by username_2 on 2020-09-18. Score: 1

As long as no pixel on the texture is used twice, you can render the geometry and display the uv-coordinates (texture coordinates). Usually they are a combination of red and green (2d).
When using a brush, you only need to read out the uv-coordinates you've hit and color the texture at position (uv) in the desired color.

**Hint:**
paint onto a second texture, so you'll not destroy the original texture color.

**Drawbacks:**
Sometimes the texture resolution of different triangles within a geometry vary. So at some faces it will look nice, and on others it will be pixelated.

As said before, if the mesh (geometry) uses the same part of a texture twice (two faces aiming at the same texture coordinates), and you draw on one of them, the other will be painted as well.

> **Comments:**
> **username_2**: to avaid the problem of two faces having the same texture-coordinates, you can re-uv map the geometry while loading the mesh and store both (the original and the new calculated uv-coordinates) into the vertex buffer object (VBO). While rendering the model, use the original uv-coordinates to render the original color and use the new uv-coordinates to add the changes.  
> **username_2**: 2nd hint: the texture resolution can be higher than the original texture, because the uv-coordinates are between 0 and 1 in both dimensions  
> **username_0**: Thanks! Could you clarify what you mean by "uv coordinates you've hit"? It's way too vague. Are you talking about uv coordinates I get after generating brush projection frustum?  

---

### Answer by username_3 on 2020-09-17. Score: 0

You could do it in different ways.

But the most logical way for me is that there are points sampled along the path, then splines/benzier curves are used to make it smooth (additonal points can be interpolated), thereafter it is projected and saved on the texture. So no "every" point has to be sampled.


---

