Dark stripes on surfaces in V-Ray when rendering to texture (baking)

Solved.  The issue was with V-Ray's global ambient occlusion on the GI tab.  Once that was shut off everything renders fine.  Interestingly, V-RayDirt maps in materials seem to be behaving just fine.

UPDATE: It appears the real root problem is deeper.  This turned out to be caused by units being set to fractions instead of decimals.  This can cause some values that are very small to round off to zero as the system tries to make it a fraction.  In v-ray lights, the shadow bias was getting rounded to zero automatically in some cases.  A zero shadow bias can produce strange results in some cases.  SOLUTION: stop using fractional units and go through and fix lights with zero bias.


Typically in a deferred engine, you would render unlit objects directly to the color buffer, bypassing the lighting system entirely. You would likely also output zeroes to the G-buffer so that deferred lights don't add anything to those pixels (and possibly also mark them in the stencil buffer, to avoid running the lighting shaders on them, for efficiency).

Don't worry about switching PSOs for this. I'm guessing you're worried about the performance, but it's not a big deal. Game engines typically switch PSOs (or switch shaders, in DX11/GL) hundreds or thousands of times per frame. Switching a couple of times for an extra color rendering pass isn't even a blip on the radar.

However, if you want to really simplify your code (possibly at the expense of performance), you could just bind the color buffer together with your G-buffers and render to it all the time, as if it were an extra G-buffer. Ordinary deferred shaders would output zero to it, while unlit objects would output their color to it.


I've also spent a good amount of time thinking about a solution for this problem. It seems like one those fundamental mismatches of taking a sparser, lower frequency structure (probe mesh, even if augmented with depth map) and lighting surfaces per pixel which themselves are occluders.

If you look at the SIGGRAPH'17 Precomputed lighting in CoD:IW (http://advances.realtimerendering.com/s2017/index.html) you'll see they partitioned the precomputed lighting into light maps for large surfaces (which have lots of occlusion) and using a probe mesh for small static objects as well as characters. They take a similar approach of storing an occlusion depth map per tetrahedra face, however because they're not lighting the occluders with the probe mesh, the discontinuities introduced are likely not noticeable.

Trying to light the occluders with the same map is extremely tricky because the probe mesh faces may not align well with the occluders. I.e. literally "corner" case where 3 planes meet doesn't encode well into a depth map. I have feeling they still get light leaks, but their density is probably fairly high and their evaluation points are distant enough from the pathologically bad cases.

Sometimes it might be worth asking level artists to not make single sided meshes or thin walls XD, at least for the time being.

Best of luck, if you find a robust solution to this, you should submit a paper to a conference and be graphics industry famous :D


How to combine rotation in 2 axis into one matrix

(This answer is essentially the same as Stefan's but I wanted to add some detail about row and column vectors, and how to determine which you are using.)

Yes, this is possible, but the details depend on whether you represent your vectors as rows or columns.

Column vectors

If you are using column vectors, you will normally transform them by left-multiplying your matrices:

vector = mRotateZ * vector;
vector = mRotateX * vector;

Of course, you can also do this in one step:

vector = mRotateX * mRotateZ * vector;

But matrix multiplication is associative, which means it doesn't matter which multiplication is performed first:

A * B * C = (A * B) * C = A * (B * C)

So we can write

Matrix mRotate = mRotateX * mRotateZ;
vector = mRotate * vector;

We have now created a single matrix, which is equivalent to first rotating about Z and second about X. This generalises trivially for any number of transformations. Notice that transformations are applied from right to left.

Row vectors

If, on the other hand, you are using row vectors, you will normally right-multiply your matrices:

vector = vector * mRotateZ;
vector = vector * mRotateX;

Again, writing it in one step, we get

vector = vector * mRotateZ * mRotateX;

which can be rewritten as

Matrix mRotate = mRotateZ * mRotateX;
vector = vector * mRotate;

Notice that in this case, the transformations applied from left to right.


Does a bumpmap need to be linearized from sRGB when being read?

We can not answer this question. This is important, and you need to get this info from the artist (or vice versa you need to give this info to the artist before he begins) It entirely depends on what kind of assumptions were made when the artist was working with the image.

Was the artist working by numbers
Was the artist working by eye
Was the artist working with a sculpting application
Was the data built from a photograph

You would be right if the artist worked by number. But if the artist was working by eye, which is incredibly hard, then you would be a bit off. sRGB does not entirely match human brightness sense so you should really fiddle a bit with the exponent if this was the case. But you might as well skip this. For sculpting and baking the image is most likely to be linear already to begin with, although the profile might not reflect this. And for the photograph case depends heavily on the method.

Safest bet is to do no conversion at all. But it is not really defined so strictly. Kind of the entire world of image manipulation has skipped how to tag images that arent you know display images but pure data. Who knows how it is encoded.


Which 3D object properties should be affected by distance?

There is only one major thing that I can think of in nonstereographic view that is affected by size and that is:

Atmospheric haze and subsurface scattering, even this is similar at most or all macroscopic ranges.

That said you can easily do this by fudging one constant in a shader. Or slightly more tediously by adding aerosols in the air if you are in fact doing this in real photography, altough getting it exactly right is hard.

This changes in stereoscopy as humans have some sense of their own scale. So the disparity between cameras sets a scale. But if you change the distance between the cameras then you change scale so even they do not have to be in absolute scale just relative scale.

So in fact most things are self similar at all practical scales. And only when you start to see effects that are fixed light wavelength do you get into this problem.


How to build a triangle-voxel list from a triangle mesh?

You could do this. Basically what you would do is generate a cube for each voxel (or actually nearby voxels could share the edges). Not exactly sure what benefit you would get form this, unless your intersection algorithm is somehow extremely finely tuned. It could be useful for real time rendering or some sort of sparse structure. While your at it why not dispense of the voxel concept altogether and go all FEM on the problem.

The thing is you want a more uniform sampling across the volumes density function. Because your integrating over a volume functions you want the step size to be somewhat equally spaced so you can control the integration better. Random or pseudo random solution converges often better, but mainly you do not do cube oriented sampling this way as it will make the structure obvious which is not your goal in most cases.

Image 1: Different sampling strategies. Images in RTL reading order samples along voxel grids,  Uniforms sampling and jittered ray length sampling.

Think of this as solving a ODE you sample the function uniformly and if your value changes too much you might do additional samples. This is probably the reason why you wouldn't really do collision to the voxels.


The GPU hardware only supports nearest-neighbor, bilinear, trilinear, and anisotropic texture filtering. However, nothing stops you from implementing your own texture filtering in the pixel shader.

To do this, you'd use textureGather to grab a 2×2 block of texels around your sample point. Then you'd manually calculate the position of the sample point relative to the texel centers. Armed with the raw texels and the interpolation position, you can then apply whatever interpolation function you like.

The position calculation would be something like this:

vec2 textureSize; // texture's width and height
vec2 offset = fract(texCoord * textureSize + (-0.5 + 1.0/512.0));

The 1/512 offset in there is needed as a rounding correction on certain GPUs, to ensure that fract()'s output steps from 1 to 0 at the exact same point that textureGather switches texels.

As an example, to re-implement bilinear filtering, you'd do this (assuming a single-channel texture for simplicity):

vec4 texels = textureGather(...);
float result = mix(mix(texels.w, texels.z, offset.x),
                   mix(texels.x, texels.y, offset.x),
                   offset.y);

But in practice, you'd be replacing the above with your own interpolation logic. This will for sure be slower than using hardware bilinear interpolation, but it may be an acceptable cost for your app.

If you need more than 2×2 block of texels, you can also use multiple calls to textureGatherOffset to grab more of the surrounding region, 2×2 at a time.

As for generating a higher-resolution version of the texture on the fly, that can certainly also be done. It would be an extension of texture streaming techniques that load larger mip levels from disk only when needed. Here, you'd be procedurally generating the larger mip levels instead of loading them, but it's a similar idea. Unfortunately I don't have a good reference handy for how to do this. (It's somewhat related to virtual texturing, but less complicated—for just plain texture streaming, you don't need sparse textures or a page table, etc.)

However, there is unfortunately no GPU hardware support for doing the thing you suggested of avoiding the extra storage, by processing the texture only a piece at a time. Doing that in hardware (“texel shaders” that would generate texel values on-demand as they are sampled by fragment shaders) is an idea that has been proposed many times but so far has not happened.

It might be possible to do it in software by having fragment shaders append sample requests into a queue, then switching to a different shader to service those requests, then re-running the fragment shaders with the results...but that would take a lot of work and might not be very fast. It would for sure be a research project.

Unless you have a good way of determining in advance which texels are likely to be needed, it would be a lot more practical to just generate the entire texture, assuming memory and performance budgets allow for it.


opengl has several methods that together let you reflect the shader IO.

However it is much simpler if the shader IO is uniform across all your shaders. IOW have a set of default bindings/locations for everything. Then you only need some meta data for each shader where you deviate from those defaults for some reason.

You then only need to query that the binding exists to see whether you need to provide the data. This has the added bonus that if an input gets optimized out you'll not waste time giving data it doesn't need.


Copying this from another thread where i posted this as the answer but as Wyck suggested, the correct answer is the first one.

There is the whole derivation of it but I'll be discussing a brief overview.
This is for the perspective projection  where the line joining the eye and the center of the projection/image plane is perpendicular to it. Like here

As we can easily see, by similar triangles $\triangle ABC$ and $\triangle AEF$ we have

$Y_p / Y = D/-Z$

where $Y_p$ is the projected $Y$ coordinate on to the image plane. $AB = D$ is the total distance from eye to the image plane and is usually set to 1.

Hence we have,

$Y_p = (Y*D)/-Z$

We  can easily ee from here that,

$tan(\theta_{fov}/2) = 1/AB$

$tan(\theta_{fov}/2) = 1/D$

$D = 1 /tan(\theta_{fov}/2)$

Hence we have,
$Y_p = \frac{Y}{-Z*tan(\theta_{fov}/2)}$

For more details check the complete answer here


Passing mat4 array to GLSL succeeding but producing GL_INVALID_OPERATION

The eye position (a vec3) was being bound to the wrong shader, resulting in the glUniform3 error when OpenGL (mesa) checked the validity of the uniforms. The error was due to a typo; I named my simple-mesh and animated-mesh shaders similar things.

glUseProgram(smesh_shader.get());
      smb.predraw();
      glUniform3fv(mesh_shader.uniform("eye"), 1, &eye[0]); // <- Oops
      graphics::ogl::render(scene.getRoot(), smb, mv, smesh_shader);
      smb.postdraw();

Following @Christian Rau's and @wychmaster's suggestions, I was able to debug the issue. I also needed to build a debug version of the mesa library to make sense of the stack-trace in gdb.


Mat4 that is written to a storage buffer in a compute shader has an altered value when read back

It appears that this is indeed a bug within renderdoc. I have encountered the bug I was debugging for in only 1 model, while all the others work fine. In every instance of using the renderdoc debugger, I get the same problem though.


Gamma correction is done in an image or a monitor?

Is every image from the internet commonly got gamma corrected?

Every image is encoded with a color profile. Most color profiles do have a gamma correction, but not every color profile has that (raw for example). Most images on the internet are encoded in sRGB, which is gamma corrected. If you would download a random image from the internet, you will more than likely have an image that is in sRGB and thus gamma corrected.

  Or our monitor is automatically gamma corrected all the image showed up?

The monitor receives its image in the color profile of the monitor (let's just say sRGB), it then decodes the image from sRGB to the actual intensity of the pixels and then displays it. Basically, the monitor receives a gamma corrected image and removes the gamma correction just before it displays it.

Images that are saved to a file or that are sent to the monitor, are generally gamma corrected. This is for a very simple reason. We only have 256 values per RGB component to define the color of a pixel. Our eyes are very good at picking up small differences in dark parts of an image, but relatively terrible at picking up small differences in bright parts.

If we try to save an image without any gamma correcting as an 8-bit image (thus 256 values per RGB component), we end up with having the same amount of precision for dark parts in the image and bright parts in the image. However, our eyes cannot see small differences in bright parts of the image, so all that precision in the bright parts is simply wasted. Gamma correction fixes this. Before we convert the image to 8 bits, we first gamma correct it so that we allocate more precision to that darker parts of the image than the brighter parts of the image. When displayed, the monitor will remove that gamma correction.

Left is an image that was gamma corrected before being saved to 8 bits and right is an image that was not gamma corrected before 8 bits. You can clearly see the lack of precision in the darker parts without gamma correction, but not in the bright parts.

Gamma correction is a very useful trick to make our images with finite bits, look better.

If you want to read more about it, you can do so here.


How do I use texture coordinates of a .obj file in OpenGL?

The OBJ file itself doesn't contain the texture, only the texture co-ordinates (which describe how to map a texture onto the surface of the mesh.) The texture is an image file (such as a PNG) whose path is given by the accompanying .mtl file. You need to parse the .mtl file to get the path to the image file, then use an image-loading library (such as DevIL) to load the image into memory. Once you have the pixels of the image, you can use glTexImage2D to load that image into a texture, which can be accessed from your fragment shader using a sampler2D.


How to count the photons when Photon Mapping

I'm not sure if I understand your question correctly and don't know what you mean by "counting" and how you want to replace this by "color". A simple photon mapper looks like this: 1) photon tracing: emit, scatter, store photons 2) path tracing: gather photons in a given radius and approximate radiance using this density.
To be more precise, in pseudocode (with russian roulette) this would look like:
xi is the position, wo the outgoing direction, W the photon power. All three are vectors of length 3. p is the probability of the Russian roulette. A photon is a struct that stores the power, direction, and position.
xi, wo, W = emitter.sample()
while(true)
   xi = traceRay(xi, wo)
   if(isDiffuse(xi))
      storePhoton(xi, wo, W)

   if(rand < p)
      break
    W /= p;

    wo = bsdf.sample(xi)
    W *= bsdf(wo)/pdf(wo)*cos

You repeatedly call this function for a random emitter until you stored a predefined amount of photons (e.g. 500'000). After this, you build the photonmap, such as a k-d tree to make querying easy. As you can see, the "color" is stored in the power of the photon.
The next step is the gathering the photons, and it's normal path tracing but when you hit a diffuse surface, you gather all the photons in a fixed radius (other kernels can also be used). Here, t is the throughput, wi the incoming direction, the photon_map is the datastructure that we build before and populated with photons, and total_photons is the total amount of photons that we emitted.
...

if(isDiffuse(xi))
   for photon: photon_map.get(xi, radius)
      radiance *= t*bsdf(xi, wi, photon.wo) * photon.power / (pi*radius^2 * total_photons)
   break
...


Controlling "zoom" and "position" of a cubemap in shaders

Reflections based on a cubemap best represent a reflection at infinity. That is, since the only factor that a cubemap provides is direction, the cubemap acts as though it were at a distance from the reflective surface where only the direction to the reflecting objects matters. And that only happens if the reflecting objects are infinitely far from the reflective surface.

That works fine for reflecting distant environmental details like skies or distant landscapes. But it's not going to work with anything even remotely close to the reflective surface. And if the reflecting object and a reflective surface touch, then you can basically forget about it.

Cubemap reflections are simply not the right tool for this scene. You will instead have to employ other strategies for reflections, like rendering the back wall under the reflective surface (using transparency of the reflective surface to create the appearance of reflection), or doing screen-space reflections or something like that.

  I don't have geometry, just a still image and information of where the floor is. So I don't have any depth, normals, or anything like that for most of the image.

If you don't have geometry information or any equivalent to it, then you're pretty much out of luck. Accurate reflection of a scene requires knowing where the scene being reflected is relative to each point on the reflective surface. Even if the surface of reflection itself is flat, the reflected scene isn't. If you don't have some approximation of geometry information, and you cannot generate such a thing, then you cannot generate an accurate reflection.

If computer vision algorithms are available to you, I would suggest trying to fine one that can reconstruct the depth information of the scene from a single image. If that's not feasible, then you're going to have to live with an inaccurate reflection in one way or another.


Difference in pixels between different monitors' sizes

The answer to both your questions is the same: the physical size of the monitor makes no difference at all. If your screen is 1920x1080 pixels, and you display a 960x540 image without any scaling, the image will stake up a quarter of the screen, and it won't take any more GPU time to display on a larger monitor.

The only caveat is that most OSes have some kind of pixel scaling method now, so that icons, buttons, and text aren't unusably small on modern high-resolution displays. In Windows the option is called "Change the size of text, apps and other items". If this setting isn't set to 100%, most things on the screen are scaled up even if the program isn't scaling them. This is more likely to be set high on a small, high-resolution screen than on a large, high-resolution screen. If you try to test displaying the same image on different-sized monitors, and they are using different values for this setting, the image probably won't be the same number of pixels on the monitors, because it's being scaled up according to this setting.


Ok so setting the "power management mode" to "prefer maximum performance" in the nvidia control panel seems to fix it.

Since I already create an optimus profile automatically through code all I had to do was to set this value as well.


Nice question! I hadn't seen the game myself but this question got me interested so I watched some footage on Youtube.

This one for example gives a good view of it: https://youtu.be/Ktjdg3zgRzk?t=15m46s

A few things I notice:

Hair and right leg: note how only the diffuse lit part has rim lighting.
Left wrist: note how the width becomes thinner at the angle with the hand.
On some other part you can also see the rim lighting is affected by cast shadows too.

From these observations, I think there are two effects used:

A diffuse shading with a sharp transition between lit a unlit, that happens somewhere below 0 so the light leaks behind a little.
The rim lighting that, as you mention, has an almost constant screenspace width.

From the look of it and some of the artifacts, my best guess is that it's actually some edge detection filter based on depth, which is then used in combination with the diffuse lighting so the rim lighting doesn't affect the unlit parts.

It's a pretty cool effect, thank you for point it out!


This answer tries to give a short overview over some important aspects. Since the HSH definition is rather complex and I couldn't find an overview over some pre-evaluated functions, I didn't provide examples simply because it would take me too much time right now.

Problem Description & Brute Force

To determine the convolution any with any set of basis function and thus computing the coefficients we generally need to compute the integral over the domain (= sphere for SH, hemisphere for HSH). Everything we need to do, to represent the hemispherical function f, which is defined over the angles theta ("up/down") and phi ("left/right"), via a coefficient c for HSH basis functions H is the following:

The sin(theta) is there because we integrate over the surface of a (hemi-)sphere. Conceptually, the size of a piece of area that comes from changing phi is bigger or smaller on the current theta. More on this here

If we don't care too much about accuracy or computing time we can solve this simply by sampling: Generate equally distributed (!) directions on the hemisphere, compute the product of f and H and average the outcomes (if you have truly equally distributed points you don't need the sin(theta)).

Get Started with an Analytical Solution

Of course we would love to have an analytical solution for our function, but this is where things can get very difficult. As a first step we may need to convert a function that is given on Cartesian directions into spherical coordinates. This part is still easy, just replace all your x, y and z as following:

Note that this gives us a system where the z-axis is the "up" of the hemisphere (theta=0) which should be represented by the HSH. After that it may already be possible to insert everything into a computer algebra system and solve the equation. Do not try to solve for all m & l but rather try one coefficient at a time, since it is unlikely that there is a compact expression that describes all of them at once. The definition of HSH is relatively complex, which makes it very tedious to evaluate these functions. In this paper the zero and 1st order HSH basis functions are mentioned in cartesian coordinates.

Notes on Rotations & Zonal Harmonics

Functions that are rotational symmetric around this z-axis are very good candidates for a successful analytic derivation, since they only affect the zonal coefficients, which are all coefficients with index m equals zero. This is especially helpful for the more general Spherical Harmonics where an easy formula exists that allows to rotate any Zonal Spherical Harmonics representation to an arbitrary direction, resulting in a Spherical Harmonics representation without any data loss (see here). This means that you can derive ZSH coefficients by assuming that your radial symmetric "function points to z" and rotate it then it into any desired direction. This works perfectly for example with various cosine lobe variations and also gives you the factors you mentioned in the question.

Now the bad news: For HSH, any rotation of a function around another axis than z is lossy, since your function will "touch" the lower undefined hemisphere after the rotation. Therefore, there is also no convenient "Hemi Zonal to HSH" rotation formula. Instead, there are multiple ways to do it with different drawbacks. For more details see the paper and the presentation.

By the way: All this is easier with the H-Basis, which is hemispherical as well (but originally only defined for a limited number of frequency-bands).


Convert RGB image to YUV420 using a hlsl shader

The problem here is that you're not storing the YUV values of a pixel at the same place in the result image. What you seem to be doing is first storing the full size Y image and after that the quarter-sized U image and then the quarter-sized V image after that (also with a strange gap of half the image size between the Y and the UV data).

This doesn't work when trying to map it to a standard pixel shader approach, since a pixel shader outputs the value of exactly 1 pixel in the output image, but you actually would need to write 3 different scattered pixels in the output image, at least as your image data is currently laid out in memory.

So there's a few ways you could get around that:

You could output the YUV values directly into a full-sized YUV image (with the Y, u, and v values of a single pixel stored together in the RGB channels of the output) and then further compress this into the disparate memory layout you need, either on the CPU or in another pixel shader that just decides what channels to read from the Yuv image based on the pixel position inside of the output image.
You could combine the above two shaders into one, doing either the Y, the U or the V computation, based on which quarter of the output image you're in.
You can switch to a different layout for your YUV data, e.g. storing the Yuv values of a 2x2 pixel block of the input image in one compressed pixel of the output image. Then it might actually be useful to write into an unsigned integer image and do the converstion to bytes and the compression yourself in the shader.
You can use your C++ algorithm and write the data in a scattered fashion, but not using a pixel shader rather than a compute shader and writing into a generalized buffer/image instead (or combining the two and writing into a generalized buffer/image rather than the frambuffer directly from the pixel shader).

However, I left the above explanations decidedly vague and general for further studying, since there are a few more direct practical mistakes in your actual shader code (e.g. Out.Color should contain your output values, you're also computing with float, so they're likely in the [0,1] range and you shouldn't do all these integer operations on them), which indicate that you might first want to delve a little deeper into how shaders and HLSL work.


In the end I decided to use Leon Sorokin's RGBQuant.js for quantization, because it offered better flexibility over the color histogram/clustering method. I'm probably going to convert the quantized palette RGB output to HSV and implement custom clustering that picks out the "highlight" colors, favoring diversity of hue for colors with high value and saturation, and low occurrence frequency.


Bevel/Chamfer algorithm for arbitrary 3D mesh

You should look into Medial Axis Transform or Straight Skeletons. They are often used to generate the information you need to bevel things properly. They give you a distance from the bounds of your object at any point in space, which allows you to generate bevels correctly. They are difficult to calculate, though.

You may also want to look into this paper from Valve Software for an alternative way to generate the distance field.


Can I ignore importance, adjoint equations, GRDFs for my path tracer?

No, you don't need to know this stuff to implement basic path tracer.

Basic unidirectional path tracer is quite simple to implement. Just trace bunch of paths of length X for each pixel with uniform distribution over the normal oriented hemisphere at the path's intersection points. Then weight the remaining path with the BRDF at each intersection point and multiply with luminance of light once the path hits a light.

You'll get quite a noisy (but unbiased!) image even for large number of paths and then you can start to look into methods to reduce noise, e.g. importance sampling & bidirectional path tracing. Just validate the more optimized path tracers towards earlier validated path tracers to avoid introducing accidental bias.


The 'jumps' to 'a bit darker' are an optical illusion due to how human perception works. Check out Mach bands in wikipedia. Now as for why you get a step function even though you have a smooth transition in code: this is due to quantization. Your monitor can reproduce a finite amount of grey levels, so you quantize your otherwise continuous range.


How to make a translation relative to the line ax + by + c = 0, affine transformations

To mirror an object in relation to an arbitrary line, you first have to find the coordinates of that object in the frame of reference of that line. For convenience, we will define a frame of reference where the $X$ axis is colinear with the line itself and the $Y$ axis will be perpendicular to the line. Then, we can flip the Y coordinate in that frame of reference to obtain the mirrored point and finally we will reverse the transform from the line coordinate system and back to the global coordinate system.

To define a coordinate system, we will need to use one point the line passes through as the origin $O$ of that coordinate system. This can be obtained easily by setting $x = 0$ or $y = 0$ in the line equation and we end up with the point $\{-c/a, 0\}$ or $\{0, -c/b\}$. Getting the $X$ and $Y$ axis of the new coordinate system is easy. The $X$ axis vector $\{X_x, X_y\}$ will be computed by normalizing $\{a, b\}$ from the line equation. Then we need a perpendicular vector for the $Y$ axis direction. This is easily obtained by using either $\{-X_y, X_x\}$ or $\{X_y, -X_x\}$ (you can verify that by taking the dot product between $X$ and $Y$ which will be zero).

Now that you have an origin and a coordinate system, let's find a mirrored point of vector P on that system. First we need to express it in the new coordinate system. This can be done by projecting $P-O$ to the basis vectors $X$ and $Y$, so the coordinates in the new system will be $\{\langle(P - O), X\rangle, \langle(P - O), Y\rangle\}$.
To obtain the mirrored point, we simply flip the Y coordinate. To obtain the final vector in the original space, we simply invert the first transform. The final point should be $O + X\cdot\langle(P-O), X\rangle - Y\cdot\langle(P-O), Y\rangle$

A note on notation: $\langle u,v\rangle$ was used for dot product between vectors. $u\cdot k$ was used for product between vector and scalar


The post-projective Z range in OpenGL extends between −1 and 1. So, a Z-value of 2 is out of range and will be clipped by the far plane.

The reason the square appears with a quadrant missing is that it's split into triangles from bottom left to top right, and so each triangle has two vertices with Z = 0 and one vertex with Z = 2. So, interpolating along the triangle, it hits Z = 1 (the far plane) halfway across, and everything beyond that is clipped. This results in half of each triangle disappearing.

BTW, you'll need a perspective projection matrix in order for far-away objects to appear smaller on screen. With no projection matrix, you're effectively using a parallel projection.


What's the main difference of pipeline process between Vulkan and DX12?

D3D12 has 4 separate kinds of command lists: direct, bundle, compute, and copy.

Vulkan has similar concepts, but they happen in a different way. Command buffers are allocated from command pools. And command pools are directly associated with a queue family. And command buffers allocated from a pool can only be submitted to the queue family for which the pool was built.

Queue families specify the kinds of operations that the queue can take: graphics, compute, or memory copying operations. D3D12's command queues have a similar concept, but D3D12's command list API has you to specify the list type. Vulkan's gets this information from the queue family the pool is meant for.

The D3D12 "bundle" command list type seems similar on the surface to Vulkan secondary command buffers. However, they are quite different.

The principle difference is this: bundles inherit all state from their executing direct command list, except for the bound PSO itself. This includes resource descriptor bindings.

Vulkan secondary command buffers inherit no state from their primary command buffer execution environment, except for secondary CBs that execute in a subpass of a renderpass instance (and queries). And those only inherit the current subpass state (and queries).

This means you do different things with them, compared to D3D bundles.

Bundles are sometimes used to modify descriptor tables and render stuff, on the assumption that the direct command list they're executed within will have set those tables up. So bundles are kind of like light-weight OpenGL display lists, only without all of the bad things those do. So the intent with bundles is that you build them once and keep them around. They're supposed to be small things.

Vulkan secondary CBs are essential for threaded building of commands intended for a single render pass instance. This is because a render pass instance can only be created within a primary CB, so for optimal use of threads, there needs to be a way to create commands meant to execute in the same subpass in different threads. That's one of the main use-cases of secondary CBs. So the intent is that you'll probably build secondary CBs each frame (though you can reuse them if you want).

So in the end, bundles and secondary CBs are intended to solve separate problems. Bundles are generally dependent on the executing environment, while secondary CBs are more stand-alone.

At the same time, Vulkan secondary CBs can do something bundles cannot: they can execute on compute/copy-only queues. Since Vulkan makes a distinction between the primary/secondary level of the command buffer and the queues where that CB can be sent, it is possible in Vulkan to have secondary command buffers that execute on compute or copy-only queues.

Direct3D 12 can't do that with bundles. The ExecuteBundle function can only be called on a direct command list. So a copy-only command list cannot execute bundles.

Granted, because Vulkan doesn't inherit anything between secondary CBs except for subpass state, and compute/copy operations don't use render passes, there isn't much to be gained from putting such commands in a secondary CB rather than a primary one.

D3D12 has the same separation between compute pipelines and graphics pipelines that Vulkan does. However, when issuing commands, D3D12 has only one pipeline binding point, to which you can bind any kind of pipeline. By contrast, Vulkan has separate binding points for compute and graphics pipelines. Of course, Vulkan doesn't have different descriptor binding points for them, so the two pipelines can interfere with one another. But if you design their resource usage carefully, it is possible to invoke a dispatch operation without disturbing the needs of the graphics pipeline.

So overall, there's no real difference in pipeline architecture here.


Is there something wrong with a column camera matrix?

What you have there is the matrix that goes from eye space (camera space) to world space. You can see this by observing that, for instance, if you apply this matrix to the column vector $[1, 0, 0, 0]^T$, which is the right-vector in eye space, then it will return $[r_x, r_y, r_z, 0]^T$, the right-vector in world space.

You want the matrix that transforms the other direction, which would be the inverse of this one. And assuming $r, u, l$ form an orthonormal basis, the inverse will simply be the transpose.


Latency difference between texture copy on GPU vs CPU?

You are talking apples and oranges here. Copying memory to memory on the CPU is using completely different hardware then memory to memory copies on the GPU, even in a unified memory architecture GPU's tend to have specialized memory.
The actual difference is going to depend heavily on the model of the GPU, a high end gddr5 gpu will easily beat DDR4 dual channel memory on the CPU.
Also, the driver implementation is going to have an effect on the gpu copy since you are really asking the driver for the copy on the gpu, which it then translates into code that does the actual copy.
The size of the chunk of memory also plays a major role in how fast the copy happens, as well as how large the cache on the cpu vs gpu is.
You should generally expect copies that are GPU to GPU to be much faster then CPU to CPU copies. CPU to GPU along with GPU to CPU copies have to cross the AGP bus on the PC while unified architecture systems have an advantage they still have synchronization issues and all systems have to synchronize access and flush cache's.
What are the actual differences?(very roughly) DDR4 runs from 1600 to 3200 MB per second and DDR5 run's at roughly 3200 to 6400 MB per second. So very roughly, a poorly balanced system will have up to 4x performance diffence, but you should expect about a 2x performance differnce with the GPU being faster. This difference fades somewhat(mostly) with unified systems. (But the gpu will probably still have memory set aside so it can work with framebuffers and what not in an efficient manner)


3D coordinates of intersection point in clipping for a rasterizer

I think it depends on what your goal is and what kind of initial data you are provided with. I don't think that dividing by the forth coordinate solves a clipping task. What it does is switching of the coordinates of your objects from 3D projective (four components) to 3D Euclidean (three components).

I would say, the basic component you need is a procedure of calculating the intersection point in projective coordinates of a line and a plane.

An arbitrary point in projective three space is written as a four-vector column:
$$X = \begin{bmatrix}X_1\\X_2\\X_3\\x_4\end{bmatrix}$$ and a plane in projective three space can be expressed as
$$N^T X = 0$$ where
$$N^T X = \begin{bmatrix}N_1 \,\, N_2 \,\, N_3\,\, N_4\end{bmatrix} \begin{bmatrix}X_1\\X_2\\X_3\\x_4\end{bmatrix} = N_1X_1 + N_2X_2 + N_3X_3 + N_4X_4$$
A line in projective three space is traditionally given by a pair of points
$$A = \begin{bmatrix}A_1\\A_2\\A_3\\A_4\end{bmatrix} \,\, \text{ and }\,\,
B = \begin{bmatrix}B_1\\B_2\\B_3\\B_4\end{bmatrix}$$ and any point $X$ on the line is expressed as
$$X = t_1 A + t_2 B$$ for two real numbers $t_1, \, t_2$ not equal to zero simultaneously. Tho find the intersection point of the line $X = t_1 A + t_2 B$ with the plane $N^TX = 0$ we simply plug
$$0 = N^TX = N^T \big(t_1 A + t_2 B \big) = t_1(N^TA) + t_2 (N^TB)$$ and since $t_1, \, t_2$ are determined uniquely up to a common factor, a solution to the equation
$$t_1(N^TA) + t_2 (N^TB) = 0$$ is $t_1 = -\, (N^TB)$ and $t_2 = (N^TA)$.

Therefore, given a plane $N^TX = 0$ and a line determined by a pair of points $A$ and $B$, the unique intersection point of the line with the plane is calculated as
$$P = - (N^TB)\, A + (N^TA)\, B$$ We us projective coordinates, because it may happen that one of the points on the line, say $B$ is "at infinity" with respect to our perferred choice of Cartesian coordinates. That means that
$B = \begin{bmatrix}B_1\\B_2\\B_3\\0\end{bmatrix}$. In this case, the line is interpreted in Cartesian coordinates as the line passing through the 3D Cartesian point $A = \begin{bmatrix}\frac{A_1}{A_4}\\\frac{A_2}{A_4}\\\frac{A_3}{A_4}\end{bmatrix}$ and aligned with the 3D vector $\vec{V}_B = \begin{bmatrix}B_1\\B_2\\B_3\end{bmatrix}$.

In order to clip a line, you can go about it in two different ways, both of which can use the formula for the intersection between a line and a plane. You can find the (projective) equations of the planes of the frustum, intersect the 3d line with them and then project the points onto the screen, which is again intersection of a plane with lines passing through the origin. Alternatively, you can project the two points that determine the line onto the screen (again you can use the formula for the intersection of the line connecing the origin with any of the two points with the plane of the screen), which allows you to know the projection of the 3D line on the screen, and then find the intersection points of the projected line with the borders of the screen. Maybe the second scenario is simpler?

The screen is parallel to the $x,\,y$ plane of the 3D Cartesian frame, while the $z$ axis is perpendicular to the screen and the screen is at a distance $l$ from the origin of the coordinate system. Therefore, the Cartesian equation of the plane of the screen is $z = l$. In projective coordinates, $z = \frac{X_3}{X_4}$, which means that $\frac{X_3}{X_4} = l$  so the projective equation is $X_3 = l\, X_4$ and can be also expressed as
$$0 = X_3 - l\, X_4 = S^TX = \begin{bmatrix} 0 \,\,\, 0 \,\,\, 1 \,\, - l\end{bmatrix} \begin{bmatrix}X_1\\X_2\\X_3\\x_4\end{bmatrix} $$ where
$S = \begin{bmatrix}0 \\ 0 \\ 1 \\ -l\end{bmatrix}$
so if the 3D line is determined by the pair of points $A = \begin{bmatrix}A_1\\A_2\\A_3\\A_4\end{bmatrix} $ and $B = \begin{bmatrix}B_1\\B_2\\B_3\\B_4\end{bmatrix}$ Hence, with $O = \begin{bmatrix}0\\0\\0\\1\end{bmatrix}$, the projection of $A$ on the screen is
$$\hat{A} =  - (S^TO)\, A + (S^TA)\, O = l\, A + (A_3-l\, A_4)\, O = \begin{bmatrix}l\,A_1\\l\,A_2\\l\,A_3\\l\,A_4\end{bmatrix} + \begin{bmatrix}0\\0\\0\\A_3-l\, A_4\end{bmatrix} = \begin{bmatrix}l\,A_1\\l\,A_2\\l\,A_3\\A_3\end{bmatrix}$$  and the projection of $B$ on the screen is $$\hat{B}= \begin{bmatrix}l\,B_1\\l\,B_2\\l\,B_3\\B_3\end{bmatrix}$$ So one can write this projection in  matrix form as follows:

$$P_{scr} = - (S^TO)\, \text{Id} + O\,S^T$$
because $P_{scr}A = \Big(- (S^TO)\, \text{Id} + O\,S^T\Big)\,A = - (S^TO)\,A + O\,(S^TA) = - (S^TO)\,A + (S^TA)\,O = \hat{A}$

\begin{align}
P_{scr} &= - (S^TO)\, \text{Id} + O\,S^T\\
&\\
&= - (-l)\begin{bmatrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0\\0 & 0 & 1 & 0\\0 & 0 & 0 & 1\end{bmatrix} + \begin{bmatrix}0\\0\\0\\1\end{bmatrix}
\begin{bmatrix}0\,\,\, 0\,\,\, 1 \,\,\, - l\end{bmatrix}\\
&\\
&= \begin{bmatrix}l & 0 & 0 & 0 \\0 & l & 0 & 0\\0 & 0 & l & 0\\0 & 0 & 0 & l\end{bmatrix} + \begin{bmatrix}0 & 0 & 0 & 0 \\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 1 & -l \end{bmatrix}\\
&\\
& = \begin{bmatrix}l & 0 & 0 & 0\\0 & l & 0 & 0\\0 & 0 & l & 0\\0 & 0 & 1 & 0\end{bmatrix}
\end{align}
And thus
$$\hat{A} = P_{scr}\, A \,\, \text{ and } \,\,
\hat{B} = P_{scr}\, B$$
When you have your points projected on the screen plane, you would like to get rid of the $z-$coordiante, so you need to remove the third (out of our) component of your projected points, i.e.:
$\hat{A} = \begin{bmatrix}l\,A_1\\l\,A_2\\l\,A_3\\A_3\end{bmatrix} \mapsto \begin{bmatrix}l\,A_1\\l\,A_2\\A_3\end{bmatrix}$ so if you take the projection from four vectors to three vectors
$$P_0 = \begin{bmatrix} 1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{bmatrix}$$ and muptiply it by  $P_{scr}$ from the right as $P_0P_{scr}$, we can take the projective map from the projective three space to the projective plane of the screen
$$P = P_0P_{scr} = \begin{bmatrix} l & 0 & 0 & 0\\0 & l & 0 & 0\\0 & 0 & 1 & 0\end{bmatrix}$$
So, to recap, given  a line in projective three space, passing through the pair of points $A$ and $B$, their screen images, in projective coordinates of the screen, are
$\hat{A} = P\, A$ and $\hat{B} =P \, B$. The formula for the intersection of a line with a plane in projective three space translates verbatim to the intersection of a line given by a pair of points $\hat{A}$ and $\hat{B}$ with another line given as an equation $N^T\hat{X} = 0$ in the projective plane, where $N = \begin{bmatrix} n_1\\n_2\\n_3\end{bmatrix}$,
$$\hat{X}_N = (- N^T\hat{B})\hat{A} + (N^T\hat{A})\hat{B}$$ so
alltogether
$$\hat{X}_N = (- N^TP{B})\,P{A} + (N^TP{A})\,P{B}$$ So this is the formula for the projected points of intersection of the three dimensional line with the frustum's boundary. For example,  the right vertical boundary of the screen is given by an $N_{right\, vert} = \begin{bmatrix} w\\0\\0\end{bmatrix}$ and the bottom horizontal boundary of the screen is given by an $N_{bottom\, hor} = \begin{bmatrix} 0\\- h\\0\end{bmatrix}$


Texture coordinates are not updating according to the vertex divisor specified

Now this texture coordinates supposed to be updated for each vertex processed by vertex shader because i set the glVertexAttribDivisor() to 0 which means it should be.

No, it doesn't.

A divisor of 0 means that the texture coordinate work exactly as if you hadn't called glVertexAttribDivisor for that attribute at all. 0 is not a special case; it's the default.

That means that the texture coordinate will be updated at the same rate as the position. So it will get the same value for each instance, exactly like it does for the positions.

There is no way to use instancing to get an attribute to always increase. It either is a per-instance attribute (changing more-or-less based on gl_InstanceID) or it is a per-vertex attribute (changing based on gl_VertexID). To get the per-instance behavior, you use a divisor 1 or greater. To get the per-vertex behavior, you use a divisor of 0.


Yes,  With a certain OpenGL extension, that is possible (thanks to NicolBolas for pointing that out). It is pretty much the same as in C++. Let's assume you have a file util.glsl, vertex.glsl and fragment.glsl, all in the same folder. $^1$
//in util.glsl
float add(float a, float b)
{
    return a + b;
}

Then you can call that function, as if it were in your shader file
//in either vertex.glsl or fragment.glsl
#include "util.glsl"

void main()
{
    float result = add(1, 2);
    ...
}

Your result would then be 3. Of course, if you have the file in a different folder, you need to adjust the path, but again, that is pretty much the same as in C++.
The only caveat is with debugging. Without having tested it right now, as far as I remember compiler errors will not display your util.glsl as the source of your error. Rather it will put the contents of util.glsl at the position, where your include call is and then say you have an error in that line in vertex.glsl / fragment.glsl, so this might be a bit confusing
$^1$Actually, this is incorrect (by default at least). In a project we used it like that, but since I did not do that setup myself, I assume it was either setup to work like that by a collegue or by OpenSceneGraph, which we used. In any case, I cannot (yet) answer this myself. I want to add this part, as soon as I understand it myself. Until then, I will simply link to a similar question with answer from NicolBolas, which hints at what you have to do: https://stackoverflow.com/questions/10754437/how-to-using-the-include-in-glsl-support-arb-shading-language-include


Reasons of the assumptions for the microfacet distribution function?

It's a geometric assumption like the other two. Consider a flat macrosurface. Its projected area in any direction $v$ is just $v\dot\ \hat N$ times its area (where $\hat N$ is the surface normal). In particular, the case where you're looking at it along the normal is simplest: the projected area is equal to the area of the surface.

Now split the macrosurface into microfacets. The total area of the microfacets is at least as much (assumption 2), but each 'kink' in the surface bends the normals of the separate microfacets away from the original normal. Whatever the shape of the microfacets, the sum of their projected areas doesn't change. In the case where you're looking along the normal, it's easy to see that the total projected area is the same: the surface would have to get larger or smaller for it to change.

For any direction, the microfacet has to cover a portion of the original projected area of the surface. Changing the orientation of the microfacet while still filling that portion doesn't change its projected area.

There's one tricky case, which is where the microfacets overhang each other. In this case, the total area is greater, because some area is covered by more than one microfacet. But in this case, at least one of the microfacets has to end up pointing away from the view direction, back into the surface. In this case, the dot product is negative, so this cancels out the area covered by more than one microfacet. This is why the text is careful to single out that it's the signed projected area.

There's one more tricky case, which is where the microfacets extend past the silhouette of the object. This might happen when you're looking from very glancing angles, or where overhanging facets overhang outside the perimeter of the surface. In this case, the projected area of the microfacets will be greater, violating the third assumption. We don't generally consider this case. Intuitively, it matches up with the fact that techniques like bump-mapping don't change the shape of the silhouette of the object.


What is this shadow mapping error called?

It's not entirely clear what the scene in your picture is supposed to look like, so I'm not sure I'm interpreting it correctly. But if I am, it seems like the problem is that you're projecting the shadow properly, but onto the wrong face of the gray wall. If you want to research it, you might have luck looking up winding order (sometimes called winding number), or back-face culling.

If I'm correct in what's shown above, I think you could solve it in your shader by only applying the shadows to the current fragment if the vector from the light source to the surface points away from the surface normal. In other words, take the dot product of those 2 vectors, and if it's negative apply shadows, otherwise, don't.


What is the definition of "motion to photon" in VR?

Ideally, motion-to-photon latency is exactly what it says. It measures time from when the user physically makes a motion (say, begins to turn their head) to when the photons physically coming out of the display change to reflect that motion.

With a bit of apparatus, this can be measured directly—for instance by using a high-speed camera aimed at the headset. You give the headset a light bump with your finger (or use some automated device to bump it), and you can see in the high-speed video when the motion occurred and when the image visible on the screen updated to reflect the motion.

The complication leading to different definitions occurs because many VR headsets have rolling displays, where the physical pixels on the display light up progressively, one row at a time, as the image is scanned out from the host PC. So, when you define motion-to-photon latency, you have to say which photons you're talking about. The very first ones to scan out? The very last ones? Somewhere in the middle?

You can pretty much make whichever definition you want here about what point in the scanout process to use as your reference. There may be an argument for one definition or another based on psychovisual effects (i.e. how much of your visual field must be updated in order for your brain to perceive motion?) but this is beyond my knowledge.

On the other hand, if a headset has a global display instead of a rolling one, then the pixels are kept dark while being updated and then the whole display is flashed on and off at the end of scanout. In this case it's clear that the appropriate reference point to measure is the beginning of the flash, since all the photons come out at the same time.

Finally, it's worth noting that latency measurements incorporated into an app or API are approximations. The software can't observe the motion nor the photons directly, so instead we must use software-visible proxy events.

The app querying for the latest head pose is a proxy for user motion, but note that this discards any latency in the sensors, USB driver, sensor fusion processing, and time the data spent sitting around in a buffer waiting for the app to query it.
The GPU flipping its display buffers is a proxy for photons beginning to come out, but this discards any latency added in the scanout and display process.

So, ideally an app-measured latency figure should include some corrections to at least estimate the above factors. Or accept that you're not really measuring the full motion-to-photon latency, but just the app-query-to-flip latency or some such.


Meaning of the parameter in a line-plane intersection in the space

The $t$ parameter represents the distance from the ray's origin point to the intersection point, as a multiple of the ray's direction vector. It is positive for intersections "in front" of the origin, negative for those "behind" the origin as judged by the direction vector.

In raytracing we always discard any intersections with $t < 0$, because we don't want to render objects behind the camera, and so forth. A ray is only half a line, and only the points $t \geq 0$ are part of the ray.

It is not correct to say "if $t<0$ the point will stay in a part of plane that does not contain the polygon". If the polygon is behind the ray origin, then the intersection point may well be inside the polygon. However, we can immediately discard this intersection due to its being located behind the ray origin, so we would not need to perform any further tests of whether the point on the plane is in fact inside the polygon.


Can I use several GPUs for a grid multi screen image?

Yes, it's certainly possible to do this kind of thing.  First of all, you can typically connect 3 or 4 displays to each GPU, and (with an appropriate motherboard) you can have up to 4 GPUs per machine.  That would give you 16 screens.  With multiple machines, you could have even more.

Rendering to multiple displays from a single GPU is no big deal.  You can create a single window that stretches across all the displays, render to it, and it should "just work".

Rendering the same scene on multiple GPUs is a bit trickier.  One way is to create a separate rendering context on each GPU, and essentially repeat all the rendering—re-upload all buffers and textures to each GPU, and re-submit the draw calls to each one every frame.  Another possibility is to use the built-in multi-GPU support in DX12, which allows you to manage several GPUs in one rendering context.


The problem with your calculations is that you use a uniform for viewPos which is the camera position.
The rest of your calculations is in Camera-Space, so the viewPos should always be $(0.0, 0.0, 0.0)$ and thereby the $viewDir$ becomes $-1*\text{fragPos}$.

In general: Always be careful about the different reference frames your calculations can be done in and aim to keep this consistent.


Any good sources on advaced/special rasterization algorithms and techniques and theory behind them?

One rasterization method I like is Haar wavelet rasterization:

http://faculty.cs.tamu.edu/schaefer/research/wavelet_rasterization.pdf

It can render polygons and Bezier curve shape and good for different resolution. I not see this algorithm in computer graphics books.


I'm going to provide the simple/naive/brute force answer to this question, which does work, and gives accurate results.

There are better answers however, which make the rendering converge faster with fewer samples, but uses more advanced math.  I'm hoping someone else will provide an answer along those lines.

The simple answer is this:
Area lights are just geometry which have an emissive value.

To add support for emissive values to path tracing is pretty easy luckily.

As you know, the amount of light heading towards a pixel is the amount of light leaving the surface that a ray down that pixel would intersect.

That is calculated recursively by seeing how much light is hitting that point, integrating the BRDF, multiplying by the cosine and all that jazz.

Emissive lighting is just added to that result.  So, whatever you have now as "outgoing light" from a surface, just add to it the emissive color of that surface.  (You could also optionally do something similar to texture mapping and have emissive values vary over the object, possibly based on a texture).

That's all there is to it, it's very simple and easy.

In the rendering equation below, $L_e(\omega_o)$ represents the emissive term (that's what the e stands for, emissive), and the rest of the equation is the light integrated over the hemisphere.  You can see that you just add the emissive to the result you return as your outgoing light from a point ($L_o( \omega_o)$) to apply that emissive lighting.

$L_o( \omega_o)= L_e(\omega_o)+\int_{\Omega}{f(\omega_i, \omega_o)L_i(\omega_i)(\omega_i \cdot n)\mathrm{d}\omega_i}$

You can read more about a simple path tracer with area lights at this link, which also has commented C++ source code.

http://blog.demofox.org/2016/09/21/path-tracing-getting-started-with-diffuse-and-emissive/


How exactly does Gouraud shading apply color across a polygon? Does it do it by pixel or is it some other method?

The difference is in the number of lighting calculations done. Where the lighting calculation produces some color.
Flat does one per face, uses the color across the face.
Gourand does 3 (for a triangle, one for each vertex), interpolates the color across the face.
Phong does a light calculation for every pixel/fragment (where the normal is interpolated across the face).


As far as I remember, you need to input data in float array. Right now your input is unsigned char array. To quick check this, try loading any HDR image in float array & pass it on to the setImage functions, this seems to work.

In your case, you would need to read individual image channel data & cast it to float. If your image color is stored in [0-255] range, you would then need to bring it to [0-1] range (Not completely sure about this stage!) & then pass the data in float array to denoiser functions.

Hope this helps.


Asset creation for signed distance field rendering?

Signed distance fields are popular in minimal graphics applications, such as the demo scene, where interesting objects can be synthesized from few simple analytic primitives such as spheres or cubes. However, signed distance fields are not restricted to these simple objects, and they do not necessarily need to be designed by a human. For example, you can synthesize the analytic signed distance field of a triangle mesh without any artist interaction - the SDF is just the signed distance to the closest triangle after all, which can be easily computed. This allows you to keep using your traditional mesh pipeline while also using SDFs in the background.

In the case of the Unreal Engine, the SDF is precomputed automatically once for all static meshes and is then sampled into a low-resolution 3D texture. It can then be cheaply evaluated everywhere using a simple texture lookup, allowing them to do soft shadows and similar at runtime. This Unreal presentation at GDC 2011 mentions the 3D textures briefly (slide 27).


Animating a smooth linear transformation

As a general rule, you cannot interpolate transformation matrices. In stead, you decompose them into their individual values, then interpolate those and recompose.

The Möbius transformation as suggested in the comments sounds interesting, but tradionally I'd just extract scale and rotation and interpolate those.

Assuming a transformation matrix

    | a  b |
T = | c  d |

the scale is [sqrt(a² + c²), sqrt(b² + d²)]
The rotation angle is atan(c/a)

You'd now interpolate this scale and rotation angle, recomposing the matrix at each frame.

[Edit] Corrected the angle (as spotted by Nathan Reed) [/Edit]


Why do we have to partition non-convex polygons into convex polygons in a map in Binary Space Partitioning?

Because with 2 non-overlapping convex polygons you can always say that 1 polygon is closer to a point than another.

If they are not convex then for example with a U surrounding a circle. You cannot say easily which should be drawn first.


It works almost exactly the same way as vertex shader inputs.

In vertex shaders, you have inputs. And these inputs represent vertex attribute indices, the numbers you pass to glVertexAttribPointer and glEnableVertexAttribArray and so forth. You set up which input pulls from which attribute. In GLSL 3.30, you use this syntax:

layout(location = 2) in color;

This sets the color vertex shader input to come from attribute location 2. Before 3.30 (or without ARB_explicit_attrib_location), you would have to either set this up explicitly with glBindAttrbLocation before linking or query the program for the attribute index with glGetAttribLocation. If you don't explicitly provide an attribute location, GLSL will assign a location arbitrarily (ie: in an implementation-defined manner).

Setting it in the shader is almost always the better option.

In any case, fragment shader outputs work almost exactly the same way. Fragment shaders can write to multiple output colors, which themselves get mapped to multiple buffers in the framebuffer. Therefore, you need to indicate which output goes to which fragment output color.

This process begins with the fragment output location value. It's set very similarly to vertex shader input locations:

layout(location = 1) out secColor;

There are also the API functions glBindFragDataLocation and glGetFragDataLocation, which are analogous to glBindAttribLocation and glGetAttribLocation.

If you don't do any explicit assignments, implementations usually will assign one of your output variables to location 0. However, the OpenGL standard does not require this behavior, so you should not depend on it either.


What's the difference between orthographic and perspective projection?

Orthographic projections are parallel projections. Each line that is originally parallel will be parallel after this transformation. The orthographic projection can be represented by a affine transformation.

In contrast a perspective projection is not a parallel projection and originally parallel lines will no longer be parallel after this operation. Thus perspective projection can not be done by a affine transform.

Why would you need orthographic projections? It is useful for several artistic and technical reasons. Orthographic projections are used in CAD drawings and other technical documentations. One of the primary reasons is to verify that your part actually fits in the space that has been reserved for it on a floor plan for example. Orthographic projections are often chosen so that the dimensions are easy to measure. In many cases this is just a convenient way to represent a problem in a different basis so that coordinates are easier to figure out.

Image 1: A number of useful orthographic projections for same object (and projection rule). The last on on the right is a special case called isometric having the property that cardinal axe directions are all in same scale.

A perspective projection is needed to be able to do 2 and 3 point perspectives, which is how we experience the world. A specific perspective projection can be decomposed as being a combination of a orthographic projection and a perspective divide.

Image 2: 2 point perspective note how the lines in prespective direction no longer are parallel

Viewport transformation allows you to pan/rotate/scale the resulting projection. Maybe because you want a off center projection like in cameras with film offset, or you have a anisotropic medium for example. It can also be convenient for the end user to zoom into image without changing the perspective in the process.


With help from the Nvidia Forum and two very helpful and friendly Nvidia employees I found the perfect solution.

In OptiX there is a so called selector node. This is a special scene-graph node that utilizes a user specified program to select which child a ray should traverse and which not.

#include <optix.h>

rtDeclareVariable(unsigned int, timestep, , );

RT_PROGRAM void animation_selector()
{
    rtIntersectChild(timestep);
}

This is a simple version of such a selection program. From the CPP-API side the current timestep will be set and can be used to make the animation. And then when the scene graph is traversed the program just says "Traverse only the child with index 'timestep'". This does not trigger a recompile or refit and therefore is nearly instant.

The selector node is simply created in following way:

optix::Selector selector = context_->createSelector();

std::string path_to_ptx = VRTPtxFinder::GetPtxPath("VistaRayTracing_generated_animation_selector.cu.ptx");

optix::Program selection_program = context_->createProgramFromPTXFile(path_to_ptx, "animation_selector");
selector["timestep"]->setUint(0u);
selector->setVisitProgram(selection_program);

for(auto group : groups)
{
  selector->addChild(group);
}

The only cost is one additional method call for each traversal and thereby also a slightly bigger stack which caused me some problems with a stack overflow so when using this, keep an eye on the stack size.


When you think about rain in the distance (generally), you will have rain at several different depths from the camera, some close up, some very far away, all of which will look slightly different as you won't be able to focus on all of them. But the effect of them all layering on top of each other as they go off into the distance is what helps give the look you're aiming for, you need to recreate that depth and layering to really give the look of rain further in the distance.

Particle emitters are the general method of creating rain, as usually they use very small meshes, or even just a texture sprite of a drop. This keeps them pretty efficient - to a degree as the more particles you use and have on-screen at a time will change how heavy or resource dependant the effect becomes.

Generally you would only use large detailed meshes for rain, when you have a close up of a puddle and wanted to animate the splash, or even a close up of the single drop for some reason.

This can be combined with distance fog to further increase the sense of depth in your scene by occluding the objects in the far distance.

Again you could use another particle emitter to create dark, heavy clouds that go off into the distance, as having clouds in the scene will help to sell the look.

A lot of this will depend on what software you're using, as well as its intended platform, as different software have different solutions but particles and fog are pretty commonplace.

tldr: Particle Effects & Distance Fog


What are the type constants that the documentation for glGetProgramResourceiv() refers to?

For whatever reason, the text for that function documentation was copied from the GL_ARB_program_interface_query extension. Extensions in OpenGL are always defined relative to a specific version of the OpenGL specification. So if an extension references something in a specification, it's talking about that version of the spec.

However, the OpenGL specification was massively restructured between 4.2 and 4.3, with whole chapters moved around. This means that table references in 4.2 become completely wrong relative to 4.3 and above.

Program interface query was written against the 4.2 specification. Oddly enough, there is no table 2.16 in that spec; the correct number for that table is 2.13 (so it's likely a typo, since 3 is right below 6 on a number-pad). But even if the extension did mention the right table number, that table moved to a whole different chapter in 4.3+ versions. It's been renumbered and moved since then, so you'll have to look at a particular specification's definition of glGetProgramResourceiv to find it.

In general, if you find something incongruous in the documentation pages, you should look at the actual OpenGL specification to see what it says about the function's behavior. In this particular case, the docs in the OpenGL wiki for this function actually includes the table.


A solution to sprite visibility in a raycasting engine with variable wall heights?

Congratulations for having progressed so far.
Context, first analysis
Your explanation:

I am using a z-buffer, with one stored length to a wall for the x coordinate of each ray.

Means the z-buffer is currently one-dimension.
One dimension is enough in the first case (your first two images) but not in the second case. To see why, imagine that the red tomato sprite was closer to the player, such that the left half of it would be partially obscured by the first wall (bottom left). Since the top of that wall is projected to a diagonal line, it means that the property "is pixel P of the sprite obscured?" does not only depend on X but also on Y.
So, in the exact setup you have, a 1D Z-buffer is definitely not capable of performing the clipping you want.
Options
From that on, here are a few options:

switch to a 2D z-buffer (slower)
switch to painter's algorithm (slower)
switch to a 2D-based occlusion algorithm
study how Doom did it

Your geometry seems similar to Doom. It is probably a very good investment for you to study how the Doom rendering engine works. It clips sprites (called "things") using a 1D criterion, then refines. One area has only one floor height and one ceiling height. This is very important and it looks like your geometry is similar.
Solution
The solution, to summarize, is to keep not only a 1D z-buffer, but actually a structure keeping for each sector (in doom parlance) the lines that make the limit and their parameters (like the a and b in y=ax+b).
Then, when you want to draw the sprite, clip X with 1D criterion but depending on the sector the sprite is in, and then limit the Y range of pixels drawn based on the y=ax+b of the sector just in front of it.
I don't know what data structure you currently have in your program. If this seems totally unreachable, then you need to rethink the data structure you keep while drawing.
If you haven't yet, you should read (sorry if you did already) https://doomwiki.org/wiki/Doom_rendering_engine#Rendering and the pages behind the links there, perhaps run their code in slow motion or see a video where someone shows/explains how it works.  This one lacks explanations but might help your intuition: https://www.youtube.com/watch?v=gfM6NpxmVGg .
Hope this helps.  Keep up the good work!


One tool I've seen people use for this sort of thing is Processing, a Java-based language for real-time artsy stuff that supports graphics and audio. I haven't ever worked with it myself, but apparently it's pretty easy to use it for programming visualizations. Here's an example of a simple visualization someone made; look in the video description for a link to their code—it's quite short. Their example loads an mp3 file, but Processing also supports real-time input from a mic.


Have you confirmed that objects that clip a chunk border are stored in both chunks?

Have you confirmed that octrA[map.x/2][map.y/2][map.z/2] can't point to anything weird?

In fact, what even is octrA? I assume that it's the array storing the chunks but I can't know for sure because I couldn't find it anywhere else in your code.

Other than that I'm having a bit of a hard time interpreting the problem and your code. Could you add some pictures and comments to your code?


Render Duplicate Miniature Scene Replica

The first thing that jumped into my head when I saw this was Multiview. You are probably already using it to render the scene if you are rendering to VR anyway. (if not...you should)
For those that don't know about Multiview:
This is a way to cast the same draw calls to multiple layers of a framebuffer image with a single draw call. It allows you to use a mask for which layers to broadcast to, so it is easy to turn broadcasting on and off (making it quick and easy to turn the WIM on and off). A new variable is available in shaders that allows the code to make adjustments for the view/projection matrix usually but the code can get much more sophisticated as needed. Its only downside is it requires relatively modern hardware.
Using multiview the cpu would set up 4 view matrices, two for the VR left and right eye, and two for the WIM left and right eye. The view could then add a constant scale factor so the WIM is sized to fit, and the camera can be positioned as needed.
To improve performance you will probably want to break it up into multiple passes where one pass renders everything that is common to both scenes and the second pass renders more expensive effects for the main scene. Again, as needed.
The last step would be merging the two scenes, probably just by bliting the WIM over the main scene using a unique background color of the WIM as a clear alpha color.
I won't go into the details of implementing multivew but googling multiview for the API you are using should give some good tutorials.
Scene interaction is simplified since the CPU only really ever works with a single scene and the only tricky part is figuring out whether the user is interacting with the WIM or the main scene. This would effect how the ray is being cast into the scene (if you use ray casting for selecting objects).
I just noticed that multiview is a opengl/vulkanism I'm not sure what the directx equivalent is.


Turn an enclosed region into a triangle mesh

If the number of vertices is not too large, I would suggest looking into "ear clipping". It's a straightforward method to triangulate possibly-concave polygons, as long as they don't have holes. However, it's $O(n^2)$ in the number of vertices, so may not a great choice if you need to handle a large number of verts. Dave Eberly has a paper on ear clipping and also an open-source implementation in C++. There are many other links available on the web, too.

The basic idea is to find an "ear" of the polygon, which is defined as a triangular subset of the polygon, formed by three consecutive vertices of the polygon. Once you find an ear, you add it to the output triangle list, and remove it from the original polygon (which reduces the number of vertices by one). Repeat this until the polygon is gone.

The reason it's $O(n^2)$ is that you have to repeatedly search the list of vertices to find another ear to remove. There are more efficient triangulation algorithms as well, but they're more difficult to implement.


Radiative Transfer Equation for Photorealistic Rendering

I suspect the answer to (a) is simply performance. Full volumetric path tracing / photon mapping based on the RTE can certainly be done (and I'm sure it sometimes is), but it's very expensive and requires enormous numbers of samples to converge to a noise-free result.

This is especially true when the scattering medium is very dense, like in human skin and hair, or materials like milk, plastic, marble, or jade. In such materials, the light scatters so quickly that it's essentially isotropic after a couple of millimeters, so a BSSRDF is a very good approximation to the true volumetric scattering, and far cheaper to evaluate. (Performance matters even for offline rendering!) A mixture of volumetric simulations and empirical data might be used to derive and validate the BSSRDF.

As for how the coefficients are measured, one technique I've read about is to shine a narrow, bright beam of light (like a laser pointer) onto the material, then measure the light scattered back out using photosensors placed at different distances from the incident point. By processing this data you can then get an estimate of the scattering coefficients. This paper shows a sensor based on this idea being applied to human skin, and this one has more details about setting up a sophisticated (multi-layer) BSSRDF based on empirical data.


Why is $pdf(\Psi) = \frac{1}{2\pi}$ when picking from hemisphere uniformly?

It's like this because the surface area of a unit sphere is $4\pi$. As ratchet freak points out, the integral of a probability distribution over its domain has to be 1. Put another way, the probability of choosing some direction is 1. The surface area of the hemisphere is $2\pi$. The constant that you integrate over a $2\pi$ domain to get 1 is $\frac{1}{2\pi}$.

The connection to steradians is that 1 steradian is defined as the solid angle that subtends a surface of area 1 on a unit sphere. That is, a unit sphere has $4\pi$ steradians and a surface area of $4\pi$.

It's just the same in two dimensions. A radian is chosen so that a 1 radian arc of a unit circle has length 1, so there are $2\pi$ radians in a circle, and $\pi$ radians in a semicircle. If you choose 2D directions uniformly on a semicircle, the probability distribution is $\frac{1}{\pi}$.


How does mip-mapping work with non-power-of-2 textures?

The rule is that to compute the next mipmap size, you divide by two and round down to the nearest integer (unless it rounds down to 0, in which case, it's 1 instead). For example, a 57x43 image would have mipmaps like:

level 0: 57x43
level 1: 28x21
level 2: 14x10
level 3: 7x5
level 4: 3x2
level 5: 1x1

UV mapping, LOD selection, and filtering work just the same way as for power-of-two texture sizes.

Generating good quality mips for a non-power-of-two texture is a little trickier, as you can't simply average a 2x2 box of pixels to downsample in all cases. However, a 2x2 box filter wasn't that great to begin with, so using a better downsampling filter such as Mitchell-Netravali is recommended regardless of the texture size.


What is the technique to generate an oblique 3d cone?

The general equation for a upright cone with the tip at $Z=0$ is

$$ 0 = X^2 + Y^2 - Z^2$$

(Ignoring the points for which $Z>0$). If you consider slices for some constant $Z$ you get circles centered on the line $X=Y=0$. To make an oblique cone with circles along the line $X=aZ,Y=bZ$ we can just shift the coordinates accordingly and get the equation

$$ 0 = (X-aZ)^2 + (Y-bZ)^2 - Z^2$$

(again for $Z<0$). You can make it narrower or wider by scaling the $Z$ coordinate. To position the tip at some other point $(x,y,z)$ you can just shift the whole thing using the transform $(X',Y',Z') := (X-x,Y-y,Z-z)$.

With the same idea you can easily parametrize the (upright) cone by

$$(u,v) \mapsto (u\cos(v),u\sin(v),u)$$

or the oblique cone by

$$(u,v) \mapsto (u\cos(v)-au,u\sin(v)-bu,u)$$


What technique would I use to create fake 3d walls as seen in Realm of the Mad God

What they're using in this game is called an orthographic projection. It is a special type of a projection, which does not bend objects, it does not mean that they are not using 3D.

Now, of course you can achieve this kind of effect using 2D coordinates only if you want. But it requires some mathematical understanding of the situation, as you need to do 3D calculations in order to calculate the correct 2D coordinates to use. If you do it this way, you can draw points or lines between the points after calculating the positions, and it  will give the effect you want.

But here in RotMG, they are also doing texture mapping on those walls, which uses information about the actual size of the surface in 3D space. This information is lost if you simply calculate the 2D positions of the corners without also doing the texture mapping yourself.

So here you have two options:

You can give up your console and start doing some real 3D stuff (OpenGL, etc...).
You can become very good at linear algebra, preferably including homogenous coordinates, project the points from 3D to 2D space, draw your lines/points in your console, or interpolate the 3D positions with some form of rasterization, if you want to decorate the wall using texture mapping. But once you understand all of that, I'm afraid you won't want to use your limited console anymore.


Is it reasonable to expect the image of the described scene to be "in grayscale"? If so, do light intensity spectra indeed are required to be multiplied by D65 spectrum?

My expectation from a physically-based spectral rendered is that it would

compute the physical spectra
use the CIE color matching functions to relate that to a quantified color space (eg CIEXYZ)
convert that into the destination color space.

D65 shall not appear anywhere till step 3, and only if we actually render into sRGB (or similar).

A flat spectra translates to XYZ = (0.33, 0.33, 0.33), which is RGB = (1.20, 0.95, 0.91) prior to gamma compression. So the "red tint" is the expected result following the above procedure.

Don't know why Mitsuba would have such a 'feature'.


What does $E_L$ mean in the context of shading equations from the book "Real time rendering"?

The line you quoted says it: $E_L$ is the incoming irradiance at the surface due to the light source under consideration. Less technically, it's a vector representing the intensity and color of the incoming light at that point.

The equation you mention is for a punctual light source (directional, point, or spot light), so in this case $E_L$ would simply be the user-defined RGB color of the light source, multiplied by any relevant attenuation factors—such as shadows, distance attenuation for point/spot lights, and cone angle attenuation for spot lights.


I think there might be a misprinting in the book. I am getting this

$((A-C) + tB)\cdot((A-C) + tB) = R\cdot R$

Let  A-C = Y

$(Y + tB)\cdot(Y + tB) = R\cdot R$

$Y\cdot Y + Y\cdot tB + tB\cdot Y + t^2B\cdot B = R\cdot R$

Substituting back

$(A-C)\cdot(A-C) + 2t * B\cdot(A-C) + t^2B\cdot B = R\cdot R$


How is lighting done in Rasterization based pipeline?

I'm guessing Rasterization based pipelines use the same procedure for shading i.e. A ray is shot from the pixel and then from the triangle that got projected on that pixel to the light source?

Close, but not quite. There's no actual tracing of rays but the underlying linear algebra is the same. It sounds like you already understand the first part of rasterisation, as far as getting some fragments which need to be shaded. Each fragment knows the attributes of the triangle it came from (the world position, normal, etc.). In traditional forward shading, the fragment shader runs once per light, and it also knows the attributes of the light, including the world-space direction from the shading point to the light: the direction you would trace a shadow ray.

Although no shadow ray is traced, the actual lighting calculation is exactly the same: you use the view and light directions and the surface normal in whatever BRDF you want. You just repeat the process for each light and sum the results. At the end, you have the same result that you'd get from a very simple ray-tracer, without any shadows, reflections, refractions, or indirect lighting.

How to add those missing features? Well, there are several possible techniques for each, but I'll briefly go over the simplest. A simple way of getting shadows is using shadow maps. Before you render your main scene, you do a render into an off-screen buffer, but using the light as a camera. (The projection and shape depend on the light, e.g. orthogonal projection for a directional/cylindrical light.) The trick is, you don't render all the colours: you only draw into the depth buffer. What you get is a picture of the distance from the light to the closest thing in each direction. Later on, when you're shading for that light's contribution in the fragment shader, you calculate the distance from the light to the shading point, and compare it to the value in the depth buffer. That tells you whether the shading point is the closest thing to the light along that ray, without having to run an intersection test.

Planar reflections and refractions are pretty simple too: just render the scene but with the camera transformed to produce the reflection or refraction image. For example, for a planar reflection in a lake or floor, you reflect the camera position about that plane to draw the whole scene upside-down. When you draw the real scene around it, any transparency shows that reflection.


Distorted OpenGL graphics in certain models of Intel GPUs under Windows; possible causes?

As mentioned on a comment, after talking about it on IRC, they helped me realize the bug was in the geometry shader that was being used for Stereo 3D.

Due to the way it was conceived (which was apparently a really bad idea), the shader was always compiled and linked if the hardware reported supporting OpenGL3.30 and GL_arb_viewport_array.

When the stereo3d option was disabled, via a very hacky solution, the shader was put into a "pass-through" mode which is supposed to do nothing but (still not sure if it's a bug in the shader itself or a hardware/driver issue) it was sending a lot of undefined values to the fragment shader, instead.

Here's the offending code:

#version 330 core
#extension GL_ARB_viewport_array : require

uniform float sepFactor;
uniform float z0;

layout(triangles) in;
layout(triangle_strip, max_vertices = 6) out;

struct vData {
    vec3 lightDir;
    vec2 texCoord;
    vec3 normal;
    vec4 color;
};

in vData vertex[];
out vData fragv;

void passthru(int i) {
    gl_Position = gl_in[i].gl_Position;
    fragv = vertex[i];
}

// Process all the vertices, applying code to them before emitting (do-while to convince Nvidia of the code getting executed)
#define PROCESS(code) i = 0; do { passthru(i); code; EmitVertex(); } while (++i < 3); EndPrimitive();

void main() {
    int i = 0;
    if (sepFactor == 0.0) {
        gl_ViewportIndex = 0; PROCESS(;); // No stereo
    } else {
        gl_ViewportIndex = 1; PROCESS(gl_Position.x -= sepFactor * (gl_Position.z - z0));  // Left eye
        gl_ViewportIndex = 2; PROCESS(gl_Position.x += sepFactor * (gl_Position.z - z0));  // Right eye
    }
}

I have sort-of solved it by actually recompiling and relinking the shaders to include the geometry shader or not when stereo3d is toggled.

I've also rewritten the shader to look like this (it still produces skewed geometry but atleast akin to the first screenshot, instead of the second),

#version 330 core
#extension GL_ARB_viewport_array : require

uniform highp float sepFactor;
uniform highp float z0;

layout(triangles) in;
layout(triangle_strip, max_vertices = 6) out;

in vData {
    highp vec3 lightDir;
    highp vec2 texCoord;
    highp vec3 normal;
    highp vec4 color;
} vertices[];

out vData {
    highp vec3 lightDir;
    highp vec2 texCoord;
    highp vec3 normal;
    highp vec4 color;
} fragv;

void passthru(int vp, int i) {
        gl_Position = gl_in[i].gl_Position;
        gl_ViewportIndex = (sepFactor == 0.0) ? 0 : vp;
        gl_Position.x -= (sepFactor * (gl_Position.z - z0));
        fragv.lightDir = vertices[i].lightDir;
        fragv.texCoord = vertices[i].texCoord;
        fragv.normal = vertices[i].normal;
        fragv.color = vertices[i].color;
}

void main() {
        // Render the left eye
    int i = 0;
    do {
        passthru(1, i);
        gl_Position.x -= (sepFactor * (gl_Position.z - z0));
        EmitVertex();
        } while (++i < 3);
        EndPrimitive();

    i = 0;
        // Render the right eye
    do {
        passthru(2, i);
        gl_Position.x += (sepFactor * (gl_Position.z - z0));
        EmitVertex();
        } while (++i < 3);
        EndPrimitive();
}

In order to provide a more complete answer, in case somebody faces the same issue: I finally realized the issue appears to be with the way the loop was written. For some reason, not defining i in the loop itself resulted in trying to access undefined members of vertices

Rewriting as follows fixed the issue:

for (int i=0; i < gl_in.length(); i++) {
    passthru(1, i);
    gl_Position = gl_in[i].gl_Position;
    gl_Position.x -= (sepFactor * (gl_in[i].gl_Position.z - z0));
    EmitVertex();
}


How are mipmap levels computed in Metal?

Mip selection is pretty well standardized across devices today—with the exception of some of the nitty-gritty details of anisotropic filtering, which is still up to the individual GPU manufacturers to define (and its precise details are generally not publicly documented).

A good place to read about mip selection  in detail is in the OpenGL spec, section 8.14, "Texture Minification". I'd assume it works the same way in Metal. (Apple could have changed something, considering they make both the hardware and the API...but I doubt they have.) I'll summarize it here.

The default mip selection (not using any of the lod_options modifiers) uses the screen-space gradients of the texture coordinates to pick the mip levels. Essentially, it tries to pick the mip levels that produce as close as possible to a 1:1 mapping of texels to pixels. For example, if the gradients have a length of 4 texels per pixel, it would pick mip level 2 (which is 1/4th the size of level 0 and therefore gives you 1 mipped texel per pixel).

With trilinear filtering, since you usually don't land on an exact 1:1 mapping, it picks the two nearest levels and linearly interpolates between them, so that you have a smooth transition between mip levels as the camera or objects in your scene move around.

To be mathematically precise: you take the screen-space gradients of the texture coordinates, scale them by the texture size (to get them in units of "texels per pixel"), take their lengths, take the longer of the two (X and Y) gradients, and calculate its logarithm base 2. This quantity is called $\lambda$ in the OpenGL spec; it's also commonly known as the texture LOD (level of detail). It's simply a continuous version of the integer mip level index. So, the mip levels used are the two integers nearest to $\lambda$, and the fractional part of $\lambda$ is used to blend between them. For example if $\lambda = 2.8$, the GPU will sample mip levels 2 and 3 of the texture, then blend to 80% level 3 and 20% level 2.

If anisotropic filtering is turned on, then instead of simply using the longer of the two gradients, you use the ratio of them to set the number of aniso samples. For example, if the X gradient is 4 times longer than the Y gradient, you'd use 4 aniso samples, with their positions spaced out along the X gradient. Each one would be a trilinear sample, using a $\lambda$ corresponding to 1/4th the length of the X gradient (i.e. two mip levels lower, since $\log_2(1/4) = -2$).

Now, as for the modifier options:

bias applies an offset to $\lambda$ before using it to select mip levels. So, for example, a bias of +1 will make it do trilinear filtering using mips one level higher than it would normally. A bias of −1 will use mips one level lower, and so on.
level completely overrides the automatic calculation of $\lambda$ from screen-space gradients, and lets you put in your own value of $\lambda$ (aka lod) directly.
gradient2d lets you put in your own gradient vectors, which substitute for the implicit screen-space gradients of the texture coordinates. The rest of the mip selection and sampling process proceeds as normal, but with the altered gradients. This lets you customize anisotropic filtering.

In other types of shaders besides fragment shaders, there's no notion of "screen-space gradients", so the latter two operations are typically the only ones allowed—any operation that tries to use implicit gradients would give a compile error. I'm not positive that's how Metal does it, but that's what I'd expect from working with other APIs.


Conceptually, a frame is the image you're trying to create, presumably for the purpose of display. It is the product of your entire rendering operation for a specific slice of time.

A "buffer" is the memory for the image you either are trying to create or have already created. Hence the common term: framebuffer. Indeed, the choice of the word "buffer" is because you are typically buffering multiple images. That is, one buffer will currently be in the process of being displayed, while another buffer is being rendered.


First off, here's the Kajiya method I think you're thinking of: Kajiya, Ray Tracing Parametric Patches, SIGGRAPH 82. The tech report version might be more informative.

What I hope you get from that is that it's not impossible and it's not conceptually difficult if you don't mind getting your hands dirty with some algebraic geometry and complex numbers. However, doing it directly is absurdly expensive.

"Real" ray tracers tend to do some combination of two things:

Placing a bounding hierarchy (e.g. AABBs) on the patch to get a good "initial value" for a numeric root finder. If you do this well, you can avoid the "wrinkle" problem.
Tesselating the patch into DDG shells and ray tracing them like polygon meshes.

That last point sounds like it kills the "super-smoothness" requirement, but it isn't nearly as bad as that if you're using ray differentials. Matching the tessellation level to the "size" of the ray bounds the error nicely. Besides, you probably need differentials for texture coordinates anyway, so you might as well use it to control the accuracy of the intersection test too.

Exploiting temporal coherence isn't a bad idea, but exactly how you'd do that depends a lot on your scene graph representation. You might want to look at ray coherence. Ask your favourite search engine about ray packet tracing and ray reordering.


How does the Painter's Algorithm handle transparency?

The common way to render transparent polygons in a rasterizer is by use of Alpha Blending, which basically combines the colour of the supposedly transparent pixel with the colour of the background at that pixel's position, this way rendering the pixel transparently onto what was already rendered before. This technique, however, requires your polygons (or at least the transparent ones) to be sorted from back to front, at least in general non-trivial cases, since you can only combine the transparent colour with what has already been rendered behind it.

But since the Painter's Algorithm already requires ploygons to be sorted from back to front (which is pretty much the entire essence of the algorithm), this just fits naturally into the rest of the algorithm. Thus, supporting transparency becomes rather straight forward by simply doing any variant of alpha blending of the pixel with the background instead of overwriting the background.


Why are normal maps blue and purple in color?

Only tangent space normal maps are primarily blue. This is because the colour blue represents a normal of (0,0,1) which would be an unchanged normal when the triangle lies in the plane x and y, i.e. perpendicular to the surface. The tangent, x and bi-tangent, y (also referred to as bi-normal) are encoded in the red and green channels and these form to create the a tangent space normal for a point on the surface of the triangle.

If a tangent space normal map was to encode a colour only in red (1.0, 0.0, 0.0) this would generate a tangent-space normal parallel to the triangle surface. This is never seen because it would mean that the triangle would only ever be lit at 90degrees from the surface and view vector at which point you wouldn't be able to see the triangle anyway.

World space normal maps encode the unit normal over a sphere so can be primarily different colours once encoded from [-1, 1] to [0, 1] per channel.

A comparison can be seen here:

In practise normal maps are usually encoded in a 2 channel format such as BC5 which actually only stores the x and y with the z being reconstructed as we know it's a unit vector. This allows you to maintain higher precision with more bits without increasing the file size.


How to compute the following integral over a polygon?

Triangulate the Voronoi cell then write the integral as a sum over the triangles:

$$\int_{\Omega}\|P - Pi\|\,dP = \sum_{k=1}^{N}\int_{\Delta_k}\|P-P_i\|\,dP.$$

Write the integration over the triangle in barycebtric coordinates. Let the Jacobian of the transformation, from barycentric coordinates to physical coordinates, for triangle $k$ be $J_k$, and $|det(J_k)| = 2|Area_{\Delta_k}|$. Then $\int_{\Delta_k}f(P)\,dP = |det(J_k)|\int_0^1\int_0^{1-\beta}f(P(\alpha, \beta))\,d\beta\,d\alpha$, where $\alpha, \beta$ are two of the barycentric coordinates.

Now we need only evaluate (let $\vec{v}_i$ be the vertices of the current triangle):
$$\int_0^1\int_0^{1-\beta}\|\vec{v}_1 + \alpha (\vec{v}_2 - \vec{v}_1) + \beta (\vec{v}_3 - \vec{v}_1) - \vec{P}_i\|^2\,d\alpha\,d\beta$$

Expand this and integrate the polynomials. The absolute value of the Jacobian determinant is: $$|det(J)| = \|(v_2-v_1) \times (v_3 - v_1)\|$$.
If you are working in 3D, $\times$ is the cross product here. If it's 2D then augment the vectors with a $0$ for $Z$ and perform the cross product.

Another option is to use a quadrature rule to compute this. For example you can still triangulate, but evaluate the integral over a triangle (let $f(P) =\|P - P_i\|^2$) through samples at the edge midpoints:
$$\int_{\Delta}f(P)\,dP = \frac{|det(J)|}{6}\sum_{k=1}^{3}f\left(\frac{\vec{v}_k + \vec{v}_{k+1}}{2}\right).$$

If memory serves me right, the above quadrature is exact for polynomials of up to degree 2 (which is what you have, and the reason I wrote $=$ instead of $\approx$). Note that $k+1$ is taken to wrap around, that is, for $k = 3$ I use $\vec{v}_4 = \vec{v}_1$.


I think you may want to take another look at the iOS user interface if you consider real-time blurs to be out of range of mobile hardware:

Blurs are totally in range of mobile hardware. Yes, you need a fairly large number of texture samples for a blur with a large kernel, but the texture samples are also cached very well, and you can use a separable blur, which effectively means that an MxN blur requires M + N samples, not M * N samples. You can also take advantage of hardware bilinear sampling to reduce the number of samples that your blur requires.