Image 1: A bad case of shadow acne. (Synthetic and a bit exaggerated)

Shadow acne is caused by the discrete nature of the shadow map. A shadow map is composed of samples, a surface is continuous. Thus, there can be a spot on the surface where the discrete surface is further than the sample. The problem does persist even if you multi sample, but you can sample smarter in ways that can nearly eliminate this at significant cost.

Image 2: A side cutaway of a shadow function and its discrete samples.

The canonical way to solve this is to offset the shadow map slightly so the object no longer self shadows itself. This offset is called a bias. One can use more smart offsets than just a fixed value but a fixed value works quite well and has minimal overhead.

Image 3: Shadow function biased (offset) forward.


What's the difference between geometric surface normal and shading surface normal?

Your existing opinion is correct, though there's one extra detail. The geometric normal is the normal of the actual triangle, based on the vertices' positions (the cross-product of edge vectors, as you said). The shading normal is altered from this, and is used when shading a fragment or ray hit. In the simple case, it's the interpolation of the three normals set at the vertices. This is used for curved objects to hide the polygon boundaries. It might also be modified by a bump map or normal map, to add finer detail to the appearance than the polygon mesh has.


3D clipping is usually done in clipspace coordinates, that means the perspective matrix is used for sending the primitives to clipspace before doing the actual perspective projection. You can visualize the clipspace as a view frustum but with the shape of a box. So it is very fast to clip primitives against a AABB. The math is a little involved, but is all explained in this blog post:

Calculating the gluPerspective matrix and other OpenGL matrix maths | Unspecified Behaviour Blog


How to determine the object-to-world matrix

No! If you dont have this info and must have a extra frame, just assume the reference frame is a identity matrix*. The point of the extra reference frame is just to be able to move the object but if you have no info about how the object should relate to other objects it does not really matter operator can move it where they want. Everything is relative, if theres no relation then no problems.

* choosing identity is like choosing a arbitrary constant to be 1, its numerically easy to work with.


glClear's result can't be seen when using bindless texture

This is clearly a bug in AMD's driver, though it's a very understandable one.

What is probably happening is that the driver does not correctly detect that there is a dependency between the framebuffer operation and the use of the texture. It's easy for an API to tell that you've bound a texture that was the subject of an outstanding render target command, since the API sees exactly what texture you're going to use. It can simply keep track of the texture objects recently used in an FBO, and if it sees you render with one of them bound, it issues whatever dependencies are needed to ensure that the operation is complete before the rendering operation.

It's much harder when you're not binding the texture; you're just passing an arbitrary number to the shader. The API has no idea if that number represents a render target.

To verify that this is the problem, insert a glFinish() between the clear command and the draw command that uses the texture. Odds are good that it will go away (or at least be partially mitigated). A glTextureBarrier() call might also fix it. If you're getting the texture handle after attaching the texture to the FBO, you might also want to get the handle first. Just in case that shakes something loose.

Note that you shouldn't have to do any of these; the specification requires that things work exactly as you expect.

What AMDs driver needs to do is to realize that you're using bindless texturing and emit the appropriate dependency operations when you change framebuffer render targets. That is, it has to assume that any glBindFramebuffer or any calls that modify the FBO that's currently bound will result in any subsequent rendering command reading from previously written textures. Therefore, it must issue a synchronization at the first rendering command that involves a bindless texture operation (if the texture used by the FBO is made resident).

Either that, or it somehow has to figure out which arbitrary integers represent which textures. This is very difficult, since you can pass those handles in a variety of ways. UBOs, for example; you don't want implementations combing through images.


Defining "inside" and "outside" of a 3D space

As no one else seems to want to suggest something, given that you said you wanted to navigate within an arbitrary enclosed volume, perhaps you could use an approach similar to that of the 1995 game Descent.

IIRC the world model of the 'mine' the spacecraft and enemy robots navigated was made up of "cuboid" units/voxels that pieced seamlessly together, some of which had solid walls that then formed the boundaries of the enclosing volume. The source code is freely available online e.g. here .

If, in the likely scenario you can't force the enclosing volume to have quadrilateral facets, you might want to consider "tetrahedral" volumes.


What is the difference and/or relationship between the OpenGL SDK and the OpenGL GPU driver?

I wouldn't really call it OpenGL SDK but OpenGL API. There's a common OpenGL API which is basically a set of functions you can call to control the GPU. This API is hardware agnostic (barring extensions), so you can use it on NVIDIA, ATI, etc. GPU's.

What the OpenGL driver does is to turn these platform agnostic function calls to hardware specific commands that the driver then sends to the GPU. For example when you tell GPU to draw a triangle via OpenGL API, then the driver converts these calls under the hood to specific commands for the specific GPU you are using. This abstraction is important so that you don't have to write different code for every GPU.

To explain things a bit in lower level, the graphics API implementation operates in user mode, while the graphics driver operates in kernel mode. User mode is "protected" mode your program runs in, in the sense that you can't mess up your entire device but only your own app and it's also relatively fast to perform the API function calls in this mode. Kernel mode on the other hand is "unprotected" and performing kernel mode calls requires an expensive mode switch. For OpenGL the IHV's provide both the OpenGL API and driver implementations and thus IHV's can provide optimal switching between the modes. This is in contrast to DirectX where API implementation is provided by Microsoft and the driver by IHV's.


What is the physical interpretation of Normal, Roughness, Albedo and Metalness map?

Those parameters have no direct physical equivalent. They were chosen to be intuitive and to range from 0-1. See the Burley's Physically-Based Shading at Disney for the full rationale and derivation of what is now commonly referred to as "PBR".

The "metalness" parameter is a simple blend between two models - one dielectric BSDF with a coloured diffuse and white specular, and a metallic BSDF without diffuse and a coloured specular component.


What is the benefit of using Half Edge over Winged Edge?

As far as I can tell, the main advantage of half-edge is that traversal can be a bit simpler due to a guarantee of edges having a consistent orientation within each face.

Consider the problem of iterating over all the vertices or edges of a given face, in counterclockwise order. In the half-edge structure, this can be done by starting with an arbitrary half-edge of that face and simply following the "next" pointers until you get back where you started.

In contrast, doing this in a winged-edge structure is a bit annoying, because the edges are oriented arbitrarily; any given edge that you encounter might point either clockwise or counterclockwise relative to the face you're trying to iterate around, so you have to do an extra conditional check at each step to see if you should traverse the current edge forward or backward.

Other kinds of connectivity queries behave similarly: the half-edge version lets you follow links in a consistent sequence while the winged-edge version requires orientation checks at each step.

Whether the conditionals are actually a performance problem for winged-edge will probably depend on other factors. For a "naive" implementation with pointers every which way and data scattered across memory, I'd expect cache miss overhead to swamp any effect of the conditionals. On the other hand, if you have a tightly packed data structure with everything hot in the cache, you might see some effects from the conditionals due to incorrect branch predictions, etc. It's hard to say.

Leaving performance alone, I'd prefer half-edge just because it seems easier to reason about and write correct code for, even if it results in a slight memory overhead.

By the way, there are a couple of other design choices with mesh data structures that often seem to get confused with this one. A commenter mentioned single vs double linking, but naturally you can do either single or double linking with either half-edge or winged-edge. (Although I don't see how single linking with winged-edge would even work, since as mentioned above, you might have to traverse edges either backward or forward in the course of a query.)

Also, there's the question of whether the vertex and face structures store a list of all their edges, or just one (and require you to traverse the edges to find the rest). Having a variable-length list of edges per vertex/face considerably complicates the logic if you want to do it efficiently (i.e. not having a separate heap allocation per vertex/face), but again this is independent of whether the edges are half-edge or winged-edge.


Degenerate case of normal computation in marching cubes

The paper doesn't say to compute the normals for the 8 "cubes" around the cube in question. It is saying to compute the normals for all the vertices of the cube you are testing. To create those normals the algorithm would need density values from the surrounding cubes but not the normals of the surrounding cubes.

The paper labels those vertices v1 to v8. So if v1 is the only vertex whose density value is 1 and the density value for all 7 other vertices is zero then v3,v6,v7 and v8 would have normals of zero (assuming the adjacent cubes were empty). But v1,v2,v4 and v5 would have non-zero normals since v1 is guaranteed to have a non-zero density and at least 1 of x,y or z would be non-zero. Interpolating those normals to the triangle with vertices at e1,e4 and e9 should give a normal pointing roughly at v7.

When reading a paper like this one (ie one that has been used around the world to visualize life saving data, and help countless people) if I think I have found an error I stop and take a long hard look for my mistake instead of looking for the mistake of the authors. Just saying.


How to get clean plate from moving footage?

There are a couple of techniques:

A motion controlled camera - The camera is on some sort of rig connected to a computer. The computer is programmed to move the camera in a particular way. The camera runs through the moves without the actors, generating a clean set of plates. Then it is run again with the actors, usually wearing green or blue costumes or parts of costumes. For example, in "Forest Gump," one character loses his legs in a war. The actor had working legs, as shown early in the movie, and in some later scenes wore blue stockings that were composited out. (They also used other techniques, such as simply hiding his legs under various things in some scenes. Whatever works!)
As you surmise, sometimes a larger scene is filmed with a locked-off camera to get a clean plate, then during editing, they zoom into the scene and pan around in software to create the motion you see on screen. This doesn't work well for anything that involves the camera rotating, but can work for scenes where the camera trucks.
Parts of the scenery that the actors are in front of are blocked off with green screens and recreated in post. I've seen outtakes of a few pieces where they did this. If you have the DVD of "The Others" with Nicole Kidman, the "DVD Extras" show how fog was added to scenes that were filmed without it. It basically involved someone walking around behind her holding a large green board. Then, they recreated the scenery in the computer, added fog to it, and replaced the green screen with the foggy scene. I have a vague recollection of them doing something similar in the HBO mini-series John Adams. This would likely involve cameras that record their motion. The motion is then imported into the computer and used to move the virtual camera around the scene they've recreated.


Is using many texture maps bad for caching?

Just adding to imallett's answer, it is true that increasing the number of accesses to different texture data in a shader will increase pressure on the GPU cache(s), but there are several other factors that can significantly influence the effect. It's also possibly complicated by the fact that, like CPU caches, there may be several layers of cache in a GPU, ie. Texture Unit <= L0 <= L1 <= ..Memory

Avoid aliasing -> Use MIP maps

If you have a scene that has minification of texture data, be it due to perspective or simple scaling and you aren't using MIP mapping, then you will get aliasing. This is not just going to produce visual artefacts; it is very likely to be a performance problem.

As soon as you get aliasing, the address accesses to the texture will become incoherent which will not only end up thrashing the caches but introduce a lot of DRAM "page breaks" (more correctly, row breaks) which can be costly. MIP mapping helps reduce the incoherency.

Texture compression

Perhaps a bit of an obvious option, but if you can use texture compression (e.g. DXTn|ETC*|PVRTC*|etc) targeting from 8bpp down to, say, 2bpp, you can greatly increase the effectiveness of the memory bandwidth/cache by factors of 4x through to 16x.
Now I can't speak for all GPUs, but some texture compression schemes (e.g. those listed above) are so simple to decode in hardware, that the data could stay compressed throughout the entire cache hierarchy and only be decompressed in the texture unit, thus effectively multiplying the size of those caches.

Data size

Obviously, some data, e.g. render targets used as texture data in subsequent renders, can't employ texture compression. Whenever you can, use the smallest pixel format that will do the job, i.e, if 32/16bpp (A)RGB will do, don't use 4x32 float formats!

Sparse access

This is somewhat related to the aliasing example above, but we've seen cases where large render targets are created, but then only very sparsely sampled. Cache lines, be it in CPUs or GPUs, are quite long so if you are only using one pixel in each cache line, you will be wasting transfers.

Also, WRT compressed textures, these achieve compression by effectively sharing data between a local region of texels. If you don't have coherent access then, apart from the memory footprint reduction, the compression probably won't help.

Dependent Texture Reads

Not so much of a cache issue (well, unless the computed accesses are quite incoherent), but texture accesses that aren't directly defined by the UV coordinates supplied with vertices might be slower than those that are directly defined.

Tiled/Morton VS Strided textures

Although I suspect most textures these days will be stored in either a tiled or Morton-like (aka Twiddled/Swizzled) order (or even a combination of both), some textures might still be in scan-line order, which means that rotation of the texture is likely to lead to a significant number of cache misses/page breaks. Unfortunately, I don't really know how to spot if a particular format is arranged in such a way.

(For background reading, try Blinn's The Truth About Texture Mapping. FWIW, taking that a few steps further led to the use of Twiddled-order (i.e. Morton order) textures in at least some early PC hardware).


PhysX is a C++ API and can thus not directly be integrated with the JavaScript-based WebGL.

Depending on your needs, you have the following options:

Use a JavaScript-based physics engine, mostly suitable for 2D use cases.
Use a Game Engine that can export for the web (e.g. Unity 3D) and build your application in there, using full 3D physics capability. Keep in mind that this will probably require your users to install some additional Web Player and is not native WebGL.

Keep in mind that Game Engines may be a viable way for 2D applications too and can make your life a whole lot easier.


A group from UC Santa Barbara published the Siggraph 2015 paper A Machine Learning Approach for Filtering Monte Carlo Noise describing a technique which trains a neural network to select filtering parameters for path tracing. The paper details the structure of the neural network, the filter parameters used and how several secondary features are pre-computed to feed the neural network.


How do I accurately compute coverage of overlapping analytical curves?

There is really no good way of doing this efficiently analytically for all corner cases. Most or all commercial 2D renderers that attempt to do analytic coverage calculation make predictable errors that multisampling methods do not.

A typical problem is two overlapping shapes that share the same edge. The common situation is that alpha channels sum up to a too thick alpha edge that aliases slightly. Or if shapes are differently colored the system confuses what color the background is. This is extremely annoying.

Image 1: The rendering engine confuses the coverage and makes a thin white outline where no outline should be.

Second perfect coverage amounts to box filtering. We can certainly do better. Considering that there are so many special corner cases that would require boolean operations on the shapes to do right, super sampling is still superior. In fact the coverage estimates may be used to concentrate sampling where it's most likely needed.

The situation could be simplified to polygons at sub pixel levels then the discrete analytical solution could be solved. But this at the expense of flexibility. For example its not out of the question that future vector systems might want to allow for variable width blurred lines which pose a problem for analytic solutions, as do other variably colored objects.

How to do it analytically

Image 2: Suppose you have this scene, exploded view on right

Now you can not just do this analytically, each piece separately and then merge the data. Because it results in wrong data. See alpha blending would let the blue shine trough the gaps if you did so.

What you have to so is split the scene up so that each shape eliminates what is under the other:

Image 3: You need to cut the underlying surfaces.

Now if everything is opaque then this is all straight forward. just calculate the area of each piece and multiply that by color and sum them together. Now you can use something like this.

This all breaks down if your individual shapes aren't opaque off course but even that can be done at some level.

Remember:

AA calculation need to be done in linear color space, and converted back to use space.


What is the difference between importance sampling and mutiple importance sampling?

When shading a point on an opaque surface, you need to gather incoming light and weight it with the bidirectional reflectance distribution function (BRDF) of the material.

The naive approach is to distribute samples equally over the hemisphere and probe all directions equally for incoming light. This is called uniform sampling (Fig. 1). While this works in most cases, it can take a long time to converge towards the correct result.

Figure 1: Uniform sampling on the hemisphere

To speed things up, let's consider the properties of the surface in question. A diffuse surface needs to take light from all directions on the hemisphere into account, where light coming from flat angles has less influence. A perfect mirror only needs a single direction according to the law of reflection. Importance sampling takes this into account and distributes samples where light is expected to have a greater influence, judging from the BRDF. For the diffuse surface, a cosine sampling would be used (Fig. 2), resulting in fewer samples at flat angles that presumably have a lower influence. For the perfect mirror, only a single ray is shot because all other directions have zero influence anyway.

Figure 2: Cosine sampling on the hemisphere

Now, the BRDF is not the only thing that can be taken into account. When you have known light sources with strong intensity, it is often desirable to direct more rays towards these light sources, as these directions presumably have a greater impact on the result (i.e. there is more light coming from these directions). The idea to combine various different importance sampling strategies into one is called multiple importance sampling. This technique is not limited to the given example of BRDF + known lights, but can be used with any kind of sensible heuristic you can come up with.

Images were generated using Holger Dammertz' Hammersley Points on the Hemisphere.


Ray-triangle intersection algorithm not intersecting (C++)

The syntax for your dot- and cross-product are a bit strange.
The methods are members of a vector, but still take two arguments. Are they static methods? Or are you really sure they do what you would expect?

Otherwise the only real difference in your implementation and the one you linked that I can see is your test against the parameter t.

The implementation you linked tests for

if(t > EPSILON)

your code tests for

if(t < abs(hitFaces.s))

I don't see in your code how you set the value for hitFaces.s since you fire this ray against a model and don't know jet if it has hit anything.
My understanding would be, that you first test if t is large enough to signal a hit ( > EPSILON ) and then in a second step sort for the smallest t (closest to the ray origin).

In my head you can't get a value for hitFaces.s without calculating this intersection test as you wrote it, and the intersection test needs the hitFaces.s to determine if an intersection was made. But I could be in the wrong here.


How do I convert a hit on an infinite plane to UV coordinates for texturing in a Raytracer?

If your plane has a normal of $\begin{pmatrix}0 & 0 & z\end{pmatrix}^T$, then your computation

vec3 u = vec3( normal.y, -normal.x, 0 ).normalized();
vec3 v = normal.cross( u );

will result in u and v both being $\begin{pmatrix}0 & 0 & 0\end{pmatrix}^T$.

A more general approach would be, for example, to compute the cross product of your normal with each of the three base vectors and use whichever result happens to be the longest as your direction for u:

vec3 computePrimaryTexDir(vec3 normal)
{
    vec3 a = cross(normal, vec3(1, 0, 0));
    vec3 b = cross(normal, vec3(0, 1, 0));

    vec3 max_ab = dot(a, a) < dot(b, b) ? b : a;

    vec3 c = cross(normal, vec3(0, 0, 1));

    return normalize(dot(max_ab, max_ab) < dot(c, c) ? c : max_ab);
}

  …

    vec3 u = computePrimaryTexDir(normal);
    vec3 v = cross(n, u);


Get binding point of sampler/image in shader

The binding value for uniforms of opaque types is, from the perspective of the OpenGL API, the uniform's value. You set it by using glProgramUniform1i.
As such, you query it by querying the value of the uniform with glGetUniformiv.


Could the Tetradic color pattern be expanded to allow /dev/random be "pretty" for cryptographic key purposes?

It depends on how you define “pretty”, but there’s plenty of possibilities. Assembling a grid of shapes as in Identicons would be one example; using the numbers for coordinates and zoom levels on a fractal like the Julia set would be another. Our eyes aren’t good at measuring absolute values of anything—perception is heavily dependent on context and lighting, cf. the dress—so you probably shouldn’t rely on color alone as the identifying factor.


There are two cases I can think of where multiple blurs would be performed in succession on a single image.

First, when performing a large-radius blur, it may reduce the total computation if you first downsample the image (which is a blur) and then perform a smaller-radius blur on the downsampled image.  For example, downsampling an image by 4x and then performing a 10px-wide Gaussian blur on the result would approximate performing a 40px-wide Gaussian blur on the original—but is likely to be significantly faster due to improved locality in sampling and fewer samples taken overall.

The initial downsampling filter is often simply a box (as shown above), but it can also itself be something more sophisticated, such as a triangle or bicubic filter, in order to improve the approximation.

This is a Mitchell-Netravali (cubic) downsample followed by a Gaussian. Interestingly, it turns out that using a Gaussian for the initial downsampling doesn't make such a great approximation if your goal is to use it to produce a larger Gaussian.

An initial downsampling step is also frequently used when implementing visual effects like depth of field and motion blur, for similar reasons.

A second reason to perform multiple Gaussian blurs is to approximate a non-separable filter by blending between various Gaussians of different radii.  This is commonly used in bloom, for example.  The standard bloom effect works by first thresholding to extract bright objects from the image, then creating several blurred copies of the bright objects (usually using the downsample-then-blur technique just discussed), and finally weighting and summing them together.  This allows artists a greater level of control over the final shape and appearance of the bloom.

Here, for example, is a weighted sum of three Gaussians (red line) which produces a shape that's more narrowly peaked and heavier-tailed than a single Gaussian (blue line).  This is a popular sort of configuration to use for bloom, as the combination of a narrow, bright center with a wide, diffuse halo is visually appealing.  But since this kind of filter shape isn't separable, it's cheaper to make it out of a mixture of Gaussians than to try to filter with it directly.

Another variation on this idea is the concept of a diffusion profile used with subsurface scattering for skin rendering. Different blur radii may be used for the red, green, and blue channels to approximate the way different wavelengths of light scatter differently, as in GPU Gems 3 skin shading chapter by Eugene d'Eon and Dave Luebke.  In fact, that paper uses a mixture of seven different Gaussians, with different R, G, and B weights for each, to approximate the complicated non-separable, wavelength-dependent scattering response of human skin.


What you describe is essentially a deep raster. Most modern renderers implement this in fact it is implemented in several ways. There are even sopecific technical positions that rely on this technology in form of lighting, color grading and compositioning artists

The first technique is multipass rendering which is basically just this dumping of render data out of the renderer for fast defered rendering.  All rendering engines on the planet can do this, though all do not implement dumping it out of one render, most do however). What you then do is you integrate these dumped buffers in a compositioning app (Nuke, Fusion, maybe even after effects, you can even do this in photoshop) where you can do final tweaks and changes. This is a very old idea and has been used for more than 30 years. It has certain drawbacks bit overall this means you night be able to pull off even calculation changes that would be too costly to do with a re-render. So color of objects

Many systems also support interactive sessions that cache the computations, in what is essentially a deep raster. This way its possible to recalculate not only color but even the position, decay of individual lights at interactive speeds. The most modern ones of these are frankly quite sophisticated and can also issue the renderer to rerender in background. Animation studios have lots of people working on these to make them better.

You can even use this trick to change camera position for the second stereo camera without recomputing your scene.


Algorithm for proper drawing of cosine function

There is in fact a way to avoid this effect and it is outlined by the NYQUIST sampling theorem - which states that the sampling rate should be at least twice as high as the maximum frequency of the function of interest.
In this case where cos(a * t) is the function of interest, the frequency is a/(2*pi) - therefore your sampling rate should be greater than 2 * a/(2*pi) (in this case you may even want the sampling rate to be even higher so that the cosine function has a nice curve to it).
To get the step size from this sampling rate just use t = 1/(sampling rate)
Hope this helps :)


are you sure you are not re-intersecting the same surface when checking for light occluder ? this is a classical precision issue. There are many way to tackle it:

First, have a good conditioning of the intersection point, by reintersecting a second time: consider your first hit distance d as approximate, create the point E' = Eye + (d-eps)*ray, recompute the intersection from here. (variant: for the first intersection, just use the bounding box).
Second, for grazing angle, depending on your shape representation, storage and computation precision and scales, tessellation, you might have dangerous imprecision here too, and must still ensure you won't reintersect the same surface. If flat or convex or tesselated, you can tag it an non-intersectable for the shadow-ray test.
A simpler trick (more approximate) is to displace the start of the shadow ray by epsilon in the normal direction.


Why do some videos look smoother and others don't on the same monitor

As you have now mentionned that your computer can actually keep up (at 150 fps no less), I suspect you have a case of temporal aliasing.
The problem is that 150 is not a multiple of 60. Let's say we look at one tenth of a second. That's 15 frames generated by your computer but only 6 can be shown on your screen. The frames you will see will probably be something like:
#1 - #3 - #6 - #8 - #11 - #13
Notice how the increment is +2, +3, +2, +3. This will cause animation to not look smooth. There is no steady increment which will let you display 60 frames out of 150.
As I wrote in the comments below, forcing the game to run at "only" 120Hz should help as the screen should show every other frame.
Enabling vsync is even more reliable but it seems your game does not deal well with latency then.
By the way, that's why serious gaming monitors now have adaptive refresh rates. They just show the frames when they're ready and have none of the above issues.


BRDF sampling and evaluation of diffuse vs specular component

So I found a solution on a shader toy.

I choose randomly from sampling diffuse or sampling specular with distribution 50:50. But I calculate PDF of both and average then. If you want to use a different proportion between sample function you just have to make sure that PDFs are combined with the same proportion as choose.


Approximating Geodesics in a half edge DS, how can I refine my mesh to get good approximations

As you pointed out, the problem here is the discretization/subdivision of the mesh. If your mesh was made of quadrilaterals instead of triangles, the obvious subdivision strategy would be to split each quadliteral into four equally sized smaller quadliterals:
$\hspace{2cm}$

$\hspace{2cm}$

For any two points $P_1$ and $P_2$, Dijkstra's algorithm would yield sets of shortest paths between these points $P_1$ and $P_2$. The more you refine the discretization with this subdivision strategy, the more shortest paths you'll find between the two points. However, intuitively it is clear that for every subdivision level $l\in\mathbb{N}$ you could pick one of these shortest paths $p_l$ such that the sequence $(p_l)_{l\in\mathbb{N}}$ converges to the actual geodesic between $P_1$ and $P_2$ (with respect to some norm that has to be specified, for example the supremum norm).
Unfortunately, the same is not true for the standard subdivision strategy of splitting a triangle into four smaller triangles, which you have proven with your example. I believe that, at its core, the problem is that there is no way to reach the center of the triangle with a straight line from each of its edges. This can be achieved by splitting a triangle in each subdivision step into 6 smaller triangles like this:
$\hspace{2cm}$

$\hspace{2cm}$

I do not have a proof that this subdivision is more useful to compute geodesics with Dijkstra's algorithm, but it seems quite likely to me. I would be very interested to see what your results look like with this subdivision strategy! However, no matter what you do, in the end you may or may not end up with sets of shortest paths instead of a single shortest path. In this case you will need some kind of heuristic or additional algorithm to decide which path resembles the true geodesic most closely.


From the error message the problem is in line 24 of your fragment shader:

vec3 transformedNormal = normalize(normalMatrix * normal);

normalMatrix is a mat4 but normal is a vec3.

The solution is to make normalMatrix a mat3 instead.


What's a proper way to clamp dither noise?

TL;DR: 2*1LSB triangular-pdf dithering breaks in edgecases at 0 and 1 due to clamping. A solution is to lerp to a 1bit uniform dither in those edgecases.

I am adding a second answer, seeing as this turned out a bit more complicated than I originally thought. It appears this issue has been a "TODO: needs clamping?" in my code since I switched from normalized to triangular dithering... in 2012. Feels good to finally look at it :) Full code for solution / images used throughout the post: https://www.shadertoy.com/view/llXfzS

First of all, here is the problem we are looking at, when quantizing a signal to 3bits with 2*1LSB triangular-pdf dithering:

 - essentially what hotmultimedia showed.

Increasing contrast, the effect described in the question becomes apparent: The output does not average to black/white in the edgecases (and actually extends well beyond 0/1 before doing so).

Looking at a graph provides a bit more insight:

(grey lines mark 0/1, also in grey is the signal we are trying to output, yellow-line is average of dithered/quantized output, red is the error (signal-average)).

Interestingly, not only is the average output not 0/1 at the limits, it is also not linear (likely due to the triangular pdf of the noise). Looking at the lower end, it makes intuitive sense why the output diverges: As the dithered signal starts to include negative values, the clamping-on-output changes the value of the lower dithered parts of the output (ie the negative values), thereby increasing the value of the average. An illustration appears to be in order (uniform, symmetric 2LSB dither, average still in yellow):

Now, if we just use a 1LSB normalized dither, there are no issues at the edges-cases at all, but then of course we lose the nice properties of triangular dithering (see e.g. this presentation).

A (pragmatic, empirical) solution (hack) then, is to revert to [-0.5;0.5[ uniform dithering for the edgecase:

float dithertri = (rnd.x + rnd.y - 1.0); //note: symmetric, triangular dither, [-1;1[
float dithernorm = rnd.x - 0.5; //note: symmetric, uniform dither [-0.5;0.5[

float sizt_lo = clamp( v/(0.5/7.0), 0.0, 1.0 );
float sizt_hi = 1.0 - clamp( (v-6.5/7.0)/(1.0-6.5/7.0), 0.0, 1.0 );

dither = lerp( dithernorm, dithertri, min(sizt_lo, sizt_hi) );

Which fixes the edgecases while keeping the triangular dithering intact for the remaining range:

So to not answer your question: I do not know if there is a more mathematically solid solution, and am equally keen to know what Masters of Past have done :) Until then, at least we have this horrific hack to keep our code functioning.

EDIT
I should probably cover the workaround-suggestion given in The Question, on simply compressing the signal. Because the average is not linear in the edgecases, simply compressing the input-signal does not produce a perfect result - though it does fix the endpoints:

References

https://www.shadertoy.com/view/llXfzS
http://loopit.dk/rendering_inside.pdf
http://loopit.dk/banding_in_games.pdf
http://32ipi028l5q82yhj72224m8j-wpengine.netdna-ssl.com/wp-content/uploads/2016/03/GdcVdrLottes.pdf


Neither of your suggestions is a solution for crisp 2D graphics at all resolutions. SVGs are great and infinitely scalable, but since you don't know where the pixel boundaries will lie in the final render, you can't get crisp edges at all resolutions. For example, say your GUI is a red box on a green background. If the edge between the red and the green falls in the middle of a pixel, and the SVG renderer does anti-aliasing, you'll get a row or column of half-red, half-green pixels, which will look like mush. If the renderer doesn't anti-alias, you instead get the edge in the wrong place, which might be fine for a single rectangle, but for a repeating pattern each instance of the pattern might come out a different width, which is just as noticeable.

Downscaling a bitmap to the display resolution is just as bad. Again, imagine your green-red edge is on a pixel boundary that will not be a pixel boundary in the downscaled bitmap. You're again going to get a row or column of pixels that are a mushy red-green mix. The lowest resolutions are the most important to have crisp edges, but they're going to be the worst-affected.

You can make either one of these techniques work somewhat, if you can accept a restriction: instead of handling any resolution, you're willing to handle a set of resolutions, ideally powers of two apart (e.g. icon sizes of 512x512, 256x256, 128x128, etc.) When you're designing your UI, draw a large bitmap (512x512) or a vector image, but use guides in your editor to show the minimum size (e.g. 64x64). Make sure all your important edges line up with those guides, but you can still use curves, gradients etc. That way, you know you won't get the blurry mush at low resolutions, but you can still get the benefit of smooth diagonals and curves at high resolutions. This is a quick way to design scalable UI.

Mobile OSes (which are designed for a range of screen dpis) use a similar technique, but with two differences. First, they "bucket" the screen's actual dpi to the nearest "standard" dpi. This means a UI might show slightly larger or smaller on different screens, but you know exactly how many pixels will be in each icon or widget. To make this work out, they don't limit the resolutions to powers of two, but have some intermediate ones as well.

Second, they let designers specify a different bitmap for each resolution. This means that you can make a higher-detail design for a higher-dpi screen, without it being too busy on a low-dpi screen. (For example, you can use a fine checkerboard or ruler pattern, and have it show up correctly at each resolution.) You could consider doing this too, but it makes designing and managing the bitmap assets a little more time-consuming.

If your buckets are powers of two apart, you can use mipmaps to store the images (instead of using the built-in mipmap generation), whether you're creating different images for each resolution or just downscaling. That way you avoid the runtime memory bandwidth cost of downscaling.


Flood filling is not a very GPU-friendly operation to begin with, and the algorithm described in the paper is not a very good way to go about flood-filling on the GPU either. It might be possible to make it happen with some heroic efforts and compute shaders, but if flood-filling is what you want to do, I would use a different approach altogether. For example this answer describes how to do flood-filling with a multi-pass approach where each pass extends the filled region by 1 pixel in all directions. It will take a lot of passes to fill a large region. You could probably use some variant of jump flooding for a more efficient method.

However, if you don't actually care about flood filling but are just looking to do exercises to learn GPU programming, I'd recommend trying things like: blur filters, outlining, color transforms, image warping, and so on as those will be a lot more straightforward than any kind of flood fill.


In what scenarios and how does GL_ARB_buffer_storage help performance?

The marquee feature of buffer storage is not immutability of the allocation itself, but instead is a feature you couldn't have without immutable allocation: persistently mapped buffers.

Pre-buffer_storage, you could not use a buffer while it was mapped. This is done to allow implementations the freedom to play games with mapping behind your back. For example, it's legal for glMapBufferRange to not actually return a mapped pointer; it could just return some CPU-accessible memory which, at unmap time, will get copied into the actual buffer. To permit such freedoms, mapping and unmapping a buffer is something you have to do semi-frequently.

Furthermore, pre-buffer_storage, it is 100% legal to call glBufferData (note the lack of Sub) on that buffer again. This will cause the old data storage to be discarded and new storage allocated. So you couldn't keep a buffer mapped if you tried to reallocate its storage.

Also, implementations couldn't really know where to allocate storage for GPUs with multiple kinds of memory, so they would tend to move the buffer's allocation around based on how you use it. If you frequently upload to it, they'll eventually put it into CPU accessible memory to speed up such operations. And so forth. But this is all done after allocating storage, and it's done based on how you use the buffer. So having the buffer be mapped would prevent such shuffling around of data.

Making a buffer's storage immutable solves these problems. You cannot reallocate an immutable buffer's storage, so there's no problem with keeping it mapped forever. glBufferStorage takes usage flags which unlike glBufferData's hints, explicitly forbid you from using the memory in certain ways. So the implementation knows exactly where to put the storage and will not need to move it around after the fact. So there's no problem with persistently mapped storage.

And I imagine that, for a console emulator, having persistently mapped storage would be very useful. After all, programs written for such consoles touch memory directly at any time, the same memory that the GPU can address. So storing it in a buffer and keeping that memory mapped all the time probably works out better than uploading pieces of it, or trying to figure out which pieces should be uploaded, or whatever.


How to determine RGB values given a display spectral response curve?

I'll try my best to give an overview of the subject, and english is not my first language so feel free to ask more detailed information in comments etc.

Here are some terms, light is an electro magnetic radiation. When the power of this radiation falls between a certain wave length, it becomes visible for human beings. S is the function that takes wave length value as input and gives a power value as output and is called a s(pectral) p(ower) d(istribution).
Color on the other hand means different things to different people. We can define color based on our perception of it, or based on the measured spectral power distribution, based on the average physical capacity of human eye etc.
The thing is spectral distributions change over time constantly, but the color does not change at the same speed. This means that variations in the spectral distribution does not map to color one to one, which shows that there must be a transformation happening between what we perceive as color and what we measure as spectral distribution.
Now the needs for an accurate and standardised representation of color was quite important for daily life (think of colors around road signs, lamps, etc), the CIE defined its trichomatic system for this purpose. The main idea behind the trichromatic system was that given a carefully chosen three reference spectral distribution, we can represent all of the perceivable colors by mixing these three spds.
Notice that I did two things over there. I passed from mixing spds, which should produce an spd at the end, to color values which are generally represented with trichromatic system, and I added perceived. The last one is to signal that this is an experimental procedure, that is people did a lot of experiments to come up to this idea of mixing reference spds.
How do we arrive from mixing spds which should produce an spd to a color value in rgb ? Simple by summing all the power values of the spd resulting from the mix.
The mix is $R(\lambda)S(\lambda)$ part where R is the reference spd and S is any given spd. $\int$ indicates that we are summing the resulting multiplication.
$\lambda$ is the wave length.
A great reference on the subject is McCluney 1994, Introduction to Radiometry, chapter 11. Read through 11.2 and then 11.4, that should clarify anything and everything with respect to spds and their transformation to trichromatic values


Another very simple, easy to implement, algorithm to generate a Poisson-disk/blue noise point distribution is the following one [ 1 ]:

Decide a radius r, and therefore the expected number N of samples per area (use the formula for maximum packing density of k disks distributed on a toroidal domain see [ 2 ])
Generate 10~20 x N samples in your area using a plain unbiased montecarlo sampling
Until all samples respect poisson disk condition:

randomly choose a sample p
remove all the samples at distance lower than r from p

Eventually step 3 can be furtherly iterated a small number of times choosing the sample with minimal number of surviving samples within distance r.

You can find an implementation of this algorithm in MeshLab.

References

[ 1 ] Efficient and Flexible Sampling with Blue Noise Properties of Triangular Meshes
Massimiliano Corsini, Paolo Cignoni, Roberto Scopigno
IEEE Transaction on Visualization and Computer Graphics, Volume 18, Number 6, page 914--924 - 2012

[ 2 ] A comparison of methods for generating Poisson disk distributions
 Ares Lagae, Philip Dutré,
Computer Graphics Forum vol:27 issue:1 , 2008


Appropriate filter steps for resolving a droplet pixel

First simple thresholding is usually not a good idea. However, if you want to do that, I suggest to also consider other color spaces. In your case, HSV might be a better choice; You can convert your image into HSV-image then you can define a range on Hue channel to select yellow.

Also, take a look at this article; it is about implementing Adaptive Thresholding.


Is anti-aliasing color processing correct in this image?

The anti-aliasing rendering is correct.
The reference gray ramp (top and bottom) is linear in sRGB color, not in brightness.
The second gray ramp made with anti-aliasing is linear in brightness. The rectangle color (black in color sRGB space) is mapped to linear space (remains black), then anti-aliasing is computed, and the resulting image is converted back to sRGB. It is thus not linear in sRGB value anymore.
The RGB values given by a screen color picker are in  sRGB color space, and the color values of a screen capture are in the sRGB color space.
SVG and nearly all browser graphic library (skia and cairo) are doing it wrong. They compute anti-aliasing over sRGB instead of linear color. They thus don't convert back from linear to sRGB. The same mistreatment  is also visible on Windows, MacOs and iOS.
The filled rectangle gray ramp doesn't have this problem because the conversion from sRGB to linear and its inverse cancel out.


Normal 2D photo to VR-compatible spherical photo

There are a few ways you can do it. Basically, you need to get your photo into a format that's suitable for using as an environment map. There are a number of such projections available. The most common ones I've run into for 360° video are:

Cube maps - 6 images that show what the camera would see if it were looking forward, backward, left, right, up and down from the same position.
Equirectangular - the 2nd image you show above is this type of photo. It must be exactly twice as wide as it is tall. The rows and columns of pixels then correspond to the latitude and longitude of the sphere you map it onto.
Dual hemispherical - this is when you take a camera (real or virtual) and make an image with a 180° field of view. There are 2 - one for the front and one for the back.

Given the image you have, there are a few ways you could create something useful from it. You could crop it or extend it so that it has a 2x1 aspect ratio and use it as-is. You'll get a weird seam where the left and right meet, but if you are handy with photo editing tools, you can probably modify it to look reasonable. Note that this will give you the correct image for an equirect projection, but it may or may not look how you want. That purple band would never be visible all at once. It would wrap around the entire field of view, for example.

Another option is to make a cube map out of it. You could crop what you have into 3 equal squares for, say, front, left, and right. Then you'd need to generate the other 3 yourself. Generating a star field isn't too difficult either in code or using a photo-editing/compositing tool. Or if you know any artists, perhaps they could help you.


How to estimate what GPU (2D/3D/none at all?) is necessary?

There are really two things you need to know about your application:

How many pixels do you need to push per second?
How many cycles do you need to spend on each?

For the first, you need to sum the size of your output framebuffer and all the intermediates to count the number of pixels written in each frame. Make sure to include overdraw: if you draw a full-screen background, and then draw a foreground that covers half of the frame, you need to multiply by $1.5$. Then just divide that by your frame time (e.g. 33 ms if you need to reach 30 fps) to count the pixels per second.

To count cycles, you need to know what size of program will be run on each pixel. For a transformation or colour correction, it might just be a couple of multiply-adds. Somebody who would write the shader code (or the Neon code in the no-GPU case) should be able to work this out for you. It'll be easier if you have a prototype running on a desktop GPU already.

There are also two important properties of the hardware to consider. First is the pixel fill rate, which you've already seen. This tends to be limited by the memory bandwidth of the system-on-chip instead of by the CPU or GPU itself. If you're buying GPU IP and integrating it into the chip itself, you'll need to talk to the vendor's application support people to find out what your design can achieve. If you're buying a whole system-on-chip, just use the value from the spec sheet.

Second is your cycle budget. This depends on the GPU or FPU itself, not on the memory bandwidth, so it's easier to calculate. You just need to talk to the silicon vendor to determine this. For a GPU it'll be the number of cores times the clock speed divided by the instruction cycles per clock - but more realistically, look at a synthetic benchmark for this chip to see its megaflops.

Comparing your number of pixels per second with what the system can achieve is easy, but you also need to multiply this by the cycles per pixel you want to spend, to get the total number of cycles per second you need, and compare this with what the compute core offers.

Don't forget you also need to weigh the difference in BOM cost and software complexity against the risk and cost of a board respin, and multiply your estimate by some fudge factor appropriate to your project risk. We can't help you with that part here :-)


Tiled raytracing is a way of speeding up rendering.  Instead of rendering pixels on a row-by-row basis down the entire image, you render them in 2D groups (say, 32x32 tiles).  Since adjacent rays tend to intersect the same objects, this improves the cache hit rate at the CPU level.

At a smaller scale, tiling at the 2x2 level lets you use SSE/AVX/etc to compute four rays with a single CPU instruction, further speeding things up.


phong equation of illumination specular component

Assuming that the classic Phong model is desired here, the dot product that goes into the specular calculation should be R·L, rather than N·H (which was introduced by Blinn). That is, Phong calculates specular using the angle between the reflected eye vector and the light vector.

In the diagram you posted, these vectors are shown and the angle is given as 15°. So, R·L = cos(15°) = 0.96 according to the table below the diagram.


What are the use cases for multiple render passes?

Instead of listing specific use cases, I will explain the technical limitations of a render pass structure which would prevent someone from putting all of their rendering within it. The use cases naturally fall out from those limitations.
The most important being this: you cannot arbitrarily access any of the images attached to the render pass throughout the entire render pass. It doesn't matter if a particular subpass doesn't use the image as a render target throughout that subpass; you cannot bind it as a texture/image and read from/write to it in a shader. This covers the entirety of the render pass from start to end.
Now, there are input attachments. These are uses of attachments that allow you to read from them in the shader. However, they aren't textures (though they are descriptors); they have more limitations than textures. The most important limitation being that you can only read from a pixel that corresponds to the current fragment's location (and also, you can only read them from a fragment shader). You can't fetch data arbitrarily from the image. A particular FS invocation can only read the specific pixel that was written previously at this FS invocation's location.
Now, you can do some things within that limitation. You can do programmatic blending, for example. You can do deferred lighting passes, since any particular FS invocation only needs to read gbuffer data from the specific pixel for that invocation's location.
But the main point is this: any effect which requires arbitrarily reading from an image that was rendered to must be in a different render pass from the render pass that rendered that image.


First, I cannot imagine how could a vertex have a normal,...

Welcome to the exciting field of computer graphics. The first thing that might lead to your confusion is to think about a vertex too much as a point in space in a geometrical way. ---> A 3d point can't have normal since it does not have a face.
But the way you should think about it is more like a checkpoint that contains all information of an arbitrary 3d curve at a certain position.

  ... and second, when calculate the reflection lights, shouldn't I use the normal of a face?

What happens if you want to render a perfect sphere that can only be represented by a limited set of triangles? It will look like the one in the middle in this question. The flat surfaces are clearly visible, but that is not what you want. You can improve this by drastically increasing the triangle count, but that comes with a huge performance impact and memory footprint.

The solution here is using vertex normals. To understand, why this is a solution, you need to know a little bit about how the render pipeline works. The most important parts for your understanding are vertex processing, rasterization, and the pixel/fragment shader.

Vertex processing is the first programable (in modern APIs) step in the pipeline and is mostly used to reposition your vertices so that your 3d world is displayed correctly on a 2d display. Therefore you apply some mathematical corrections (perspective for example) to your vertex positions. You can do a lot more, but that is not important here. What is important is, that the vertex processor does not know to which triangle a vertex belongs. He sees it in total isolation. Therefore, he can not calculate any face normals, so you have to provide it in some way, for example by adding it as vertex attribute.

After some optional pipeline steps, you get to the rasterizer. Its job is to turn your triangles into actual pixels on your screen. The information he gets are three vertices that form a triangle. Now here is, where the magic happens. For each pixel, he interpolates the properties of the three vertices and passes them to the final step, the pixel shader (also programable in modern APIs). So if your vertices had normals, you will have an interpolated normal for each pixel of your triangle. This enables you to render perfectly smooth surfaces (or at least fake them) since you can perform the lighting calculation for each pixel with individual normals in the pixel shader.

This is why you need vertex normals. The vertices are checkpoints that hold information about your surfaces at specific points. You connect these points to triangles. The information in between is interpolated by the rasterizer. So you essentially describe a whole triangle-shaped field of normals with its three edge values.

However, even if you want to render flat surfaces, you might still need vertex normals. As I said before, the vertex processor does not know anything about triangles, so you can not calculate any surface normals there. So you have to pass them in somehow. There are other ways, but attaching the normals to the vertices is usually the best and easiest solution.

Additional note: You do not necessarily need to perform the lighting calculation in the pixel shader. You can also do this in the vertex stage. So you need vertex normals again ;). This is called Gouraud Shading and gives you a performance boost since you do not need to perform the light calculations per pixel. However, the visual quality of curved surfaces suffers from that. For flat surfaces, you should not see any differences.


How to scale primitive shapes [in Blender] so that they are within a [unit] sphere?

Yes, there is a fairly general algorithm to calculate this scaling factor, which works for all shapes with a known parametric representation. First, substitute the parametric equation of the shape (e.g. a torus, cylinder or cone) into the implicit equation of the sphere, $x^2 + y^2 + z^2 = S^2$. Then, solve this equation for the radius of the sphere. The result will be one or more intervals for values of $S$ for which the shape intersects the sphere with radius $S$. The largest value of $S$ is the radius of the shape's bounding sphere.

Example (torus): Substituting the parametric equation of the torus centered at the origin with major radius $R$ and minor radius $r$ into the implicit equation of the sphere yields $2Rr \cos(\phi) + R^2 + r^2 - S^2 = 0$, which is equivalent to $S\in[R-r,R+r]$. So if you want a torus to fit into a unit sphere, you need to choose the major radius and minor radius such that $R+r\le 1$.


Probability density function while using spherical coordinates

Normally $sin(\theta)$ isn't written as part of the pdf, because it's an artifact from using double integral for spherical integration. The double integral does the integration over a rectangular domain (in you case of size $2\pi$ x $\pi/2$), while you want integration over a sphere/hemisphere, so $sin(\theta)$ is added there as a weight function to transform rectangular to spherical integration.

A common notation is to use integration over solid angle with single integral and differential solid angle $d\omega$ denoting 2D integration over sphere, and you won't see the $sin(\theta)$ term in there.

To clarify, the notation with single integral has constant infinitesimally small solid angle $d\omega$ over the sphere. However, for the double integral over $\theta$ and $\phi$ the solid angle isn't constant (think of sphere tessellation) but a function of $sin(\theta)$. In case of Monte Carlo integration and even distribution of samples over the hemisphere, each sample represents solid angle of $2\pi/N$ steradians and there's no $sin(\theta)$ solid angle weighting required.


I'd recommend using OpenVDB. It's quite battle-tested.

Use meshToVolume to voxelize the mesh. Then use volumeToMesh to produce a high-quality quad mesh.

meshToVolume voxelizes the mesh into a SDF. volumeToMesh seems to use dual contouring / SurfaceNets.


Interlacing vertex buffer data with extra efficiency

Is it even possible to have opengl assume that the Z is always 0?

Yes it is. Just set the component count to 2 in glVertexAttribPointer and the other 2 components (z and w) will be auto filled with 0 and 1 resp.

  Is it possible to reuse the color like to hat?

No it is not. Opengl (and most other graphics apis) require that each vertex is referenced by only a single index.


Getting sRGB values outside [0, 1] is expected and normal when using spectral rendering.

The sRGB gamut only covers a triangle in the middle of the CIE chromaticity space:

(diagram from Wikipedia)

The big "horseshoe" shape is the set of all physically possible chromaticities (CIE xy coordinates) from any possible spectrum. So your xy coordinates should be expected to land somewhere in the horseshoe...but only the ones that land inside the triangle are representable as sRGB! The others are "too saturated" for sRGB, and will have to be clamped or remapped somehow. (Tone mapping per se doesn't fix this, as it's concerned with luminance rather than chromaticity.)

There are a variety of approaches to deal with out-of-gamut colors depending on how sophisticated you want to be:

You can of course simply clamp the sRGB values, but this may change the hue and/or the luminance of the color in the process.
To preserve both hue and luminance, you can edit the xy coordinates before calculating scaledXyz. If the initial xy coordinates are outside the sRGB triangle, find the line segment from the xy coordinates to the white point (e.g. D65); then find the intersection of that line segement with the sRGB triangle. That's the closest sRGB point that has the same luminance and hue as the original point, but lower saturation. (The vertices of the sRGB triangle can be obtained by converting RGB(1, 0, 0), RGB(0, 1, 0), and RGB(0, 0, 1) to CIE xy space.)
Alternatively, rather than clamping (which throws away any detail in the saturation outside the sRGB gamut), you could try doing a linear or non-linear remapping of saturation values in the image, similar to how you do for luminance. By "saturation" here I mean distance from the white point. The idea would be to squish the whole image toward the white point so as to bring everything within the gamut. But that may affect the apparent saturation of in-gamut colors in visually undesirable ways.


Multiple samples per pixel and texture blurring

In signal processing, it is well-understood that you cannot accurately reproduce an analog signal of higher frequency than half the frequency of your digital sampling rate. That's just how the math works. Various aspects of rendering are just forms of signal processing, so this applies here too.

So you're going to get some kind of artifact. You have a choice: aliasing or noise (aka: blur). You're always going to have some of these. Aliasing artifacts are usually very noticeable and considered quite distracting. Human vision tends to focus on motion, and aliasing artifacts almost always create motion where none existed (especially when animating). Human vision is generally much more tolerant of noise, as noise patterns don't appear to move nearly as much. Plus, our eyes have their own anti-aliasing filters that impart some noise to what we see, so we're somewhat used to it.

Broadly speaking, you deal with it by not dealing with it, because it's preferable to the alternative. Yes, you can reduce the effect of noise with better anti-aliasing mechanisms, but broadly speaking, you're still going to have noise.


Unbound descriptor tables and descriptors for fat root signatures

Everything is allowed to be left unbound / invalid / uninitialized as long as the shader doesn't try to look at it. However, if the shader does try to look at it, it could trigger a GPU crash / TDR.
In practice, it may be a good idea to explicitly clear out descriptors that you know aren't going to be valid: either setting them to a null descriptor, or pointing to a dummy resource. Loading/sampling from a null SRV/UAV descriptor in a descriptor table is defined to just return 0 and not crash. Likewise, stores to a null UAV descriptor are dropped on the floor without crashing.
However, this guarantee doesn't apply to descriptors directly in the root signature; trying to load/sample from a null root descriptor can crash the GPU. Likewise trying to read from a null descriptor table can crash. See the D3D12 functional spec for more.


Microfacet shading for diffuse materials

You can combine Oren-Nayar with GGX, if your normalize the result. A BRDF is defined by two properties: Helmholtz reciprocity and energy conservation.

$f(l_i, l,_o) = f(l_o, l_i)$

$f(l_i, l_o) \leq 1$

Your Oren-Nayar is the diffuse part $f_d(l_i, l_o)$ and your GGX is your specular part $f_s(l_i, l_o)$. If both are energy conserving, then both are at most $1$, but combined they may be $\geq 1$ and here lies your problem. Oren-Nayar has been combined with specular terms before (see for example The Rendering of Ryze from Crytek, you should have a look at how others did it. If you combine them, keep in mind that you need two different roughness values. The ones of Oren-Nayar and GGX don't match (which by the looks of it you know already) but neither can you convert from one to the other via an algorithm.

A better solution imo is to use a different diffuse term. Have a look at the papers Physically Based Shading at Disney by Brent Burley and PBR diffuse Lighting for GGX+Smith Microsurfaces by Earl Hammon Jr.. Physically Based Shading discusses several diffuse lighting BRDFs, including Oren-Nayar and then goes on to introduce an own diffuse BRDF which can use the same roughness value as GGX (be careful, as the roughness here is squared before using, iirc, because that was easier to use for artists). This resulting BRDF is not energy conserving, which was accepted by the author, since

  This seems to provide a reasonable match to the MERL data and was also found to be artistically pleasing.

A solution for this was introduced by Sebastien Lagarde and Charles de Rousiers in Moving Frostbite to Physocally Based Rendering, they renormalized the entire BRDF.

PBR diffuse Lighting for GGX+Smith Microsurfaces by Earl Hammon takes a different approach, he calculates an approximation like Oren and Nayar did, only he did it for GGX NDF and Smith shadowing/masking. Both the Disney Diffuse and Hammon's Diffuse BRDF can easily be combined with the GGX specular term and are visually more pleasing than Lambert.


Why is cos() in the denominator in the reflectance equation and in the numerator in the shading equation?

The first is a definition of radiance in terms of the flux - it's a the derivative of flux with respect to both area and direction. It's simply a relation between $\Phi$ and $L$. It simply says that if you integrate radiance over the area of some surface, and for each point you integrate over all possible directions, then you will get the flux arriving/leaving that surface.
The second part comes from the restriction of the rendering equation to a single point light source.
Remember that:
$$L(x,\omega_o) = L_e(x,\omega_o) + \int_{\Omega}{f(\omega_o,x,\omega_i)L_i(x,\omega_i)\cos\theta_i \,d\omega_i}$$
Where $x$ is a point on some surface, and $L(x,\omega_o)$ gives you the radiance incoming from $x$ in the direction $\omega_o$. This radiance is the sum of emitted radiance from $x$ in direction $\omega_o$: $L_e(x,\omega_o)$, and the scattered radiance by $x$ in direction $\omega_o$. The integral is necessary, since light may arrive from any direction at $x$, so you need to integrate over all directions. $f$ is the brdf, which really tells you how much of the light arriving from direction $\omega_i$ is reflected towards $\omega_o$ at $x$. $L_i(x,\omega_i)$ is the incoming radiance from direction $\omega_i$ at $x$, and the cosine is there to account for Lambert's law. What happens when you evaluate only direct lighting from a point light is that the integral is over a domain that is a single point, so you get only the integrand part, which you have in your code. Note that this is just an intuitive explanation, there are various issues with radiance not being defined for point lights.


Resulting Probabilty Density in Path Tracer for paths using Next Event Estimation

I don't have any experience with Gradient Domain Path Tracing, but here are my thoughts:

There seems to be a different problem

If you look carefully at the little spikes of distortion in the final image, you will see that they are all lit from the same direction - on their top left side at a consistent 45 degrees. The sphere also appears to be lit from this angle, rather than from above by the light source.

This is unlikely to be explained by an incorrect probability estimation for a path. I would expect there to be a different problem with the code, that these distortions are hinting at.

I will therefore address these two separate points:

You want to know how to calculate the Probability Density of a path when using Next Event Estimation.
There is evidence of some problem unrelated to this.

I'll also review the code for non-essential points - but I'll leave this until after the essentials.

Probability Density of a path when using Next Event Estimation

Looking at the paper on which the code you are following was based it seems that the novel shift mapping described in section 5.2 is defined in terms of the reflective properties of the surfaces found at the vertices of the path. I must emphasise that I don't have a full understanding of this, but it suggests that Next Event Estimation may not require a change in this approach, as the surfaces encountered will be the same. Hopefully once the other problems are cleared up it will be easier to judge whether the image looks correct.

Note that section 5.2 of the paper already mentions (just below Figure 10) that they take into account sampling the emitter "either using BSDF or area sampling".

The difference with Next Event Estimation is that the area sampling happens at every vertex of the path, but it isn't obvious to me that this should cause a problem.

The fact that your scene uses only diffuse surfaces means that the offset path should in most cases rejoin the base path at the second vertex, so you would only need to recalculate the area sampling for the first vertex of the offset path.

The cause of the incorrect lighting direction

In reading through the code to familiarise myself with how it works, I noticed that NLdotL is calculated, but then not used. A text search revealed that the only other occurrence has a different case: nldotl. Here are the two variables in context (first and ninth lines of this excerpt):

float NLdotL = Math.Abs( Vector3.Dot( NL, -L ) );
float NdotL = Vector3.Dot( ray.N, L );
if (NdotL > 0)
{
    Ray r = new Ray( I + L * EPSILON, L, dist - 2 * EPSILON ); //make it a tiny bit shorter otherwise I risk to hit my starting and destination point
    Scene.Intersect( r );
    if (r.objIdx == -1) //no occlusion towards the light
    {
        float solidAngle= (nldotl * light.getArea()) / (dist * dist);
        E += T * (NdotL) * solidAngle * BRDF * light.emission;
    }
}

Since nldotl is not defined, the result of the code is undefined behaviour. In practice, the program is likely to act as if nldotl is zero, or for some compilers perhaps a constant arbitrary value, or even a different arbitrary value at each iteration. For your particular compiler it appears to be a constant value, and I strongly suspect this is the cause of the distinctive alignment of lighting angle on all the speckles and the sphere. If there is also another contributing problem it will be easier to analyse it in a separate question once this initial problem has been fixed.

It may be worth considering using a compiler and/or flag setting that gives an error or at the very least a warning for undefined variables, as this type of mistake is both very easy to make, and very easy to overlook later.

Additional contribution of light source

There appears to be another problem that will cause results to be incorrect in a more subtle way, with no obvious distortion. Due to Next Event Estimation, the light source is contributing to each step along the path. This means it should not make a contribution if the path itself hits the light source directly. Otherwise, for that path the light source will be contributing twice. You can correct this by changing:

if (material.emissive) //case of a light
{
    E += material.diffuse;
    break;
}

to:

if (material.emissive) //case of a light
{
    break;
}

This will give a zero contribution from the intersection with the light.

Note that since I can only see this one function, I can't guess whether this will also make the light appear black in the image. You may or may not need to adjust for rays that start at the camera and hit the light directly.

Code review

Doubly finite rays

I'm used to defining a ray as a half infinite line segment - having a starting point but no ending point. I notice that this code gives a ray both a starting point and a length. The only place I can see a reason for this is when testing a shadow ray against the light source: the code checks that there are no intersections on the way to the light, so intersections behind the light (or on the light itself) must be excluded. In all other places, the ray is defined with a pseudo-infinite length (1e34f).

The following suggestion won't affect the correctness of your code, but it may be more readable and avoid you having to work around the need for infinity and having to account for epsilon twice.

If the ray is simply a starting point and a direction, then shadow rays can simply check that the first intersection is the light, rather than checking that there is no intersection. For example, by replacing:

if (r.objIdx == -1) //no occlusion towards the light

with:

if (r.objIdx == LIGHT) // There is no object intersected before the light

Here I use LIGHT as a placeholder for the id of the light source, since that part of the code is not included in the question.

This will always give false if the light is occluded by a nearer object.
This will always give true if the ray hits the light before any more distant objects.

This is therefore equivalent to the current code, but does not require the ray to store a length.

Lights that don't reflect

The code currently models lights and surfaces separately. This means that if an object is a light then it is only emissive, and reflects no light from other objects.

This causes a negligible difference in the example scene in the question, which has a single bright light. However, if used with a number of dimmer lights, it would be more noticeable that they do not light each other. In many cases the difference will not be noticeable, so this is not a problem if it works for the scenes you wish to render.


Elaboration on temporal solve:

I don't have much concrete info for you, but I'm drawing from the idea of "temporal anti aliasing".

Basically, if a camera was stationary, you could average pixel values over the last N frames, possibly using harmonic mean or something else like that to help filter out spikes.  The result would be a cleaner, less noisy, more correct image.

But not all cameras (or objects!) are stationary, so what then?  Well, if you have some way of identifying where a pixel this frame matches a pixel on the previous N frames, you could average them in the same way.  If a current pixel has no matching previous pixel (due to something previously occluding becoming visible) you just show the raw current value.

Games use this for antialiasing by simulating super sampling over time, but they have the benefit of per pixel motion vectors as well as the current and previous camera matrix, so it's a lot harder in your situation!


Why does checkerboard pattern on a computer screen appear with a yellowish tint?

Because your monitor is not properly calibrated.

On my screen at home the top and bottom parts have the same hue. At my office though, the top part tends to looks a bit yellow compared to the bottom part that looks more red.

The difference: my screen at home was from a series that was decently calibrated out of the factory, and on top of that I did calibrate it properly with a color calibration tool. That was quite some years ago and its colors have probably degraded since, but it still makes a difference.

A poorly calibrated monitor will not display exactly the color intensity requested for R, G and B, resulting in differences between the color that should be displayed and the one actually displayed. It can even vary depending the area of the screen (many consumer level LCD screens tend to have more light leaking near the edges for example).

The image you included is a good example to highlight a calibration issue, because it allows to compare directly the average color at 50% intensity (combining two halves at 0% and 100% intensity) with the color displayed when requesting 50% intensity. Moreover, the gradient allows to see if the hue is constant from 0% to 100% intensity.


Measure how long a rendered frame takes to arrive on a client from the cloud

Try putting a timestamp over some key frames as the last step on your render pipeline. Then compare it to the local systems time. If both clocks are correct you should get a delay in ms (using unix rime-stamps).

Another idea to measure the round-trip-delay (might be important for UX reasons):
input to server,
server render,
result from to server on your screen
is to have a bot give an input than measure the time till the result is back


How are Physically Based Rendering Methods Researched?

I guess physicists, and before that term was used philosophers, have always been thinking about the nature of light, color and shadow. Knowledge about those matters is not purely rooted in people trying to create the ultimate rendering technique for CGI. In Wikipedia there are whole categories that show formulas and models that were invented to describe attributes of lighting. What programmers do is to try to combine and fit those models into a probably more simple model that can be implemented in some way.
With the increase in computing power, those models align more and more with the models that were created by physicists to describe lighting in the real world.
If you look into recent (by that i mean 5 years back) papers that describe physically based rendering techniques, you will see that almost all of them base their algorithms on formulas and units of measure that are somehow used to describe real lighting.


Is there a method to do ray marching style modulus repeat with raytracing?

I've tried to figure this out as well, but I don't think there is a solution that is just as performant as tracing one sphere.  Basically you would want to cast one ray to hit a infinite set of repeating spheres with no marching iteration.  That would be awesome but is not possible:  Let me explain.

Ray-tracing is basically finding the analytical solution to this equation:
$$
f(\vec{r(t)}) = 0 \ for \ t
$$

Where $f(\vec{p})$ is the implicit surface function (which gives the distance the given point $\vec{p}$ is from the surface) and $\vec{r(t)}$ is the parametric ray equation ($\vec{r(t)} = \vec{origin} + \vec{direction}*t$).  $t$ would represent the distance along the ray the intersection happens.

In order to ray-trace domain manipulations (like repeating space), these domain manipulations need to be present in the implicit surface function $f(\vec{p})$ while finding an analytical solution.  Let's take a 3D repeating sphere for instance.
$$
f'(\vec{p}) = \|\vec{p}\bmod \vec{d}\| - r
$$
(where $\vec{d}$ is a constant representing the grid size and $r$ is the radius)

In order to solve $f'(\vec{r(t)}) = 0$ for $t$ we would need the inverse of the modulo operator which doesn't really exist : http://mathforum.org/library/drmath/view/51619.html.  That means there is no analytical solution to this equation, but there is a numerical one. Hence Ray-marching.

This is true for a lot of cool distance functions you see on shadertoy.com, the analytical solution does not exist for these surfaces so you cannot ray-trace.


Is there a term for mapping between two arbitrary quadrilaterals without using homography / perspective projection?

Yes, this is called bilinear interpolation or a bilinear mapping. The term "bilinear" crops up more commonly in the context of image resizing or resampling, but it's the same basic idea for transforming quadrilaterals—it's just that the interpolation is applied to the vertex positions of the quad, instead of the colors of pixels in an image.

One paper that briefly discusses bilinear mappings of quadrilaterals is Fundamentals of Texture Mapping and Image Warping by Paul Heckbert; see section 2.2.2.


How much does driver quality affect Vulkan and DX12 compared to OpenGL and DX11?

TL;DR:  There's some room for optimization in Vulkan drivers, but not nearly to the degree that there is in GL / DX11 drivers.

  how much do drivers affect performance?

It depends a great deal on how the application is written and the hardware.  Specifically it depends on whether you're using features where there are large opportunities for optimization on the driver side.

For example, if an application is written to take full advantage of renderpass subpasses and properly specifies the subpass dependencies and properly sets up transient attachments for intermediate work, then the difference between an extremely naive Vulkan driver and a well written one could easily be an order of magnitude.  But at the same time, this is basically the difference between naive client code and well written code both running on a well written driver.

  people on the internet talk a lot about NVIDIA drivers being more finely tuned and optimized (especially for the most popular games)

This isn't just a matter of driver tuning.  It's arguably a form of cheating.  It's pretty well known that in the case of certain AAA games, nVidia will actually swap out functionality from the game with baked functionality from it's own drivers, doing things like swapping out shaders that were bundled in the game with their own customized shaders, or toggling internal flags that conventional developers have no access to.

The problem is that DX11 and GL are both very old APIs with lots of legacy functionality.  Their very design in some places is at odds with modern GPU hardware, especially in the area of bundling individual commands together before actually sending them to be executed.  This is why all modern APIs like Metal, Vulkan and DX12 have the concept of a command buffer or batch.  So that the developer can actually be involved in the process of bundling lots of small bind and draw commands together in a logical way.  Older API's like OpenGL have to guess at where the command buffer boundaries should be and (one assumes) drivers tend to have a variety of strategies for doing this depending on how the client application is calling functions.

This is where the driver level optimizations come in, where nVidia or AMD can look at how a given game is calling functions and decide that strategy A is better than the strategy B (the one that driver version X is using), and so they release driver X + 1, and it forces the use of strategy A, or even a new strategy C that they came up with for this specific game.

The end result is that OpenGL (and I assume DX < 12) drivers end up being monstrously complex, not just because of the enormous amount of legacy functionality they have to support, but also because of this per-game baggage that they accumulate (although I suspect that such per-game baggage has some kind of finite lifetime.... no one is going to be benchmarking a RTX 2080 with Battlefield 1942).


Speeding up convergence: am I reinventing the wheel?

It sounds like you're trying to do adaptive subdivision on the surface of a hemisphere.

This has indeed been tried before, although perhaps not in the exact way you're thinking.  Usually for hemispheric integration, people use some well-distributed, pre-defined sampling density over the hemisphere and forget about adaptive subdivision.

But you may be missing the point.

The whole idea of doing such things is to get a better estimate of the integral.  But it's not the integral over the hemisphere you care about.  You're integrating over the pixel to get AA, over the aperture to get depth-of-field, over the lights to get soft-shadowing, over the hemisphere to get GI, over time to get motion blur, over wavelength to get dispersion, and so on.  You're integrating over all that.  This is an absurdly high-dimensional space, and decades of development have shown that you need to be exploring it not more carefully, but faster.  That is, you get better results from breadth, not depth.

All very abstract.

The point is this: you don't want to sample over the hemisphere with lots of samples.  You want to sample over your hemisphere with one sample and then move on.  This is called path tracing.  It differs from ray tracing in that the maximum width of the ray tree is 1 (or maybe 2).

The key insight is that if your random samples are well-distributed with respect to all dimensions, other rays will "fill-in" the rest of your missing tree.

For an intuitive example: say you're doing 4x AA.  So you shoot four rays through a pixel.  Probably, they hit in very similar points (creating similar reflectance hemispheres).  Then, they each shoot exactly one recursive ray.  But, since the four hemispheres are in almost the same place, the recursive rays (kindof) count for all the hemispheres.  So each of the four paths gets some of the benefit of the other three.

That's sortof handwavy math, but since each path is actually independent of the others, and the color you get is an unbiased estimate of the entire scene's illumination, averaging them together is unbiased.  It just converges much faster in practice.

Obviously, if a cheap estimate of the hemisphere's integral is available, you can just use that--this is preintegration (cone tracing, etc.), importance sampling (next event estimation, light sampling), or better paths (BDPT, photon mapping)--but for a simple approach, it's basically always better to use path tracing instead of stochastic ray tracing.

Also matt -> matte (or mat, in a pinch).  It's a French loanword, not a person's name, and I don't care what the American dictionaries say.


How do people come up with subdivision schemes?

The subdivision schemes are not arbitrary. Catmull-Clark, arguably the most used subdivision scheme, generalizes bicubic B-splines to meshes of arbitrary topology.

Most, other subdivision schemes also generalize other degree B-splines to arbitrary topology. Doo-Sabin for biquadratic B-splines and Loop subdivision generalizes quartic box-splines defined over regular triangulations.

Most of the coefficients for subdivision are obtained by looking at knot-insertion. Knot-insertion is the process of adding knots into B-splines such that the same curve is obtained by expressing it as more curve pieces, i.e. subdividing the original curves. The rules for subdivision and thus the coefficients are derived from these algorithms. However, these rules only hold for the regular regions of the control mesh. For the irregular regions the rules are generalized, such that they give a satisfactory result.

Another way to look at it is through subdivision stencils and masks. The simplest subdivision stencil $[1,1]$ applied to the control polygon with points $P_i, \ldots, P_N$ will generate new points $Q_i = P_i$, will simply insert points at the same positions as the original points.
We can convolve the stencil with itself $[1,1] * [1,1] = [1,2,1]$. We can decompose this stencil into an even [1,1] and an odd mask [2] and again apply it to the control polygon by multiplying the coefficients in the mask and dividing by the sum of coefficients. The odd mask $Q_{i*2} = 2 P_i / 2$ will keep the original points, but the even mask will generate points $Q_{i*2+1} = (1 P_i + 1 P_{i+1})/2$ half-way between points.

Continuing the convolution process to the second stencil $[1,2,1] * [1,1] = [1, 3, 3, 1]$ will generate odd $[1, 3]$ and even $[3,1]$ masks which will generate points one quarter and three quarters along the line piece between points $P_i$ and $P_{i+1}$. This are of course the exact coefficients used in Chaikin's corner cutting algorithm, which in the limit of subdivision will generate quadratic B-splines. Applying it again $[1,3, 3,1] * [1,1] = [1, 4, 6, 4, 1]$ will generate the stencils that generate cubic B-splines.

To generalize this again to surfaces we can apply this convolution process in two parametric directions to obtain bi-degree B-splines. In the cubic case we obtain this matrix of coefficients through convolution:

$$\begin{bmatrix}1 & 4 & 6 & 4 & 1\\
                4 & 16 & 24 & 16 & 4\\
                6 & 24 & 36 & 24 & 6\\
                4 & 16 & 24 & 16 & 4\\
                1 & 4 & 6 & 4 & 1\\ \end{bmatrix}$$

We can decompose this matrix into vertex mask:

$\begin{bmatrix}1 &  6 & 1\\
                6 & 36 & 6\\
                1 &  6 & 1\\ \end{bmatrix}$

edge point mask:

$\begin{bmatrix}4 &  24 & 4\\ 4 & 24 & 4\\ \end{bmatrix}$ and $\begin{bmatrix}4 & 4\\ 24 & 24\\ 4 & 4\\ \end{bmatrix}$

face point mask:

$\begin{bmatrix}
16 & 16 \\
16 & 16 \\
\end{bmatrix}$

Applying these as control point masks to a regular control grid will obtain bicubic B-splines. The observation made for the Catmull-Clark algorithm is that the inserted points through the face stencils are the average of the surrounding vertices. This can easily be generalized for arbitrary polygons, the same thing goes for the edge stencil. The hard case is the vertex stencil as the original is only valid for vertices with valency 4. Catmull-Clark came up with this expression

$$\frac{Q}{N} + \frac{2R}{N} + \frac{S(N-3)}{N}$$

Where $S$ is the old vertex position $R$ is the average of the points created by the edge mask for the outgoing edges of the vertex and $Q$ the average of the points created by face mask for the surrounding faces. The ratios in this expression are such that it reproduces the regular vertex mask in the case $N=4$ and is aesthetically pleasing for other $N$.


How to create specular surface in Mitsuba Renderer

You want to use the "conductor" plugin to create a smooth conductor / metal type BRDF. See section 8.2.6 in the documentation.

As far as Mitsuba is concerned, a mirror is a special instance of a metal, where the material is specified as "none" (which corresponds to using complex IOR of $ \eta = 0 $ and $ k = 1 $).

Here is an example in xml:

<shape type="...">
    <bsdf type="conductor">
        <string name="material" value="none"/>
    </bsdf>
<shape>


OpenGL : How to rotate an object around viewing space x-axis

A rotation around the x-axis is achieved by this matrix
$$
\mathtt{T}_\mathrm{rot X} = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0& \cos \alpha
            & -\sin \alpha& 0 \\ 0 & \sin \alpha & \cos \alpha & 0 \\ 0 & 0 & 0 &1 \\ \end{bmatrix}
$$

In order to rotate around the x-axis in the viewing space, one needs to think about when to apply the above transformation in the chain of transformations that is applied to the 3D model vertices $\mathbf{X}_{\mathrm{3d}}$ in the employed rendering pipeline.

A typical transformation chain is:

$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}
$$

Let's split the camera tranformation into its rotational and translational part:
$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{camera\ transform}}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}
$$

Then we need to apply the x-rotation in-between the camera rotation and translation, because we want to have the rotation of the camera applied beforehand, but not the camera's translation:

$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\underbrace{\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}}_{\mathtt{T}_{\mathrm{new\ camera\ transform}}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}
$$

This would achieve the desired rotation, but typically we do not want to change the camera transformation but the the mesh transformation. This is achieved by
$$
\mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\
\Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{cam\ trans}}\,(\mathtt{T}_{\mathrm{cam\ rot}}\,\mathtt{T}_{\mathrm{cam\ rot}}^{-1})\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}\,\mathbf{X}_{\mathrm{3d}}\\
\Leftrightarrow \mathbf{X}_{\mathrm{view}} = \mathtt{T}_{\mathrm{projection}} \,\mathtt{T}_{\mathrm{camera\ transform}}\,\underbrace{\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}\,
\mathtt{T}_{\mathrm{mesh\ transform}}}_{\mathtt{T}_{\mathrm{new\ mesh\ transform}}}\,\mathbf{X}_{\mathrm{3d}}
$$
Therefore, the short answer to your question is:

Left-multiply the current transformation matrix of your model with $\mathtt{T}_{\mathrm{cam\ rot}}^{-1}\,\mathtt{T}_\mathrm{rot X}\,\mathtt{T}_{\mathrm{cam\ rot}}$.

I have generated an "cow" example scene in the GSN Composer that can be used for interactive testing in a WebGL-capable browser:

Example: CowQuestion | GSN Composer


What metrics are used for deciding if a 3D object is visible?

Visibility for culling (or passing to the render pipeline) is very sensitive to false negatives. False positives are not a big deal because when rendering the Z buffer will take care of that.

This means that one very simple metric is simply assuming that it's always visible. If any other metric gives too much overhead than simply processing the object requires then this is a good fallback. For example in a redering api with command buffers rebuilding the command buffer would be too expensive compared to the processing required to render a single object to warrant it.

The next option is to see if the object is in front or behind the camera, This can be done with a dot product with the view vector and the forward most point of the bounding volume of the object. If it's negative then the object is behind the camera and you can skip it.

The next is using the view pyramid. From the camera you extrude the frustum and do a collision check between it and the object. If it collides then the object is visible. Instead of 1 plane you now check against 4 and the object must pass All 4 to be considered visible.

Next you can take into account when the object is fully occluded by another object. This can be done by various methods. With static geometry you can partition the world and say for each section which other sections you can see from it. You can also ask the GPU to do occlusion queries which basically means "did any fragments of these render commands pass the depth test". But getting the information back from the GPU typically has a decently long latency, by which time the camera may have moved.


Edit: I suddenly understood the issue. The noise in the top right corner of (b) is because it's struggling to find the tiny fraction of angles in which the BSDF is not extremely close to zero.

Yes your reasoning here is correct.

Taken to the extreme if the top most surface was perfectly mirror (dirac delta BRDF), then it would never* sample the right ray from the light source that was perfectly aligned with the camera's view direction. For such a BRDF $\omega_i$ must equal $\omega_o$.


Fourier transforms wouldn't help you with a rotation. You'd just end up having to rotate the matrix of Fourier coefficients, instead of rotating the original image.

Consider for example an image made of a perfect sine wave along the x-axis with wave-vector $(k, 0)$.  (The wave-vector is the spacial frequencies along the $x$ and $y$ axes).  The Fourier transform of this would be all black, with a single white pixel at position $(k, 0)$.

When you rotate the image by an angle $\theta$, you'll get a perfect sine wave along an oblique axis.  If you draw a diagram and do a little trigonometry, you can see that the new wave-vector will be $(k \cos \theta, k \sin \theta)$.  (It helps for this part to know that the frequency $k$ is one over the period.)

Therefore, the Fourier transform of the rotated result would be all black, with a single white pixel at position $(k \cos \theta, k \sin \theta)$.  But that's just what you'd get if you rotated the Fourier transform of the original wave by $\theta$.

As for the Gibbs phenomenon, it's an issue when downsampling an image, or compressing it by frequency quantization (as in JPEG). But simply taking the Fourier transform of an image doesn't introduce any Gibbs ringing. The Fourier transform is lossless and perfectly reversible if done properly: it represents all the information in the original image.