Selection of GI techniques working in real time

In this vein a bit, Killzone renders one bounce reflections, but uses temporal anti aliasing to combine the last frame with the current frame.

In practice, this means you get MANY bounce reflections, since in effect each frame rendered catches the previous number of reflections as the new reflection - adding one each frame.

Not quite GI, but i wouldn't be surprised if people pursue that direction more in the future, if they aren't already working on it.

The temporal anti aliasing used normally lets you get more samples with subsequent frames, but used in this way, it also gives you higher bounce counts with subsequent frames.

I could totally see doing path tracing type path following, choosing reflection or BSDF rays based on fresnel equations for instance, and having the temporal AA combine it over time.


The function calculates the intersection points between two edges of a triangle and a plane that is assumed to cut it. It returns a line segment running along the cut.

Or more generally, the function calculates where the intersection points with the two edges would be, if the edges were extended to infinite lines. Given that it's called "project", that might be the intended use-case; hard to know without seeing how this gets used in context.

The first two lines of math are calculating the intersection point of the $p_0 p_1$ edge with the z-plane. It works by linearly interpolating the $x, y$ values along the edge based on the proportion between $z$ values. The second two lines do the same thing for the $p_0 p_2$ edge.

One odd thing is that the function is using integers everywhere. Presumably it's because the program is using fixed-point coordinates (which makes good sense for a lot of geometry processing applications due to its uniform precision). However, the way it's done here seems oblivious to rounding: the integer divisions will truncate any fractional component instead of rounding to nearest.


1) Scale the texture coordinates from [0,1]x[0,1] -> [0,3]x[0,3]. It works with higher integer scales as well. This creates an imaginary 3x3 tiles grid.

st *= 3.;

2) Split the upscaled texture coordinates into the integer part and its float reminder.

vec2 i_st = floor(st); //the tile coords of the fragment - an integer pair {0,1,2}x{0,1,2}
vec2 f_st = fract(st); //fragment coords within the tile - a float vec2 [0,1]x[0,1]

//e.g. let's take
//  st == vec2(1.1,1.7)
//  i_st == vec2(1.,1.)
//  f_st == vec2(.1,.7)

Now st == i_st + f_st holds.

3) In order to avoid terms confusion I will call the points that build the Voronoi cells Voronoi points. Each of the 9 tiles has one Voronoi point inside to produce a Voronoi cell. So there will be 9 Voronoi cells visible in the resulting texture.

The main hack here is to have this regular 3x3 tiles grid with the same amount of Voronoi points and randomize their positions only within the respective tile. So the distribution will be pseudo-even. You give up irregular distributions by this assumption, e.g. all points grouped in the upper-left corner, but such arrangements are highly unlikely to result from biologic processes that the cellular noise tries to mimic.

Since random2 is deterministic, it will return the same vec2 each time you enter the same i_st, so for a fixed i_st it will stay constant across all fragments.

vec2 point = random2(i_st); //pick the Voronoi point, e.g. vec2(.6,.4)

The global position of the picked Voronoi point is in fact i_st + point.

4) Take the distance to the Voronoi point inside of the current tile as the initial minimum distance.

vec2 diff = point - f_st //e.g. vec2(.6,.4) - vec2(.1,.7) == vec2(.5,-.3)
float dist = length(diff); //e.g.0.583

5) So you know that there is exactly one Voronoi point in each cell, thus by simple geometric means you can prove that you need to check only the direct neighbors. You need to test if any of the neighboring tiles has a Voronoi point closer to the current fragment than the Voronoi point from 4). Two loops iterate over all 3x3 neighborhood of the current tile. It is much easier to do that relatively {-1, 0, 1}x{-1, 0, 1} instead of using absolute numbers {i_st.x-1, i_st.x, i_st.x+1}x{i_st.y-1, i_st.y, i_st.y+1}. Moreover, we know that each tile is 1x1 in size.

for (int y= -1; y <= 1; y++) {
    for (int x= -1; x <= 1; x++) {
        // relative neighbor coordinates will be sufficient
        // this is the integer part indicating the neighboring tile coordinates
        vec2 neighbor = vec2(float(x),float(y)); //e.g. vec2(-1., 1)

        //to be continued

6) Now you will need to compute the absolute coordinates of neighboring tile to obtain its Voronoi point consistently across all fragments.

        //the global neighbor tile coordinates are neighbor + i_st
        vec2 point = random2(i_st + neighbor); //note that this is a local scope variable

        //to be continued

7) And now the most interesting step for you, computing the distance to the neighboring Voronoi point

        //neighbor: relative tile coords, e.g. vec2(-1., 1.)
        //point:    Voronoi point inside of the neighboring tile e.g. vec2(0.9, 0.2)
        //f_st:     current fragment position within its home tile,
        //          e.g. when st == vec2(1.1, 1.7) then f_st == vec2(.1,.7)

        vec2 diff = neighbor + point - f_st;

        //to be continued

The example values yield:

neighbor + point: vec2(-1.,1.) + vec2(0.9, 0.2) == vec2(-0.1, 1.2)

That is the relative position of the neighboring tile Voronoi point. Relative means **with respect to the lower-left corner of the current fragment's tile.
The same relative coordinates apply to f_st and also to point from 3).

(neighbor + point) - f_st: vec2(-0.1, 1.2) - vec2(.1,.7) == vec2(-.2,.5)

This is shorter distance (0.538 vs. previous 0.583) so for the example positions the Voronoi point from the upper left cell is closer.

8) The rest is pretty straight forward. Compute the magnitude of the diff vector and keep the minimum.

        float dist = length(diff); //distance to the neighboring Voronoi point

        m_dist = min(m_dist, dist); //remember the minimum of distances
    }
}

I hope this will help you to understand that. The main simplification is the switch to relative coordinates as described in 7).

Looking at the code I see that the middle tile (current fragment tile) will be evaluated twice. This can be reduced by omiting step 4) initializing dist to 2*sqrt(2) which is very roughly 3. That is the length of the diagonal of two cells - the farthest distance the current fragment st could be from one of the neighboring Voronoi points.
Border tiles are handled in an elegant implicit way. random2 generates a Voronoi point for them as well and involves it in the minimal distance computation. So You will see 9 complete and possibly up to 16 more incomplete Voronoi cells.

Hope that helps, feel free to ask any further details.


How to draw depth-based volume in real-time

The issue is that an extra depth pass won't cut it. You may need an arbitrary number of extra depth passes. Just imagine the volume between two sinusoidal surfaces, you would have infinitely many alternating z-intervals of volume/empty space as long as you're looking from a specific direction.

Edit:
Taking into consideration the updated formulation, here's my suggestion for a rather general algorithm with different attenuation coefficients $\kappa_i$ for each volume. Note that there are non-trivial parts that are missing, also if you do not require different attenuation coefficients it can be optimized even further.
The algorithm relies on you drawing the volumes pixels in reverse z-order (this is the non-trivial missing part). You draw your volumes in a separate pass. You keep an attenuation density accumulation variable $\kappa$ for each pixel (initialize to $0$), as well as a distance time stamp $\delta$ (initialize to the the $z$ of the far plane, which I will denote $\max$), and a "color" variable $c$ (initialize to $0$). Whenever you draw a pixel:

if $\delta==\max$ and you drew a front face of volume $i$ then:
$$d = \delta-pixelDepth$$
$$c = attenuate(c,\kappa_i,d)$$

Where $\kappa_i$ is the attenuation of the currently drawn pixel (you can simply draw the volumes in the "color" of their attenuation and use that).

if $\delta<\max$ and you drew a back face of volume $i$ then:

$$d = \delta - pixelDepth$$
$$\delta = pixelDepth$$
$$c = attenuate(c,\kappa, d)$$
$$\kappa = \kappa + \kappa_i$$

if $\delta<\max$ and you drew a front face of volume $i$ then:
$$d = \delta -pixelDepth$$
$$\delta = pixelDepth$$
$$c = attenuate(c,\kappa, d)$$
$$\kappa = \kappa - \kappa_i$$
after you've drawn all pixels if $\kappa > 0$:
$$c = attenuate(c,\kappa,\delta)$$

Note that the 4th check handles the case where the camera is inside a volume (and it ends behind the camera), but the frustum is not entirely inside it. Similarly the first check handles the case where a volume is cutoff by the frustum. However, the case where the camera frustum is totally inside a volume is not handled (you can try to check this by verifying that the camera is inside a volume and then simply adding a constant to $\kappa$, then you have to modify the end case too, to check $>const$, or just not have such large volumes).
Note that the $\delta$ part can be handled by the depth buffer. Obviously you can use a compute shader, or use additional buffers to perform all of this. The function $attenuate$ can be standard Beer-Lambert uniform attenuation or whatever else you want. Combining the volumes with the original scene happens by setting the "far plane" for each pixel to the closest drawn solid geometry depth (that is initialize $\delta$ based on what you have in the depth buffer for your solid geometry).


How to convert Non-Axis Aligned Bounding Boxes to AABB

You can assign a coordinate system to each nAABB in such a way that the nAABB becomes an AABB in its own coordinate system. We call this a local coordinate system.

I assume rays are expressed in a world or global coordinate system. In order to test an nAABB for intersection, one first needs to apply the world-to-local transformation on the ray (origin and direction). This way we obtain a transformed ray which we can intersect with an AABB. Once an intersection point is found, we need to transform this back to world space via the inverse transformation (i.e. local-to-world transformation).

In practice an nAABB is thus expressed as an AABB with a pair of world-to-local and local-to-world transformation matrices.

If one applies a scaling ($S$) or translation ($T$) on an AABB, one obtains another AABB. If one applies a rotation ($R$) on an AABB, one obtains a nAABB.
Typically the corresponding transformation matrices are multiplied as follows $T * R * S$. Thus translation is applied after rotation. Since the $R$ component must be included in the world-to-local transformation, you need to include the $T$ component as well. The $S$ component can be included or directly applied to a unit AABB centered at the origin of the world-coordinate system. Including a $T$ component requires your world-to-local and local-to-world transformation matrices to be of size $4$x$4$ (expressed in homogeneous coordinates).

Here you can find C++ code to construct all the $4$x$4$ world-to-local matrices and their inverses you probably need.

Note that this approach works for other geometrical objects as well (if the transformations involve scaling, rotation and translation).


YouTube Help: Upload 360-degree videos

  We recommend uploading  360 videos (equirectangular format with a 2:1 aspect ratio) at a resolution of 7168x3584 or higher, up to 8192x4096.

That means the format uses Equirectangular projection for capturing the view from all directions to a normal video format with a 2:1 aspect ratio. The projection should translate nicely to a raytracing engine with a simple mapping from pixel coordinates to the ray direction using the formula in the wiki article.


The relative size of the spacing of knots is irrelevant for the NURBS curve. The only thing that matters is that they keep the relation. Note this may not be wise as parametrization may have other uses behind the scenes.

Image 1: 3 differently parametrized knots result in same curve if knot values are relatively the same.

So you can scale and offset knot points as you wish. However you can not make the relative distances between entries different or your curve will change.

Image 2: On the other hand if you change the relative spacing your in trouble. So beware of floating point errors if you need to be really accurate.


Why does my code generate a figure whose kurtosis looks a little higher than the one I am trying to reproduce?

The value 's' in the original graph is the variance, which is the square of the Gaussian sigma (i.e. the scale parameter of the normal distribution). So norm.pdf(xx,0,np.sqrt(yy)) should be correct.

Note that the shape of the peak you see is very much dependent on the y value where the graph starts. Your starting point of 0.1 may be a little smaller than in the original graph, which looks like it starts at about 1/6 ~= 0.167, so you may see a narrower and taller peak for that reason. Also, the original graph doesn't quite have enough horizontal resolution to resolve the shape of the peak entirely, as you can see from how far apart the grid lines are spaced vertically on the sides of the peak.


Power heuristic in multiple importance sampling

PBRT v3 Page 799:

Veach determined it empirically.


What modern real-time rendering applications still use fragment shader depth writes / late-Z?

Any techniques that involve raytracing in the fragment shader might want to write Z in order that the depth buffer contain an accurate representation of the raytraced surface. For example:

Distance-field ray marching, as seen in many shadertoys and demoscene productions. Here, only a full-screen quad gets rasterized, and all the geometry is generated by the fragment shader. Writing Z would be necessary if you want to use the depth buffer for deferred lighting or post-effects, or to composite distance fields with ordinary triangle geometry and get correct occlusion.
Voxel ray marching, similar to the above. Even when the voxels are translucent (as in a smoke cloud or suchlike), it may be useful to write Z when the voxel density is high enough to become opaque; for instance, that can be useful for later motion blur / depth-of-field calculations.
Parallax occlusion mapping techniques, in which a flat surface is given the appearance of depth by ray-marching against a heightfield.

There are probably other similar cases where the shape of a surface is defined by the fragment shader rather than by the rasterized triangles.

By the way, newer APIs and GPUs include support for a conservative depth output mode, in which the shader-written Z is only allowed to be greater-equal to the geometric Z, but not less. This allows early-Z / hierarchical-Z optimizations to still work based on the geometric Z: if the rasterized geometry gets culled for being behind something, then you know the shader-written Z would have been culled as well. This is a good fit for cases like the above, as you can rasterize some bounding geometry for the distance-field / voxel / heightfield object; then the actual depth will be greater-equal to that of the bounding geometry.


How does hardware texture compression work?

"How (hardware) texture compression works" is a large topic. Hopefully I can provide some insights without duplicating the content of Nathan's answer.

Requirements

Texture compression typically differs from 'standard' image compression techniques e.g. JPEG/PNG in four main ways, as outlined in Beers et al's Rendering from Compressed Textures:

Decoding Speed: You don't want texture compression to be slower (at least not noticeably so) than using uncompressed textures. It should also be relatively simple to decompress since that can help achieve fast decompression without excessive hardware and power costs.
Random Access: You can't easily predict which texels will be required during a given render. If some subset, M, of the accessed texels come from, say, the middle of the image, it's essential that you don't have to decode all of the 'previous' lines of the texture in order to determine M; with JPEG and PNG this is necessary as pixel decoding depends on the previously decoded data.
Note, having said this, just because you have "random" access, doesn't mean you should try to sample completely arbitrarily
Compression Rate and Visual Quality: Beers et al argue (convincingly) that losing some quality in the compressed result in order to improve compression rate is a worthwhile trade-off. In 3D rendering, the data is probably going to be manipulated (e.g. filtered & shaded etc) and so some loss of quality may well be masked.
Asymmetric encoding/decoding: Though perhaps slightly more contentious, they argue that it is acceptable to have the encoding process much slower than the decoding. Given that the decoding needs to be at HW fill rates, this is generally acceptable. (I will admit that compression of PVRTC, ETC2 and some others at maximum quality could be faster)

Early History & Techniques

It may surprise some to learn that texture compression has been around for over three decades. Flight simulators from the 70s and 80s needed access to relatively large amounts of texture data and given that 1MB of RAM in 1980 was > $6000, reducing the texture footprint was essential. As another example, in the mid 70s, even a small amount of high speed memory and logic, e.g. enough for a modest 512x512 RGB frame buffer) could set you back the price of small house.

Though, AFAIK, not explicitly referred to as texture compression, in the literature and patents you can find references to techniques including:
 a. simple forms of mathematical/procedural texture synthesis,
 b. use of a single channel texture (e.g. 4bpp) that is then multiplied by a per-texture RGB value,
 c. YUV, and
 d. palettes (the literature suggesting use of Heckbert's approach to do the compression)

Modelling Image Data

As noted above, texture compression is nearly always lossy and thus the problem becomes one of trying to represent the important data in a compact way whilst disposing of the less significant information. The various schemes that will be described below all have an implicit 'parameterised' model that approximates the typical behaviour of texture data and of the eye's response.

Further, since texture compression tends to use fixed-rate encoding, the compression process usually includes a search step to find the set of parameters which, when fed in to the model, will generate a good approximation of the original texture. That search step, however, can be time consuming.
(With the possible exception of tools such as optipng, this is another area where typical use of PNG & JPEG differ from texture compression schemes)

Before progressing further, to help with further understanding of TC it's worth taking a look at Principal Component Analysis (PCA) - a very useful mathematical tool for data compression.

Example texture

To compare the various methods, we'll use the following image:

Note that this is a fairly tough image, especially for palette and VQTC methods as it spans much of the RGB colour cube and only 15% of the texels use repeated colours.

PC and (post mid 90s) Console Texture compression

To reduce data costs, some PC games and early games consoles also made use of palette images, which is a form of Vector Quantisation (VQ).
Palette-based approaches make the assumption that a given image only uses relatively small portions of the RGB(A) colour cube.  A problem with palette textures is that the compression rates for the achieved quality is generally rather modest. The example texture compressed to "4bpp" (using GIMP) produced

Note again that this is a relatively tough image for VQ schemes.

VQ with larger vectors (e.g. 2bpp ARGB)

Inspired by Beers et al, the Dreamcast console used VQ to encode 2x2  or even 2x4 pixel blocks with single bytes. While the "vectors" in the palette textures are 3 or 4 dimensional, the 2x2 pixel blocks can be considered to be 16 dimensional. The compression scheme assumes there is sufficient, approximate repetition of these vectors.

Even though VQ can achieve satisfactory quality with ~2bpp, the problem with these schemes is that it requires dependent memory reads: An initial read from the index map to determine the code for the pixel is followed by a second to actually fetch the pixel data associated with that code. Additional caches can help alleviate some of the incurred latency, but adds complexity to the hardware.

The example image compressed with the 2bpp Dreamcast scheme is
.
The index map is:

Compression of VQ data can be done in various ways however, IIRC, the above was done using PCA to derive and then partition the 16D vectors along the principal vector into 2 sets such that two representative vectors minimised the mean squared error. The process then recursed until 256 candidate vectors were produced.
A global k-means/Lloyd's algorithm approach was then applied to improve the representatives.

Colour Space Transformations

Colour space transformations also make use of PCA noting that the global distribution of colour is often spread along a major axis with far less spread along the other axes. For YUV representations, the assumptions are that a) the major axis is often in the luma direction and that b) the eye is more sensitive to changes in this direction.

The 3dfx Voodoo system provided "YAB", an 8bpp, "Narrow Channel" compression system that split each 8 bit texel into a 322 format, and applied a user selected colour transform to that data to map it into RGB. The main axis thus had 8 levels and the smaller axes, 4 each.

The S3 Virge chip had a slightly simpler, 4bpp, scheme that allowed the user to specify, for the entire texture, two end colours, which should lie on the principal axis, along with a 4bpp monochrome texture. The per-pixel value then blended the end colours with appropriate weights to produce the RGB result.

BTC-based schemes

Rewinding some number of years, Delp and Mitchell designed a simple (monochrome) image compression scheme called Block Truncation Coding, (BTC). This paper also included a compression algorithm but, for our purposes, we are mainly interested in the resulting compressed data and of the decompression process.

In this scheme, images are broken into, typically, 4x4 pixel blocks, which can be compressed independently with, in effect, a localised VQ algorithm. Each block is represented by two "values", a and b, and a 4x4 set of index bits, which identify which of the two values to use for each pixel.

S3TC:  4bpp RGB (+1bit alpha)
Although several colour-variants of BTC for image compression were proposed, of interest to us is Iourcha et al's S3TC, some of which appears to be a rediscovery of the somewhat forgotten work of Hoffert et al that was used in Apple's Quicktime.

The original S3TC, without the DirectX variants, compresses blocks of either RGB or RGB+1bit Alpha to 4bpp. Each 4x4 block in the texture is replaced by two end colours, A and B, from which up to two other colours are derived by fixed-weight, linear blends. Further, each texel in the block has a 2-bit index that determines how to select one of these four colours.

For example the following is a 4x4 pixel section of the test image compressed with the AMD/ATI Compressenator tool. (Technically it's taken from a 512x512 version of the test image but forgive my lack of time to update the examples).

This illustrates the compression process: The average and the principal axis of the colours are calculated.  A best fit is then performed to find two end points that 'lie on' the axis which, along with the two derived 1:2 and 2:1 blends (or in some cases a 50:50 blend) of those end points, that minimises the error.  Each original pixel is then mapped to one of those colours to produce the result.

If, as in this case, the colours are reasonable approximated by the principal axis, the error will be relatively low. However if, like in the neighbouring 4x4 block shown below, the colours are more diverse, the error will be higher.

The example image, compressed with the AMD Compressonator produces:

Since the colours are determined independently per-block, there can be discontinuities at block boundaries but, as long as the resolution is kept sufficiently high, these block artefacts may go unnoticed:

ETC1:  4bpp RGB
Ericsson Texture Compression also works with 4x4 blocks of texels but makes the assumption that, much like YUV, the principal axis of a local set of texels is often very strongly correlated with "luma". The set of texels can then be represented by just an average colour and a highly quantised, scalar 'length' of the projection of the texels onto that assumed axis.

Since this reduces the data storage costs relative to say, S3TC, it allows ETC to introduce a partitioning scheme, whereby the 4x4 block is subdivided into a pair of horizontal 4x2 or vertical 2x4 sub-blocks. These each have their own average colour.
The example image produces:

The area around the beak also illustrates the horizontal and vertical partitioning of the 4x4 blocks.

Global+Local

There are some texture compression systems that are a cross between global and local schemes, such as that of the distributed palettes of Ivanov and Kuzmin or the method of PVRTC.

PVRTC:  4 & 2 bpp RGBA
PVRTC assumes that an (in practice, bilinearly) upscaled image is a good approximation to the full-resolution target and that the difference between the approximation and the target, i.e. the delta image, is locally monochromatic, i.e. has a dominant principal axis. Further, it assumes the local principal axis can be interpolated across the image.

(to do: Add images showing breakdown)

The example texture, compressed with PVRTC1 4bpp produces:

with the area around the beak:

Compared to BTC-schemes, the block artefacts typically are eliminated but there can sometimes be "overshoot" if there are strong discontinuities in the source image, for example around the silhouette of the lorikeet's head.

The 2bpp variant has, naturally, higher error than the 4bpp (note loss of precision around the blue, high frequency areas near the neck) but arguably still of reasonably quality:

A note on decompression costs

Although the compression algorithms for the schemes described above have a moderate to high evaluation cost, the decompression algorithms, especially for hardware implementations, are relatively inexpensive. ETC1, for example, requires little more than a few MUXes and low-precision adders; S3TC effectively slightly more addition units to perform the blending; and PVRTC, slightly more again.  In theory, these simple TC schemes could allow a GPU architecture to avoid decompression until just prior to the filtering stage, thus maximising the effectiveness of internal caches.

Other Schemes

Other common TC modes that should be mentioned are:

ETC2 - is a (4bpp) superset of ETC1 that improves the handling of regions with colour distributions that don't align well with 'luma'. There are also a 4bpp variant that supports 1 bit alpha, and an 8bpp format for RGBA.
ATC - Is effectively a small variation on S3TC.
FXT1 (3dfx) was a more ambitious variant of the S3TC theme.
BC6 & BC7 : An 8bpp, block-based system supporting ARGB. Apart from HDR modes, these use a more complex partitioning system than that of ETC to attempt to better model image colour distribution.
PVRTC2 : 2&4bpp ARGB. This introduces additional modes including one to overcome limitations with strong boundaries in the images.
ASTC : This is also a block based system but is somewhat more complicated in that it has a large number of possible block sizes targeting a wide range of bpp. It also includes features such as up to 4 partition regions with a pseudo-random partition generator, and variable resolution for the index data and/or colour precision and colour models.


Why the gradients generated in original Perlin Noise causing axis-aligned clumping?

Very briefly, Perlin noise ideally is meant to have, on average, the same frequency characteristics no matter where in the texture you are or in what direction you are looking.

However, the way it's done is to define pseudo-random values (and derivatives) on a regular 3D grid which introduces some compromises.

To make this easier to understand, imagine instead using just a 2D grid, and on every grid location with an even-numbered X coordinate, you set the value to 0.0, and on every odd X, 1.0.  So at (0,0), (2,0), (4,99) etc you have 0.0, while at at (1,0) etc, you have 1.0.

If you followed a horizontal line, e.g (0,0) (1,0) (2,0) etc, it would go up and down, in a sine-wave-like pattern every 2 steps, but if you followed it in a diagonal direction,  (0,0), (1,1), (2,2,) then, per distance traveled, it would go up and down more slowly.

I believe that is what Perlin is referring to.


How to read illum value from mtl file using Assimp library?

According to the source (https://github.com/assimp/assimp/blob/b7df376836e36c4b6998ce95d97626e30f61c2f0/code/AssetLib/Obj/ObjFileImporter.cpp#L594) only illum values of 0, 1, 2 are supported:
    // convert illumination model
    int sm = 0;
    switch (pCurrentMaterial->illumination_model) {
    case 0:
        sm = aiShadingMode_NoShading;
        break;
    case 1:
        sm = aiShadingMode_Gouraud;
        break;
    case 2:
        sm = aiShadingMode_Phong;
        break;
    default:
        sm = aiShadingMode_Gouraud;
        ASSIMP_LOG_ERROR("OBJ: unexpected illumination model (0-2 recognized)");
    }


Yes, you should set the lightmap texture as the render target and use UV-coordinates as vertex position coordinates in your shader to bake lighting into lightmaps. This way you'll write the baked lighting data of triangles into the correct lightmap texels that are used for rendering the triangles on the screen. The UV-map should be unique and no triangles should overlap in the UV-space. The lightmap data needs to be also expanded so that when you later use bilinear filtering to fetch the data from the texture that you don't end up having dark borders following triangle edges at UV seams.

Instead of baking indirect illumination into surface lightmaps, you could also use volume textures. Nice thing is that the with volume textures you'll have the indirect illumination data available everywhere in the scene and you don't need unique UV-mapping, but you'll also have less resolution due to the added memory usage.

If you need to bake indirect illumination into textures depends on the complexity of the scene, quality of the lighting and performance/mem usage you want to achieve, so there's no easy answer.


Understanding Jump Flooding Algorithm (JFA) for Voronoi Diagrams

I think that there is a bit of confusion in terminology. My understanding is that only the initially colored points, before step 1, are called seeds.

Maybe this helps clarify the algorithm as well. When the a point $ p $ with color $ s $ finds a neighbor $ q $ with color $ s' $, he compares the distance $ d(p,s) $ with $ d(p,s') $ (not $ d(p,q) $) to decide his new color. This is why each pixel crucially holds the $ (x,y) $ coordinates of his seed color (in the paper they used the r,g channels for this) and not just the distance. In each step this data is also passed to the neighbor.

So in that example, in moving from step length 4 to step length 2, that red square (the circled one that remains red) sees the green square who tells red square that the green seed is at $ (8,6) $. Red square then computes his distance to green square to be $ \sqrt{2^2 + 4^2} $, which is bigger than the distance to red ($ 4 $), and so remains red.


Is it possible to emulate Vulkan on a non Vulkan compatible gpu?

It's possible to emulate Vulkan on a sufficiently capable non-Vulkan API.  This is the point of the Vulkan Portability Initiative.  Indeed, there is already a Vulkan implementation on top of the Apple proprietary API Metal, and there is work being done to implement Vulkan on top of D3D12.

It probably would be possible to implement a version of Vulkan that ran partially on a GPU (to which you had some other API access) and partially in software on the CPU, but unless you're working with some kind of mobile device with integrated memory, the performance of such a system would probably be utter garbage.

If your goal is to better understand and develop for Vulkan, I'd suggest you investigate some of the existing efforts at creating a fully software Vulkan implementation.

If you're just trying to get some game to run better, I suggest you get a better laptop.


Given two convex patches, how to find points/faces that have distance < x?

Yes GJK is appropriate for this.

The overall algorithm of GJK goes something like

simplex = empty
direction n = arbitrary

while(not finished){

    find furthest point p1 in mesh 1 in direction n
    find furthest point p2 in mesh 2 in direction -n

    supplement simplex with point p1-p2

    if(simplex is tetrahedron) simplify simplex

    direction n = pick direction perpendicular to simplex pointing to origin

}

you can then use the simplex at the end of the loop to calculate the distance from it to the origin and set finished to true and you will get a collection of points on each mesh which make up a pair of faces that are closer than x.


How does Texture Cache work considering multiple shader units

At the top level, a GPU is subdivided into a number of shader cores.  A small GPU in a notebook or tablet may have only a few cores while a high-end desktop GPU may have dozens.

In addition to the shader cores there are also texture units. They may be grouped together with one texture unit per shader core, or one texture unit shared among two or three shader cores, depending on the GPU.

The whole chip shares a single L2 cache, but the different units will have individual L1 caches.  Texture units have texture caches, and shader units have caches for instructions and constants/uniforms, and maybe a separate cache for buffer data depending on whether buffer loads are a separate path from texture loads or not (varies by GPU architecture).

Texture units operate independently and asynchronously from shader cores. When a shader performs a texture read, it sends a request to the texture unit across a little bus between them; the shader can then continue executing if possible, or it may get suspended and allow other shader threads to run while it waits for the texture read to finish.

The texture unit batches up a bunch of requests and performs the addressing math on them—selecting mip levels and anisotropy, converting UVs to texel coordinates, applying clamp/wrap modes, etc. Once it knows which texels it needs, it reads them through the cache hierarchy, the same way that memory reads work on a CPU (look in L1 first, if not there then L2, then DRAM). If many pending texture requests all want the same or nearby texels (as they often do), then you get a lot of efficiency here, as you can satisfy many pending requests with only a few memory transactions.  All these operations are pipelined, so while the texture unit is waiting for memory on one batch it can be doing the addressing math for another batch of requests, and so on.

Once the data comes back, the texture unit will decode compressed formats, do sRGB conversion and filtering as necessary, then return the results back to the shader core.


Huge thanks to @MJP who answered this.
The aim is to avoid the simplification made when using tangent space normals.
Here is the paper : Blending in detail

But only implement equation (4) which gives you this.

float3 ReorientNormal(in float3 u, in float3 t, in float3 s)
{
    // Build the shortest-arc quaternion
    float4 q = float4(cross(s, t), dot(s, t) + 1) / sqrt(2 * (dot(s, t) + 1));
    // Rotate the normal
    return u * (q.w * q.w - dot(q.xyz, q.xyz)) + 2 * q.xyz * dot(q.xyz, u) + 2 * q.w * cross(q.xyz, u);
}

Original answer is here.


How can I create a bounding volume hierarchy for constantly moving objects?

Consider using spatial hashing, especially if your objects are similarly sized.

Basically, divide your world into uniformly-sized grid cells (2D and 3D are both valid possibilities depending on the amount of vertical motion). Each update, assign your object to each bin that it overlaps—if the cells are decently sized relative to the objects, most objects should end up in a single bin.

Each bin is inserted into a hash table, with the key being the coordinates of the bin. (You can also think of it as a hash table with multiple values for the same key, and inserting an object once for every cell that it overlaps.)

There's no hierarchy to rebuild in this scheme, which makes it well suited for dynamic scenes. You can still test the cell's dimensions against the frustum or against occluders at a coarse level and discard many objects at once. Also, it's easier to manage this structure incrementally—you can keep the hash table the same from frame to frame and only move objects from one bin to another when they cross the boundary of a cell.


Pixel dispersion (dissolving) algorithms

A very simple low memory approach

If you really want to use as little memory as possible, it can be done with not much more memory than that required to store a single image (the first frame) provided it is acceptable to do some preprocessing in advance.

If you copy the following jumbled image, this jsfiddle will take it as input:

It will then move the pixels around one step at a time so they drift into place to give the original image again:

This does not require any memory for tracking particles. The algorithm is very simple:

Pick a pixel at random
Swap it with one of its 4 neighbours
Repeat a million times

No additional memory is required as no particles are being used. We're simply swapping pixels at random, with no destination locations in mind. The key is in choosing that apparently jumbled initial image to have the pixels in exactly the right places such that applying a million random swaps will happen to leave all the pixels in exactly the right places to give the desired text.

Setting up the initial jumbled image

To do that, this preparatory jsfiddle takes an image as input and outputs a jumbled image that is precisely arranged to work as input for the main jsfiddle.

This one takes more memory, but is still a very simple algorithm:

Generate a large number of random numbers and store them
Using these numbers in reverse order* (using the last first):

Pick a pixel at random
Swap it with one of its 4 neighbours
Repeat a million times

This is exactly the opposite of what the main jsfiddle will do, so the jumbling will be completely reversed, restoring the original image. You can paste in a colour image, or if you want just text with only black and white pixels that will work too. The example image shown above is between these two extremes, as it has greyscale pixels for the antialiasing, which wander around alongside the black and white pixels.

A million pixel swaps works for fairly small images. For larger sizes this won't jumble them enough to completely obscure the original image, so you would need to adjust the number of swaps.

Languages without reseeding

Note that this only works because the pseudorandom number generator is reseeded at the start of both algorithms, to ensure that they are both working with exactly the same list of "random" numbers. Most languages will allow you to reseed the random number generator, making this an easy approach. JavaScript does not, so the jsfiddles include an implementation of xorshift to allow reseeding.

Even with a language that does allow reseeding, if the hardware is particularly limited then you may want to consider xorshift as it is very fast and uses only 4 bytes of memory for its state. It's also only a few lines of code.

*Note that the order isn't quite the same as just reversed. In this example code, three random numbers are used for each pair of pixels to be swapped. So the order of each triple must be kept the same to avoid choosing different pairs of pixels. For example:

[1, 2, 3], [4, 5, 6], [7, 8, 9]

must be reversed to

[7, 8, 9], [4, 5, 6], [1, 2, 3]

in order that the same x value, y value and direction be chosen for each pixel pair when running in reverse.


Multiplication of SH coefficient sets representing occlusion and color

I was overthinking it. The usual reconstruction for given direction and then multiplication will work fine probably.


Generate this kind of 2d burst or pulse algorithm?

Rather than using an image, I would suggest doing this kind of effect using a shader. I'm not familiar with Cocos2d-x, but some quick googling suggests that it can work with shaders. You'd use a pixel shader that calculates the distance of each pixel to the center of the pulse effect, then applies a function based on that distance to define the shape and appearance of the pulse.

I made a quick shadertoy as a proof of concept and example.


How can DSA functions be used without extension names on lower versions of OpenGL?

How is this possible?

Because ARB_direct_state_access is an extension. Note that the function/enumerator names in that extension lack the ARB suffix. There are dozens of ARB extensions that work like that. Suffix-less extensions were introduced in OpenGL 3.0 for compatibility purposes. You could write code that could work on OpenGL 3.0 or extensions that expose 3.0 functionality, without having to write different function calls.

The same goes here. You're not using OpenGL 4.5; you're using OpenGL 4.4+some extensions.


Why is thread safety such a huge deal for Graphics APIs?

The main gain would be that it would be easier to divide CPU tasks into multiple threads, without having to solve all the difficult issues with accessing the graphics API. Normally you either would have to make the context current (which might have bad performance implications) or provide a queue and call the graphics api in a single thread. I don't think that any performance is gained this way, because the GPU indeed processes them sequentially anyway, but it makes the developer job a lot easier.

The reason that it was not done until now probably is because directx and opengl were created in a time where multithreading was not really apparent. Also the Khronos board is very conservative in changing the API. Their view on Vulkan is also that it will coexist next to OpenGL, because both serve different purposes. It probably was not until recently that paralism became so important, as consumers get access to more and more processors.

EDIT: I don't mean that no performance is gained from doing work in multiple CPUs, it is not useful to split your calls into multiple threads to create textures/shaders faster. Rather the performance is gained due to having more processors busy and keeping the gpu busy with things to perform.


What are some new Real-Time Graphics Rendering Techniques to get closer to real life?

PBR isn't just a feature you "add to a rendering engine" but an entire philosophy how to approach solving rendering problems. This seems to be a common prevailing misconception when people talk about PBR. Quite often people assume that when they have implemented physically based BRDF (e.g. GGX) into their engines, they are "done implementing PBR". They are not - they are barely getting started.

Let's take some of the topics you list and how do you approach implementing them in the "physically based" way.

Depth of Field, Lens Flares, Motion Blur and Bloom & HDR for example. For years these effects have been implemented and slapped as post effect on the screen with some adhoc factors exposed to artists. However, as you know these are effects happening in the camera system and they should be exposed and implemented considering how the light interacts with this system. How camera aperture, shutter speed, ISO, camera sensor and lense system influence the effects and how they should be implemented and controlled considering these physical characteristics of the camera.

This same principle applies to all the aspects of rendering when you go PBR. For example shadows you mention, how would you implement shadows in physically based way? First of all you would have to consider all lights as area light sources as opposed to infinitesimally small lights that people have been doing for ages. Recently there has been push to implement physically based area lights in real-time which is an important step towards more complete PBR, but they are completely omitting proper area shadowing. This is a difficult problem in real-time rendering and there has been only little research done in the area of area shadows. "Real Time Area Lighting - Now and Next" by Sam Martin is a good thing to read if you are interested in the topic.

Talking about dynamic reflections or lighting in general. How should all the different lighting techniques such as SSLR, local/global environment maps, (textured) area lights, SSAO, GI, etc. be implemented and integrated together to form a cohesive lighting system? That you doesn't do double counting and that they are properly accounted in diffuse and specular lighting calculations.

When implementing real-time PBR algorithms, you will eventually have to introduce some bias due to performance constraints and resort to some kind of approximations. But the important thing is that you know what the real result should be and are aware of the compromises you are making. PBR is a paradigm shift from the classic artistic observation-based real-time rendering algorithm development towards more scientific way of solving the rendering problems and this shift in mentality and the direction it pushes you helps you to get more realistic results in the end.


Need help with change of basis (world to camera frame)

Since the transformation that is asked for includes a translation (the camera is located at a position other than the origin), you will indeed have to use homogeneous coordinates to describe this transformation in matrix form.

There are basically two ways to get to the matrix you're looking for. You can either set up the matrix that would transform from camera coordinates to world-space and then take the inverse of that, or you can build the matrix directly. The first approach is what is typically done and what the exercise seems to be aiming for since it explicitly asks for the base vectors of the camera system as seen from world-space, which is the first step in that process. The advantage of this approach is that it is arguably a bit more intuitive since what you're essentially doing is treat the camera as just another object that you place in world-space by setting up a matrix. Just that once you have that matrix, you invert it because you didn't actually want to get from camera-local coordinates to world coordinates but rather the other way around. I'll explain both approaches below using the concrete example given in your exercise. At least for now, I'll leave the generalization of the ideas presented below as an exercise to the reader (so that at least some of the fun remains for you ).

Let's look at a sketch of the whole situation:

$$
\def\mvec#1{\begin{pmatrix}#1\end{pmatrix}}
\def\ivec#1{\left(\begin{smallmatrix}#1\end{smallmatrix}\right)}
\def\mmat#1{\begin{bmatrix}#1\end{bmatrix}}
\def\vec#1{\mathrm{\mathbf{#1}}}
\def\mat#1{\mathrm{\mathbf{#1}}}
$$

Your camera is located at $\vec o = \mvec{3 & 3 & 0}^T$ and is supposed to be looking at the origin. By general convention, the camera is set up to be looking along the line of the z-axis of its local coordinate frame, while the local x-axis points right and the local y-axis points up. Therefore, we know that $\vec e_3$ should be pointing either at or away from the origin. Since you're talking about OpenGL, I assumed right-handed world and camera coordinate systems as is typically done in OpenGL-land. To get a right-handed coordinate frame where the local x points to the right, local y points up, and local z lines up with the viewing direction, $\vec e_3$ can only to be pointing in the direction away from the point at which the camera is looking. Once we have fixed $\vec e_3$, we have to define the remaining two base vectors. Note that no matter what we choose for $\vec e_1$ and $\vec e_2$, our camera will be looking at the origin. Our choice of $\vec e_1$ and $\vec e_2$ basically just fixes the rotation of the camera around the viewing direction, which is set by $\vec e_3$. Generally, one choses $\vec e_1$ such that it is orthogonal to both, $\vec e_3$, and a general "up" direction. $\vec e_2$ is then simply computed to be orthogonal to both, $\vec e_1$ and $\vec e_3$.

Approach 1

From the above sketch, it is immediately apparent that the coordinates of $\vec e_3$ have to be
\begin{equation}
  \vec e_3 = \frac{1}{\sqrt{3^2 + 3^2 + 0}} \mvec{3 \\ 3 \\ 0} = \frac{1}{\sqrt{2\cdot3^2}} \mvec{3 \\ 3 \\ 0} = \mvec{\frac{1}{\sqrt2} \\ \frac{1}{\sqrt2} \\ 0}.
\end{equation}
Since our camera is located in the x-y-plane in this example, the base vector $\vec e_2$ is just $\vec e_3$ rotated by $90°$:
\begin{equation}
  \vec e_2 = \mvec{-\frac{1}{\sqrt2} \\ \frac{1}{\sqrt2} \\ 0},
\end{equation}
and $\vec e_1$ just points in the direction opposite of the world-space z-axis:
\begin{equation}
  \vec e_1 = \mvec{0 \\ 0 \\ -1}.
\end{equation}
Using the world-space base vectors of the camera frame, we can write down a matrix that transforms from camera-space to world-space. The camera matrix that transforms from camera to world coordinates is then simply the inverse of that matrix:
\begin{equation}
  \mat M_{cam}^{-1} = \mmat{\vec e_1 & \vec e_2 & \vec e_3 & \vec o \\ 0 & 0 & 0 & 1}
         = \mmat{
    0 & -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 3 \\
    0 & \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 3 \\
    -1 & 0 & 0 & 0 \\
    0 & 0 & 0 & 1}.
\end{equation}
Since the camera frame has an orthonormal base just offset by $\vec o$, this matrix is composed of just the rotation defined by the orthonormal base vectors followed by a translation:
\begin{equation}
  \mat M_{cam}^{-1} = \mat T \mat R
\end{equation}
where
\begin{align}
  \mat R &= \mmat{
    0 & -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 \\
    0 & \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 \\
    -1 & 0 & 0 & 0 \\
    0 & 0 & 0 & 1} &
  \mat T &= \mmat{
    1 & 0 & 0 & 3 \\
    0 & 1 & 0 & 3 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1}.
\end{align}
You can convince yourself that this is true by simply performing the multiplication. Knowing this, we can compute the camera matrix:
\begin{equation}
  \mat M_{cam} = (\mat T \mat R)^{-1} = \mat R^{-1} \mat T^{-1}.
\end{equation}
Since $\mat R$ is composed of orthonormal base vectors, it is an orthogonal matrix, which means that its inverse is just its transpose. $\mat T$ is just a translation, its inverse is a translation in the opposite direction. Therefore,
\begin{equation}
  \begin{split}
  \mat M_{cam} &= \mmat{
    0 & 0 & -1 & 0 \\
    -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    0 & 0 & 0 & 1} \cdot \mmat{
    1 & 0 & 0 & -3 \\
    0 & 1 & 0 & -3 \\
    0 & 0 & 1 & 0 \\
    0 & 0 & 0 & 1} \\
    &= \mmat{
    0 & 0 & -1 & 0 \\
    -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & -3\sqrt2 \\
    0 & 0 & 0 & 1}.
  \end{split}
\end{equation}

Approach 2

An alternative to the approach above is to just directly compute the coordinates of the world-space base vectors as seen from the perspective of the camera. Given these coordinates, we can, again, just write down a matrix that applies the corresponding transformation from world to camera space.

Looking at the figure again, we see that the coordinates of the world-space x-axis as seen from camera space are
\begin{equation}
  \vec x' = \mvec{0 \\ -\frac{1}{\sqrt2} \\ \frac{1}{\sqrt2}}.
\end{equation}
Analogously, we find the camera-space coordinates of the world-space y- and z-axes:
\begin{align}
  \vec y' &= \mvec{0 \\ \frac{1}{\sqrt2} \\ \frac{1}{\sqrt2}} &
  \vec z' &= \mvec{-1 \\ 0 \\ 0}.
\end{align}
Also looking at the figure, we note that the world-space origin as seen from the camera is simply located at a distance of $\sqrt{3^2 + 3^2} = 3\sqrt{2}$ units along the negative $\vec e_3$ axis. Therefore, the camera-space coordinates of the world origin are
\begin{equation}
  \vec o' = \mvec{0 \\ 0 \\ -3\sqrt{2}}.
\end{equation} and we can, once again, write down the matrix
\begin{equation}
  \begin{split}
  \mat M_{cam} &= \mmat{
    0 & 0 & -1 & 0 \\
    -\frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & 0 \\
    \frac{1}{\sqrt2} & \frac{1}{\sqrt2} & 0 & -3\sqrt2 \\
    0 & 0 & 0 & 1}
  \end{split},
\end{equation}
and sleep well tonight because both ways did indeed lead to the same result…


In your first example, the call to WaitForSingleObject blocks the thread. This puts it into a waiting state, which takes it off the run queue and allows other threads to run on this CPU core. If no threads can run, the CPU will idle. When the fenceEvent completes, it wakes up this thread.

Your proposed replacement is called a spinlock. The thread keeps running, using 100% of a core until the fence's completed flag changes. No other code can run on this core, so there's no interleaving of CPU and GPU work. Worse yet, this code has to read from memory to check the flag every time around the loop, so it'll slow down any threads on other cores that are also accessing memory. Finally, because it's keeping the core and the memory bus active, it'll burn more power and generate more heat.

This is why fence events are needed: to enable the CPU to wait efficiently.


Collision detection methods for 2D and 3D environments

There is a book about real-time collision detection that contains a variety of algorithms for all sorts of intersection tests and lists a large number of collision detection algorithms. It's a decent overview if that's what you are looking for:

http://realtimecollisiondetection.net

https://books.google.com/books?isbn=0080474144


Is depth of field incongruous in a 3D still image?

In traditional stereo 3D, I don't believe that there is a way to make a fixed focal plane feel natural to the viewer. When looking at an out-of-focus object in stereo 3D, the object remains out-of-focus, causing conflicting cues. The lens in the eye tries to adjust to bring the object into focus, but of course it won't succeed, causing eye strain and headaches.

However, there is hope outside stereo 3D: Lightfield displays, such as this nvidia prototype, go a different route. In stereo 3D, the light in the scene is already captured by two virtual (or physical) cameras, "baking" in the focal plane. Head-mounted displays like Oculus Rift then attempt to tape two displays in front of your eyes in such a way that the retina receives the exact same image that was captured by the camera. Lightfield displays go a different route: Instead of capturing two images ahead of time, they reproduce the entire 4D light field in front of your eyes, allowing your eyes to capture the image as if they were sitting directly inside the virtual scene. This has a number of benefits, including much smaller and lighter hardware as well as giving your eyes the ability to refocus.

If there is a way to make lightfield displays technically and commercially viable, then I believe they can remove the need for depth of field and fixed focal planes entirely and make VR feel a whole lot more comfortable for the viewer. However, it is likely not possible to construct lightfield screens, so televisions and cinemas won't be able to use this technology.


I need advice and good reference in geometric modeling?

As a graphics professional, not a researcher, it's a bit hard to know what exactly is or is not part of geometric modeling, but looking at this page (http://graphics.cs.ucdavis.edu/~joy/GeometricModelingLectures/) it looks like it's just all about different ways to model surfaces, whether that's with triangles, mathematical descriptions, or other things.

If that is the case, yeah, there are still open problems.

Here is a journal with some papers on the topic:
http://jcgt.org/

I would also look at the SIGGRAPH conference topics over the years, there is a lot of that sort of thing there.

IMO you are going to find that the problems are broken up into two main groups:

Real Time Graphics
Non Real Time Graphics

Video games and real time simulation software has the primary need of needing representations that can be rendered in real time, and are also fairly efficient as far as size, so that they don't take up too much memory.

Modern day movies, and also "offline processes" (such as baking out lighting into textures to use in real time applications) use non real time rendering techniques to get high quality results.  Speed and memory still matter to them of course, but the restrictions are much more relaxed as it isn't uncommon for a single frame of a movie to take a day to render, using a cluster of machines in a rendering farm.

Both real time and non real time graphics also have needs beyond rigid body geometry though, and also are interested in finding good ways to model liquids, gasses like smoke, and other things.

Besides the obvious representations of geometry like triangle and quad meshes, or bezier and other analytical shapes, there is also a lesser known concept of the signed distance field which can be used for both 3d modeling as well as 2d vector graphics.  In both cases, when you zoom into these shapes they have inifinite detail - such as being completely smooth, or having a perfectly sharp corner or whatever else.

Here's a video about one way subdivision surfaces can be achieved in real time graphics, shown at SIGGRAPH 2016, regarding the call of duty game:
https://www.youtube.com/watch?v=PTGTviYwolE

If I were in your shoes, part of what I would research would be what the "demo scene" is up to.  By demo scene I mean the people who make 64KB executables that when you run them, display photo realistic movies and have music and audio as well.  Those folks have some very creative ways of doing things that often ties into cutting edge research topics.

Shadertoy.com is a place where you can write webgl pixel shaders and share them with others.  There is a lot of really interesting and exotic graphics stuff there, including signed distance field rendering, ray marching, ray tracing, path tracing, and more.
https://www.shadertoy.com/

Here's a demoscene website:
http://www.pouet.net/

Lastly, if I may humbly point you towards a paper I wrote, I came up with a technique for getting the GPU texture sampler to calculate Bezier curve points of arbitrary dimension for you as part of a texture lookup, using the non programmable N-dimensional texture sample linear interpolator.

I believe it can be expanded to support Bezier surfaces as well, but haven't yet looked into the details of that.

http://blog.demofox.org/2016/02/22/gpu-texture-sampler-bezier-curve-evaluation/


What is the difference between shading and shadowing?

In simplified terms, shading controls how object's surface's brightness changes when the angle between its normal and light vector changes. Shadows are areas where the light cannot reach because it's occluded by other objects. So no, they both are not responsible for creating shadows.


You need to make use of a discriminant, also when dealing with floating point comparisons it is wise to use an epsilon (small value for accommodating fp-error).
Also keep in mind that it matters whether the surface normal is pointing along the ray, or against it.
Below is a function which I use in my own raytracer, also based on the same book. It is written in C and refracts a given vector3. You would assign the return value to your ray.direction in your case.
// vector*vector means dot product here

#define EPSILON = 0.000001f

static inline vec3 refract(
    vec3 dir, // direction of incoming light ray
    vec3 n, // surface normal of refracting surface, pointing outwards (so roughly in the against the direction of light ray)
    float ni_over_nt // refractive indices fraction
){
    float dt = dir*n; // dot prod.
    float discriminant = 1.f - ni_over_nt*ni_over_nt*(1.f-dt*dt);
    if (discriminant > EPSILON){
        return (dir - n*dt)*ni_over_nt - n*sqrtf(discriminant);
    } else {
        // total internal reflection
        // reflect the direction against the surface normal
        return reflect(direction, normal);
    }
}

Usage of the function is as follows:
if (direction*normal) > EPSILON) // dot prod. of ray dir. and surf. normal
{
    // ray is entering the object
    nOut = -normal;
    ni_over_nt = mat->IOR; // surface material index of refraction.
} else {
    // ray is exiting the object
    nOut = normal;
    ni_over_nt = 1.f/mat->IOR;
}

vec3 refracted = refract(direction, nOut, ni_over_nt);
// this is your refracted ray direction

Added bonus: reflection function used in the refraction:
static inline vec3 reflect3(vec3 v, vec3 n){
    // reflect a vector v off of a plane with normal n
    return  v -n*2*(v*n); //v*n = dot prod.
}


Was interested in this myself and did some digging, I was wondering if they where spectral, but it appears not (although some interesting work on spectral BRDF is here http://web.cs.wpi.edu/~emmanuel/courses/cs563/write_ups/cliffl/cliffl_spectral_brdf.html )

This chat seems to indicate they are camera native linear, probably with sRGB primaries.

https://groups.google.com/forum/#!msg/pbrt/p9zMvZlaodA/4RD7-We8h4gJ


How to find Light Direction from a Point on a surface to an Area Light

If you want to use only one sample to approximate analytical area lighting (e.g. for real-time applications), you can use Most Representative Point (MRP) approximation as described in "Lighting in Killzone Shadow Fall", "Real Shading in Unreal Engine 4" and "Moving Frostbite to PBR". This works reasonably well for specular approximation, though the energy conservation is a challenge. What I have found working best is to find a point on a light source with smallest angle from the view reflection vector, and the solution depends on the shape of the light (for example here's my solution for rectangular lights).

For diffuse evaluation there are numerical and analytical approximations depending on light shape presented in the papers, which also properly handle the horizon case. You can also try to use MRP for diffuse lighting, but I haven't found good approximations for it. Depending your application though diffuse MRP might result in feasible quality.

There's also some more recent work for evaluating area lighting integrals not using MRP. "Accurate Analytic Approximations For Real-Time Specular Area Lighting" deals with the specular evaluation via edge integrals, though I recall this method is patented. "Real-Time Polygonal-Light Shading with Linearly Transformed Cosines" looks quite promising, but I haven't checked the paper in more details.

For non-real time applications and ray tracing area lighting is fundamentally and mathematically much simpler problem. For your integrator you just check the incident radiance at each sample (ray intersection with the scene) and accumulate results for total radiance towards the output direction. The challenge is how to reduce variance of your integrator, but getting basic area lighting to work in a ray tracer is relatively simple.


Hardware-accelerated drawing of curved shapes

You can use OpenGl 4.x tessellation shaders to convert Bezier control points into polygons.

A google search for "tessellation shader bezier" found this outline describing the tessellation of Bezier surfaces and curves:

http://web.engr.oregonstate.edu/~mjb/cs519/Handouts/tessellation.1pp.pdf

This offloads the Bezier evaluation from the CPU to the GPU and reduces the data flow across the bus.


kinematics and dynamics of a sphere in a spring mass system?

The rigid body simulation is very similar to the point mass case except for
that we have to handle rotational motion (orientation and angular momentum).

In addition to common particle attributes such as mass, position, and linear velocity, we will need to store initial moment of inertia $\mathbf{\hat{I}}$,
current angular momentum $\mathbf{L}^n$, and current orientation $\mathbf{R}^n$.
If you are using a data driven particle system, this step is as easy as declaring a few new attributes.

Contract to the mass, the moment of inertia at current time step $\mathbf{I}^n$ depends on the current orientation $\mathbf{R}^n$: $\mathbf{I}^n=\mathbf{R}^n \mathbf{\hat{I}} (\mathbf{R}^n)^T$.

The kinetic of a rigid body is analogous to the linear motion:
$$
\begin{align}
\dot{\mathbf{R}} &= \omega^* \mathbf{R} \\
\frac{\mathbf{R}^{n+1} - \mathbf{R}^n}{\Delta t} &= (\omega^{n*}) \mathbf{R}^n \\
\omega^n &= (\mathbf{I}^n)^{-1} \mathbf{L}^n
\end{align}
$$
where $\omega$ is the angular velocity and $\omega^*$ is the "cross product" matrix.

For dynamics, we also have a similar version of Newton's "f=ma":
$$
\frac{\mathbf{L}^{n+1} - \mathbf{L}^n}{\Delta t} = \tau^n
$$
where $\tau$ is the torque.

Because there will be interactions between rigid spheres, you will also need to implement the collision detection for spheres.

Now we have known the kinematics and dynamics of rigids. The only thing left is how to solve (integrate) above equations. The naive forward Euler may not give stable result. A good introduction of this topic can be found at Witkin and Baraff SIGGRAPH course, and here is an another more recent SIGGRAPH course by Bargteil and Shinar.


How can I transform an ellipse into a circle?

You can use a shear, or equivalently a rotation by 45, scale along one of the axes, rotate back (which is equivalent to the shear which you are looking for).

EDIT: On second thought, you don't need to even rotate back unless your circle is marked, since a circle is rotation invariant (if the rotation is around its center).


What is the best practice for data in a VBO?

If the color needs to vary per vertex, it will need to be an attribute in the vertex buffer. If it's acceptable to have the color be constant per draw call, then by all means make it a uniform.

By the way, editing a vertex buffer and resubmitting data to the GPU, while certainly slower than updating a uniform, doesn't have to be a huge inefficiency. By using GL_DYNAMIC_DRAW when you set up the buffer, combined with using glMapBufferRange and GL_MAP_INVALIDATE_BUFFER_BIT to update it, the driver can manage the buffer memory and transfers efficiently. That's how you would typically do a particle system, for instance, with the particles' positions, colors, etc. changing every frame. Furthermore, if only one or two vertex attributes need to change, you can split those off into their own buffer to decrease the total data updated.


How does a PenTile display controller interpret its RGB framebuffer?

I can't say for certain, but I was interested in this myself and trying to look it up (with little success). I did analyze the many macro photos you can find on the web of these kinds of displays, though.

RGB pixels      Pentile Subpixels (RGBG Layout)

--+---+--           --+---+--
X | X | X           G r G b G
--+---+--   -->     --+---+--
X | X | X           G b G r G
--+---+--           --+---+--

As far as I can tell, it looks like the green subpixels' locations exactly match the framebuffer pixel locations (X) and the intensity is transferred 1:1. The red and blue subpixels are shared between the two neighbouring pixels horizontally and their intensity seems to be the average of the red/blue channels of the two neighbouring framebuffer pixels left and right, effectively halving the horizontal resolution of R and B.

This strategy seems to carry over to the diamond Pentile layout even though the the red and blue subpixels are shifted downwards (or upwards) by half a pixel:

RGB Pixels       Diamond Pentile subpixels
                         _________
X   X   X                G | G | G
              -->        __r___b__
X   X   X                G | G | G
                         __b___r__

Due to the additional half pixel vertical shift for red and blue of this layout that isn't accounted for in driving the red and blue subpixels, you might be able to notice that a horizonal edge between a black area and a white area might appear green or magenta. You can see this effect in macro photos of diamond Pentile subpixel matrices rendering text. But there is actually no consistent shift. In some photos it's downwards and in others it's upwards.  I don't know if the displays were manufactured differently or just installed upside-down in some devices.

So, yes, you can easily control all subpixels of a Pentile display. Each color channel value in your frame buffer is used exactly once and for only one pixel. It's just that for the red and blue subpixels two frame buffer pixels' red and blue values are averaged.

Knowing this, you could try to do a more sophisticated subpixel rendering. But keep in mind that your typical color space like sRGB is nonlinear. The filtering should be done in a linear domain. For example, a checker board of alternating 0 and 255 will have the same brightness as a flat area with value around 185 (if I remember correctly) instead of 128.

And finally, I want to point out that it's possible to come up with anti-alias filters for non-rectangular sampling lattices. For example, for this checkerboard sampling lattice you have for the red and blue subpixels, the ideal lowpass filter would have a pass band in form of a diamond shape in the spectral domain. If you tried to reduce the resolution by dropping every 2nd sample in a checkerboard fashion, reasonable anti-alias filters would be these 5x5 and 3x3 kernels while the 5x5 version retains more detail:

 0   -1    0   -1    0
-1    0   10    0   -1              0  1  0
 0   10   32   10    0              1  4  1
-1    0   10    0   -1              0  1  0
 0   -1    0   -1    0

These can be applied in-place due to the zeros' locations.

You might also want to include a half-pixel shift for one or two dimensions in there to account for the relative positioning of the red blue and green subpixels. For example, for the diamond pentile layout you might want to use one of these filters:

  3    0    0    0    0    3
  0  -25    0    0  -25    0          -1   0   0  -1
  0    0  150  150    0    0           0   9   9   0
  0    0  150  150    0    0           0   9   9   0
  0  -25    0    0  -25    0          -1   0   0  -1
  3    0    0    0    0    3

to come up with the right red/blue subpixels instead of just averaging four values.

The RGBG (non-diamond) Pentile pattern would require only a half-pixel shift horizontally, leading to a filter like this:

 -3   3   3  -3
  2  30  30   2
 -3   3   3  -3

Some care must be taken with clipping due to the negative filter coefficients. For example, you could try to redistribute the residual to surrounding pixels to keep the average brightness.


Not enough triangles. The situation is analogous to the sampling theorem that states that you can not reconstruct a signal if your sample frequency is below a certain threshold. Although in this, case you're not attempting to rebuild the signal with a higher order filter so the result is even more dramatic. And it results in interpolation errors.

You can alleviate the problem with less points if your point wavefront follows your waves. So you may want to shift points towards the gradient maximum in other directions than UP.


This appears to be simply skeletal animation, which is a standard technique that is available in all modern animation packages. Whether applied to 3D meshes or (as here) 2D ones, the principle is the same.

The pseudo-3D effect on the sprites is created by using a mesh where vertices are placed along the contours of the sprite texture, then animated to squash and stretch parts the image in such a way that it creates the illusion of 3D motion, even though the sprite images are authored in 2D. There's a description of the technique in the Spine docs.


Max number of elements in a vertex attribute

The standard solution is to reduce the amount of per-vertex data by only specifying the 4 indices that have the highest weights (and rescaling the weights to add up to 1.0). That way, you can pass 4 weights as a vec4 and 4 indices as a uvec4. Also, you should pass the weights as normalized shorts (or bytes if you can accept the quality loss) and indices as bytes if at all possible, to further reduce the per-vertex data storage.
Most real-time rendering systems are happy with these limitations.
If that's not acceptable for your needs, then you're going to have to use multiple attributes to send the data. Each attribute can only have 4 components. And while attributes can be arrayed, arraying them like your C struct would waste a bunch of attribute locations, since each array element is considered a separate location.
So you need to send values in multiples of 4. Instead of sending 10 weights/indices, send 12, grouped into 4-element groupings. So the GLSL would look like this:
layout(location = X) uvec4 indices[3];    //Stores 12 indices, consumes 3 locations.
layout(location = X + 3) vec4 weights[3]; //Stores 12 weights, consumes 3 locations.

The C data structure ought to also store 12 elements for each array.
To specify them in the vertex format, you would do this (where X is the location given above):
glVertexAttribIPointer(3, 4, GL_INT, sizeof(Vertex), (void*)(offsetof(Vertex, groups) + 0));
glVertexAttribIPointer(4, 4, GL_INT, sizeof(Vertex), (void*)(offsetof(Vertex, groups) + 16));
glVertexAttribIPointer(5, 4, GL_INT, sizeof(Vertex), (void*)(offsetof(Vertex, groups) + 32));
glVertexAttribPointer(6, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)(offsetof(Vertex, weights) + 0));
glVertexAttribPointer(7, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)(offsetof(Vertex, weights) + 16));
glVertexAttribPointer(8, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)(offsetof(Vertex, weights) + 32));


Illumination components

If our scenes only contain point lights (e.g., omni lights, spotlights, etc.) and emissive surfaces, the illumination contributions at a surface position, $x$, are computed as follows:

The self emission associated with the emissive surface (i.e. 0 bounces/surface interactions) is computed as usual without using the scene's voxelization (as is the case for adding this illumination contribution to the scene's voxelization);
The direct illumination associated with the point lights (i.e. 1 bounce/surface interaction) is computed as usual without using the scene's voxelization (as is the case for adding this illumination contribution to the scene's voxelization);
The direct illumination associated with the emissive surfaces (i.e. 1 bounce/surface interaction) is obtained from the scene's voxelization using voxel cone tracing;
The indirect illumination associated with the point lights (i.e. 2 bounces/surface interactions) is obtained from the scene's voxelization using voxel cone tracing.

By computing the diffuse illumination for all the voxels from the scene's voxelization using voxel cone tracing, the diffuse illumination from additional bounces/surface interactions can be (optionally) accumulated inside the voxels. This requires each voxel to store (an approximation) to the normal distribution of the scene's surfaces overlapping those voxels as well.

Illumination from the scene's voxelization (3)-(4)

The outgoing radiance obtained from the scene's voxelization ($L_v$):

$$L_o\!\left(x, \hat\omega_o\right) = \int_\Omega f_{r}\!\left(x, \hat\omega_o, \hat\omega_i\right) L_v\!\left(x, \hat\omega_i\right) \left(\hat{n} \cdot \hat\omega_i\right) \mathrm{d}\hat\omega_i.$$

For a diffuse BRDF $f_{r}\!\left(x, \hat\omega_o, \hat\omega_i\right) = \frac{k_d}{\pi}$:

$$L_o\!\left(x, \hat\omega_o\right) = \frac{k_d}{\pi} \int_\Omega L_v\!\left(x, \hat\omega_i\right) \left(\hat{n} \cdot \hat\omega_i\right) \mathrm{d}\hat\omega_i.$$

For a partitioning of $\Omega$ in $N$ disjunct subdomains, $\Omega_j$, (e.g., $\approx$ cones) (i.e. $\Omega_1 \cup~...~\cup \Omega_N = \Omega$ and $\Omega_i \cap \Omega_j = \emptyset$ for each $i \ne j$):

$$L_o\!\left(x, \hat\omega_o\right) = \frac{k_d}{\pi} \sum_{j = 1}^{N} \int_{\Omega_{j}}  L_v\!\left(x, \hat\omega_{j,i}\right) \left(\hat{n} \cdot \hat\omega_{j,i}\right) \mathrm{d}\hat\omega_{j,i}.$$

Relying on a single cone with a direction, $\hat\omega_{j}$, and an aperture, $\alpha_{j}$, instead of individual rays with a direction, $\hat\omega_{j,i}$, for each subdomain, $\Omega_j$:

$$L_o\!\left(x, \hat\omega_o\right) \approx \frac{k_d}{\pi} \sum_{j = 1}^{N} L_v\!\left(x, \hat\omega_{j}, \alpha_{j}\right) \int_{\Omega_{j}} \left(\hat{n} \cdot \hat\omega_{j,i}\right) \mathrm{d}\hat\omega_{j,i}$$

$$\hat{W}_j = \frac{1}{\pi} \int_{\Omega_{j}} \left(\hat{n} \cdot \hat\omega_{j,i}\right) \mathrm{d}\hat\omega_{j,i}$$

$$L_o\!\left(x, \hat\omega_o\right) \approx k_d \sum_{j = 1}^{N} \hat{W}_j L_v\!\left(x, \hat\omega_{j}, \alpha_{j}\right).$$

We can use for example the following six cones, each with an aperture of $\frac{\pi}{6}$.

The normalized weight of the blue cone about the surface normal is equal to:

$$\hat{W}_{\mathrm{blue}} = \frac{1}{\pi} \int_{0}^{2\pi} \int_{0}^{\frac{\pi}{6}} \cos\!\theta \sin\!\theta \, \mathrm{d}\theta \, \mathrm{d}\phi = \frac{1}{4}.$$

The normalized weight of the other cones is approximately equal to:

$$\hat{W}_{\mathrm{purple|red|green|orange|brown}} \approx \frac{1-\hat{W}_{\mathrm{blue}}}{5} = \frac{3}{20}.$$

For a specular BRDF, one typically uses a single cone with a direction, $2 \left(\hat{n} \cdot \hat\omega_o\right) \hat{n}-\hat\omega_o$ (i.e. reflected direction of $\hat\omega_o$ about $\hat{n}$), and an aperture based on the roughness of the surface material.

Voxel Cone Tracing

$L_v\!\left(x, \hat\omega_{j}, \alpha_{j}\right)$ is computed using voxel cone tracing, which can be implemented by marching the mip-mapped 3D voxel texture in UVW texture space.

Our accumulated radiance (red, green, blue channels) and opacity (alpha channel) are initialized to zero.

Lv ← [0, 0, 0, 0]

Our cone marching distance is initialized to an offset to avoid sampling the self emission associated with the emissive surfaces and the direct illumination associated with the point lights twice. We need to basically skip the voxel containing the surface position for which the outgoing radiance needs to be estimated.

distance ← voxel_offset

Marching continues until we reach an accumulated opacity of 1 or more:

while Lv.alpha < 1:
    compute diameter (i.e. cone voxel footprint)
    compute mip_level
    compute position at distance
    if mip_level >= max_mip_level or position ∉ [0,1]^3
        break
    sample Lv_step(position, mip_level)
    Lv ← Lv + (1-Lv.alpha) * Lv_step // blend equation
    distance ← distance + cone_step  // marching -> diameter and mip_level will increase
return Lv


Creating a Smooth 3D Mesh from a 2D Outline

One algorithm that's pretty good for this, but very difficult to implement is to find the Medial Axis of the shape and then have various profiles that are based on the signed distance from the medial axis. You can also use the Straight Skeleton instead of the Medial Axis, but it's also a bit of a pain to calculate.

An easier way that's not quite as accurate, but usually good enough is described here: https://steamcdn-a.akamaihd.net/apps/valve/2007/SIGGRAPH2007_AlphaTestedMagnification.pdf. You can take a bitmap of the shape the user drew, shrink it down to some manageable size, calculate the distance of every pixel from the edges of the shape, then scale it to the size you need (and scale the distances accordingly). Once you have signed distances you can use them with the same 3D profiles mentioned above.

For example, the pillow in the video you showed is basically drawn as a square. Your 3D profile would set the front face height to 0 at the squares edges, and increasing up to some maximum as you moved towards the center, peaking after a short distance and holding up to the center. The back face height would be the same but in the opposite direction.


Angle between two points in Cartesian coordinate system C++

The parametric equation for a spiral is:

$$
\begin{eqnarray}
\begin{aligned}
    x &= &(a + b \theta) \times \cos(\theta)\\
    z &= &(a + b \theta) \times \sin(\theta)
\end{aligned}
\end{eqnarray}
$$

The direction on the spiral point is simply the derivative of the system with respect to $\theta$. Thus the direction is:

$$
\begin{eqnarray}
\begin{aligned}
 x_{dir} &=&-(a + b \theta) \times \sin(\theta) + b \times \cos(\theta)\\
 z_{dir} &=&(a + b \theta) \times \cos(\theta) + b \times \sin(\theta)
\end{aligned}
\end{eqnarray}
$$

Finally angle is is simply atan2 of the direction equation.

Image 1: This is what I get when i plot squares at $(x, z)$ with direction $(x_{dir}, z_{dir})$

Mathematica source for image


Converting cartesian pixels to polar pixels

I have implemented the cartesian-to-polar-conversion and have used different interpolation methods:

1) nearest neighbor

2) a subsampling approach, which averages 81 subpixel locations

3) bilinear interpolation

The 2nd row in the image below shows detail magnifications of the output for the three approaches:

Here is the GLSL shader code for approach 1 and 3:

precision mediump float;
varying vec2 tc; // texture coordinate of current output pixel
uniform sampler2D inputImg; // input image
const float PI = 3.141592653589793238462643383;
void main() {
  float phi = tc.x; // phi is x-direction in output
  float r = tc.y;   // radius is y-direction in output
  float xx = r * cos(2.0 * PI * phi);  // unit circle in range [-1.0, 1.0]
  float yy = r * sin(2.0 * PI * phi);
  float x = xx * 0.5 + 0.5; // input texture coordinate in range [0.0, 1.0];
  float y = yy * 0.5 + 0.5;
  gl_FragColor = texture2D(inputImg, vec2(x,y));
}

This is the GLSL shader code for approach 2:

precision mediump float;
varying vec2 tc; // texture coordinate of current output pixel
uniform sampler2D inputImg; // input image
uniform int outputWidth; // width of output image
uniform int outputHeight; // height of output image
const float PI = 3.141592653589793238462643383;
const int samplesPerSide = 4;
void main() {
  int samplesPerDirection = samplesPerSide * 2 + 1;
  // compute size of subsample step in texture coordinates
  float sampleStepX = 1.0 / float(outputWidth * samplesPerDirection);
  float sampleStepY = 1.0 / float(outputHeight * samplesPerDirection);
  vec4 sampleSum = vec4(0); // init sum to zero
  for(int i = -samplesPerSide; i <= samplesPerSide; i++) {
    for(int j = -samplesPerSide; j <= samplesPerSide; j++) {
      float phi = tc.x + float(i) * sampleStepX; // phi is x-direction in output
      float r = tc.y + float(j) * sampleStepY;   // radius is y-direction in output
      float xx = r * cos(2.0 * PI * phi);  // unit circle in range [-1.0, 1.0]
      float yy = r * sin(2.0 * PI * phi);
      float x = xx * 0.5 + 0.5; // input texture coordinate in range [0.0, 1.0];
      float y = yy * 0.5 + 0.5;
      sampleSum += texture2D(inputImg, vec2(x,y));
    }
  }
  gl_FragColor += sampleSum / float(samplesPerDirection * samplesPerDirection);
}

Approach 2, which is approx. factor 81 slower than 1 and 3, is probably the one you are looking for but I also like the bilinear interpolation result.

With a WebGL capable browser you can try out these implementations here:

https://www.gsn-lib.org/index.html#projectName=CartesianToPolar&graphName=CartesianToPolar


Using depth texture to mask local pixels

Use the depth texture you mentioned, comparing it to the fragment's current depth to apply an alpha factor to your color. Remember to enable blending on the host. You can also upload the color texture and apply a more controlled blending on the shader.

The shader code involved would look like this:

#version 430 core

out vec4 finalColor;

...

in vec2 screenUV;

...

uniform sampler2D depthTexture;

...

void main()
{
    float texDepth = texture(depthTexture, screenUV).r;
    float alpha = texDepth < gl_FragDepth? 0.5 : 1.0;

    ...

    finalColor = vec4(..., alpha);

}


The GLSL Specification in section 6.1.1 "Function Calling Conventions" states:

The keyword in is used as a qualifier to denote a parameter is to be copied in, but not copied out.

The keyword out is used as a qualifier to denote a parameter is to be copied out, but not copied in. This should be used whenever possible to avoid unnecessarily copying parameters in.

The keyword inout is used as a qualifier to denote the parameter is to be both copied in and copied out. It means the same thing as specifying both in and out.

A function parameter declared with no such qualifier means the same thing as specifying in.


Image based lighting, tangent space coordinates, and optimization

Yes, you do need to transform the fetch direction into the space of the cubemap. If you could somehow figure out the fetch direction in the vertex shader, then you could do the transformation there instead, but that would produce worse lighting.

It also may be worth optimizing for a smaller number of interpolants between the vertex and pixel shaders (rather than a smaller number of matrix-vector multiplications in the pixel shader). As soon as you have one transformed view vector and two transformed light vectors that must be interpolated (which would otherwise be constant), you've matched the size of the tangent/bitangent/normal matrix that you would otherwise be interpolating. That has the potential to have more performance impact than doing extra matrix multiplications in the shader.


This is just a visualisation. Type the command vis scenedepthz uv0 to see the actual depth buffer used. UE4 uses a "reversed" depth buffer for the scene.


Unable to pass custom Matrix4 to GLSL as a uniform

glUniformMatrix4fv(transformLocation, sizeof(transform), GL_FALSE, &transform.m[0][0]);

The second parameter to glUniformMatrix is not the number of bytes in the data. OpenGL can figure that out from the 4fv part of the command (a 4x4 matrix of GLfloats). The second parameter is the number of matrices you're sending to that uniform, in case it is an array of mat4s.

So that value ought to be 1. You almost certainly got a GL_INVALID_OPERATION error from calling it with sizeof(transform).


Why is the valence of regular vertices 6?

If require that all faces have the same number of sides $s$ and require that all vertices also have a certain valency $t$. We see that the following relation between edges, and faces hold for a regular mesh:
$$s\cdot f = 2e,$$
$$t\cdot v = 2e.$$ Substitution in the Euler-Poincare formula yields:
$$\left(\frac{1}{s} + \frac{1}{t} - \frac{1}{2}\right)e = 1 - g$$
If we then take for instance a regular plane which can be said to have the topology of a torus with genus $g = 1$. We then set the valency of faces $s = 3$ then
$$\left(\frac{1}{3} + \frac{1}{t} - \frac{1}{2}\right)e = 0,$$
$$\left(-\frac{1}{6} + \frac{1}{t}\right)e = 0.$$
The solution for $t$ is $6$, which says the valency of a vertex in a regular triangulation is 6.


I don't think it's trying to represent anything in particular.

In CG there is a concept of textures, which is a bit like painting the surface of an object a specific color/look. For example, on a car you may want to paint scratches and dust on the surface.

Here they have created a human model from their capture data, and when visualising it they have decided to texture/paint a red/white checker pattern on top.

They have most likely added it for the viewer's benifit, as it helps emphasise the curves and overall shape of the model, esspesially as it animates. They also probably chose contrasting colors so the model was easier to view on a white background.


This question is somewhat related to this one.

As Alan has already said, following the actual light path through each layer leads to more physically accurate results. I will base my answer on a paper by Andrea Weidlich and Alexander Wilkie ("Arbitrarily Layered Micro-Facet Surfaces") that I have read and partially implemented. In the paper the authors assume that the distance between two layers is smaller than the radius of a differential area element. This assumption simplifies the implementation because we do not have to calculate intersection points separately for each layer, actually we assume that all intersection points over the layers are just the same point.

According to the paper, two problems must be solved in order to render multilayered material. The first one is to properly sample the layers and the second is to find the resulting BSDF generated by the combination of the multiple BSDFs that are found along the sampling path.

UPDATE: Actually I have adopted a different method to implement the evaluation of this layered model. While I have kept with the idea of considering the intersection points to be just the same point along the layers, I have computed the sampling and the final BRDF differently: for sampling, I have used ordinary ray tracing, but through the layers (using Russian Roulette to select between reflection/refraction when that's the case); for the final BRDF evaluation, I just multiply each BRDF traversed by the ray path (weighting the incident radiances according to the cosine of the incident ray).

Sampling

In this first stage we will determine the actual light path through the layers. When a light ray is moving from a less dense medium, e.g. air, to a denser medium, e.g. glass, part of its energy is reflected and the remaining part is transmitted. You can find the amount of energy that is reflected through the Fresnel reflectance equations. So, for instance, if the Fresnel reflectance of a given dielectric is 0.3, we know that 30% of the energy is reflected and 70% will be transmitted:

When the light ray is moving from a denser to a less dense medium, the same principle described by the Fresnel reflectance applies. However, in this specific case, total internal reflection (a.k.a TIR) might also happen if the angle of the incident ray is above the critical angle. In the case of TIR, 100% of the energy is reflected back into the material:

When light hits a conductor or a diffuse surface, it will always be reflected (being the direction of reflection related to the type of the BRDF). In a multilayer material, the resulting light path will be the aggregate result of all those possibilities. Thus, in the case of a 3-layer material, assuming that the first and second layers are dielectrics and the third layer is diffuse, we might end up, for instance, with the following light path (a tree actually):

We can simulate this type of interaction using recursion and weighting each light path according to the actual reflectance/transmitance at the corresponding incident points. A problem regarding the use of recursion in this case is that the number of rays increases with the deepness of the recursion, concentrating computational effort on rays that individually might contribute almost nothing to the final result. On the other hand, the aggregate result of those individual rays at deep recursion levels can be significant and should not be discarded. In this case, we can use Russian Roulette (RR) in order to avoid branching and to probabilistic end light paths without losing energy, but at the cost of a higher variance (noisier result). In this case, the result of the Fresnel reflectance, or the TIR, will be used to randomly select which path to follow. For instance:

As can be seen, TIR or Fresnel reflectance might keep some rays bouncing indefinitely among layers. As far as I know, Mitsuba implements plastic as a two layer material, and it uses a closed form solution for this specific case that accounts for an infinity number of light bounces among layers. However, Mitsuba also allows for the creation of multilayer materials with an arbitrary number of layers, in which case it imposes a maximum number of internal bounces since no closed form solution seems to exist for the general case. As a side effect, some energy can be lost in the rendering process, making the material look darker than it should eventually be.

In my current multilayer material implementation I allow for an arbitrary number of internal bounces at the cost of longer rendering times (well... actually, I've implemented only two layers.. one dielectric and one diffuse :).

An additional option is to mix branching and RR. For instance, the initial rays (lower deep levels) might present substantial contribution to the final image. Thus, one might choose to branch only at the first one or two intersections, using only RR afterwards. This is, for example, the approached used by smallpt.

An interesting point regarding multilayered materials is that individual reflected/transmitted rays can be importance sampled according to the corresponding BRDFs/BTDFs of each layer.

Evaluating the Final BSDF

Considering the following light path computed using RR:

We can evaluate the total amount of radiance $L_r$ reflected by a multilayer BSDF considering each layer as a individual object and applying the same approach used in ordinary path tracing (i.e. the radiance leaving a layer will be the incident radiance for the next layer). The final estimator can thus be represented by the product of each individual Monte Carlo estimator:

$$ L_r =  \left( \frac{fr_1 \cos \theta_1}{pdf_1}  \left( \frac{fr_2 \cos \theta_2}{pdf_2} \left( \frac{fr_3 \cos \theta_3}{pdf_3} \left( \frac{fr_2 \cos \theta_4}{pdf_2} \left( \frac{L_i fr_1 \cos \theta_5}{pdf_1} \right)\right)\right)\right)\right)$$

Since all terms of the estimator are multiplied, we can simplify the implementation by computing the final BSDF and $pdf$ and factoring out the $L_i$ term:

$$fr = fr_1 \cdot fr_2 \cdot fr_3 \cdot fr_2 \cdot fr_1$$

$$pdf = pdf_1 \cdot pdf_2 \cdot pdf_3 \cdot pdf_2 \cdot pdf_1$$

$$\cos \theta= \cos \theta_1 \cdot \cos \theta_2 \cdot \cos \theta_3 \cdot \cos \theta_2 \cdot \cos \theta_1$$

$$ L_r =  \left( \frac{fr \cos \theta}{pdf} \right) L_i$$

The paper by Andrea Weidlich and Alexander Wilkie also takes absorption into consideration, i.e. each light ray might be attenuated according to the absorption factor of each transmissive layer and its thickness. I've not included absorption into my renderer yet, but it is represented by just one scalar value, which will be evaluated according to the Beer's Law.

Alternate approaches

The Mitsuba renderer uses an alternate representation for multilayered material based on the "tabulation of reflectance functions in a Fourier basis". I have not yet dig into it, but might be of interest: "A Comprehensive Framework for Rendering Layered Materials" by Wenzel Jacob et al. There is also an expanded version of this paper.


Tone Mapping in a physically based rendering ray tracer

That's just how the Reinhard operator works. If the scene has very high dynamic range important detail may be lost near the high luminance region as you found since both will map near 0.99. Reinhard is a form of global operator. There are other types of algorithms using local operators which tonemap the pixel based on the intensity of the underlying neighborhood like a 3x3 or 5x5 region.

I'd recommend reading HDR Imaging Acquisition, Display and Image Based Lighting by Erik Reinhard. It's a very good book and explains much of the concepts.


BRDF for point lights should not return values over 1

I think a single direction can be greater than 1.
For a BRDF the integral over the hemisphere must be less than 1 for energy conservation.
If you consider a specular brdf there is only one direction, where brdf > 0 and all other directions return 0. Therefore when integrating over the hemisphere that direction must be much greater than 1.


What's wrong with my computation of the intersection of a ray with a sphere

This line doesn't look quite right:

float t = (-b - sqrt (det)) / 2.0 * a;

You're missing parentheses around (2.0 * a). Remember that multiplication and division have equal operator precedence, so the expression as it's currently written means ((-b - sqrt (det)) / 2.0) * a, putting the a in the numerator instead of the denominator.

(Oh and this is a nitpick, but the thing you named det is called a “discriminant”, not a determinant.)


How does DirectX 12 SLI VRAM stacking work?

In DX12, there is no implicit driver-implemented SLI as there was in DX11. Instead, multiple GPUs are exposed to the application as separate "nodes" within a single DX12 device, and each VRAM resource lives on a single node, specified at creation time. There is no implicit mirroring of resources to both GPUs as in DX11 SLI.

So, the game engine or application has full control and responsibility over how data and work are distributed between the GPUs. It's up to the app developers to implement patterns like alternating frames between GPUs, or splitting the frame across GPUs, if they wish. For example, to implement alternating frames, the app would have to allocate all buffers, textures, render targets etc. on both GPUs and populate both of them. Then, when rendering a frame, it would generate a command list for the current GPU using its local copies of all resources. Any inter-GPU transfers needed (for temporal effects, for instance) would also be the app's responsibility.

More information on this topic can be found on this MSDN article on DX12 multi-GPU support.


Compute shader textures compared 2D float4 arrays

It depends on what you’re doing. If any given thread of your shader is working with data from multiple nearby pixels, like by sampling from fractional between-pixel locations or doing some calculation involving a pixel’s immediate neighbors, then you should probably use a texture: the GPU can do some caching and even load multiple values in a single operation in that case. If your threads are more independent and working on individual values, an array is fine.


Saving a mesh and textures into a single file

glTF is capable if embedding both geometry and texture data in a single file. It also has binary version (*.glb). There are several open-source savers/loaders, one of them being header-only tinygltf by Syoyo Fujita.


Remove orphaned geometry after Constructive Solid Geometry operation

I've not done CSG per se, but I did work on a, not-unrelated, problem of triangulation of arbitrary, self-intersecting, multi-contoured polygons. If you don't get the implicit intersections correct, all sorts of nasty artefacts can result.

Initially, I tried using standard, floating-point maths to compute intersections etc, but this is fraught with precision problems and no amount of trickery of ordering etc eliminated these accuracy issues.

In the end, I replaced all intersection calculations with rational maths (high precision integer numerator and denominators) with sufficient precision that all intersections would be represented exactly. Once the process was completed, then these high precision results could be reduced to back to floating point.

Perhaps a quick way to start would be to use a package such as GNU's GMP which supports arbitrary rational representations.