In the following, I suggest that, before using the generic $T(\alpha_1 x_1+\alpha_2 x_2)$ versus  $\alpha_1 T( x_1)+\alpha_2T( x_2)$, it can be more informative to try with simpler partial tests, or try counterexamples, based on your intuition.

I don't really understand the motivation behind the second group of equations, or why $b$ gets multiplied by $\alpha_i$.

Here, you are using the generic version of the linearity test, which is good, but can raise some doubts, as apparent from your question. You can, in complement, use other simpler tests, that can show that the system is not linear, as a double check. Those are counter-examples. For instance:

is the output of the zero signal zero? For a linear system ,  $T(\vec{0})=0$.
is a single  output  with a zero coefficient  zero? This tests if $T(0.\vec{x})=0.T(\vec{x}) = 0$
is a single  output linear? This tests if $T(\alpha.\vec{x})=\alpha.T(\vec{x})$
is a simple addition   linear? This tests if $T(\vec{x}+\vec{y})=T(\vec{x})+T(\vec{y})$

Those, if not passed,  prove that the system is non-linear. And instead of using the generic version, they can show to your (clever) professor that you have some intuition about what is going on, and the risk of errors is reduced. I personally  appreciate a lot when students use minimal arguments: they go straight to the point, spend less time on such questions, to focus on more involved ones.

Of course, if the system is linear, more is required.

Here, your system is non-linear... unless $b=0$.


Circular convolution is just linear convolution aliased by DFT length $n$. The length of linear convolution of $a$ and $b$ will be $2n-1$. So take $FFTs$ of $a$ and $b$ , padding each of them to length nearest power of 2 more than or equal to $2n-1$. Multiply the corresponding $FFTs$ point by point to get a power of 2 length sequence and take $IFFT$ of it. This sequence is actually the linear convolution of $a$ and $b$ since we had done enough padding before taking their individual $FFT$. Let this sequence be named $c$. Now, alias in time domain by shifting copies of $c$ by $n$ and adding them on top of each other.
$$
d[m] = \sum_{k=-\infty}^{k=+\infty} c[m - nk]
$$
The final output you want is $d[m]\,\text{for}\, 0\le m \le n-1$


complex multiplier in divide and combine FFT

Now as the document says, let $x[n]$ be a length $N$ (even) sequence whose even and odd indexed samples are denoted as $g_1[n]$ and $g_2[n]$ of length $N/2$ each.

Then the $N$-point DFT $X[k]$ of $x[n]$ can be written as folows:

$$\begin{align}
X[k]  &=  \sum_{n=0}^{N-1} x[n] e^{ -j \frac{2\pi}{N} n k} ~~~,~~ k = 0,1,...,N-1 \\\\
&=  \sum_{n=0,2,4}^{N-1} x[n] e^{ -j \frac{2\pi}{N} n k} + \sum_{n=1,3,5}^{N-1} x[n] e^{ -j \frac{2\pi}{N} n k}\\\\
&=  \sum_{m=0}^{N/2-1} x[2m] e^{ -j \frac{2\pi}{N} 2m k} + \sum_{m=0}^{N/2-1} x[2m+1] e^{ -j \frac{2\pi}{N} (2m+1) k}\\\\
&=  \sum_{m=0}^{N/2-1} g_1[m] e^{ -j \frac{2\pi}{N/2} m k} + e^{ -j \frac{2\pi}{N} k} \sum_{m=1}^{N/2-1} g_2[m] e^{ -j \frac{2\pi}{N/2} m k}  \\\\
&=  \left( \sum_{m=0}^{N/2-1} g_1[m] e^{ -j \frac{2\pi}{N/2} m k} \right) + e^{ -j \frac{2\pi}{N} k} \left( \sum_{m=1}^{N/2-1} g_2[m] e^{ -j \frac{2\pi}{N/2} m k} \right) \\\\
&=  G_1[k] + W_N^k G_2[k] ~~~,~~ k = 0,1,2...,N-1 \\
\end{align}$$

Where we have recognized the summations inside the parenthesis as the $N/2$ point DFTs of the sequences $g_1[n]$ and $g_2[n]$ respectively. Note that when $k$ spans the range $k=0,1,...,N-1$, the $N/2$ point DFTs will repeat twice.


When utilizing MFCCs, it is common to split the MFCC matrix into pieces. Lets assume your MFCC-matrix $\bf{M}$ has the dimensions $[\bf{M}]$$=N \times 39$, where $N$ denotes your time index and you have 39 MFCC bins. Then you cut your matrix into blocks of length $B$ (I propose $B=39$ so your blocks have a square shape) and save them in a list. For the last block, you can either zero-pad until it has the same length or discard it completely. Each block has to be annotated with the label of the original audio sample.

A standard approach for classification is a Deep Neural Network, that consists of Convolutional and adjacent Pooling layer since MFCCs are like images locally connected/correlated and Convolutional layers take care of that. Of course, you should not forget to add an MLP layer at the end with a softmax activation to get an estimated probability distribution as an output of your net as well as an appropriate regularization technique.

A more advanced approach would be the use of a recurrent network structure like an LSTM-gate or a plain Recurrent layer. Therefor, you have to store the blocks of a sample in chronological order so the network can learn the temporal structure of a whole sample.

EDIT: Actually one often takes only 13 MFCC bins. The $N \times 39$ matrix also contains the Deltas and DeltaDeltas of the MFCC bins.


Amplitude value after a Discrete-Time integration calculation?

I figured it out coming back to basic maths, I  was making a mistake when integrating my sin function that Matlab block wasn't doing, I was missing behind the $\omega$ variable

$$ve[n] = -B \cdot \cos[\Omega\cdot n] = \int A\sin(\omega\cdot t) = (A/\omega)(-\cos(\omega\cdot t)) + Const$$ $$\mathbf {B = (A/\omega) }$$


How does ICA handle inevitable delays in signals?

One of the most successful uses of ICA has been in the study of electrophysiology (i.e. brain activity), principally EEG (Electroencephalography) and MEG (Magnetoencephalography). They are used to remove artefacts (such as electrical impulses caused by muscle movements (eye blinks etc)) without the need for reference channels. In this application the spatial separations between sensors is minute compared with the propagation speed of the waves, and as such the assumptions of ICA effectively hold.

For fMRI, which relies on blood flow in the brain, the temporal delay issue is more significant. One approach, taken in the paper Latency (in)sensitive ICA. Group independent component analysis of fMRI data in the temporal frequency domain by Calhoun et al (2003) attempted to solve this problem by making estimates of the time delay in each voxel, and then using this as prior information in a modified ICA. Maybe something like this could be applied in your domain?


How to choose sampling rate for non-band limited signal given as an expression?

It depends on the spectral distribution of the signal under concern. If, for example, as you said the signal has most of its energy around zero frequency and gets progressively smaller as frequency increases then it may be possible to sample it with an acceptable error, provided that the progress is rapid enough. So it depends on the progress: Provided that it decays fast enough, you may find an acceptable sample rate which would provide a bounded error. After all many practical signals will have nonzero (but very small) energy above their half the sampling rate even after aa filtering, but nevertheless they are sampled with sufficient accuracy.

And your bit resolution vs stop band attenuation concern. That is the noise floor of the N-bit representation. Any error less than noise floor, and being uncorrelated with the signal, may be considered ok.

Finaly please note that, when the aliased frequencies fold up (say at 0 hz) for every next shifted spectra to be added, the folded tail will be progressiveley smaller, as it comes from progressiveley farther away from its center. Hence the sum of such infinitely many decaying terms may converge. Provided the folded terms are getting smaller and smaller.


Complexity: Number of MAC operations on a DSP

It appears to me that your analysis is basically correct, except that in the second case there are $\frac{N}{2}$ sums, not $N$ sums.

I'm not sure what you are looking for when you ask "how a DSP deals with complex numbers", but generally they are represented in one of two ways: as separate real and imaginary values, or as a "complex" entity (structure/object/whatever- depends on the language) that contains real and imaginary members.


Impulse response of a time scaling system

The input-output relation of such a system can be written as

$$y(t)=\int_{-\infty}^{\infty}\delta(t/2-\tau)x(\tau)d\tau\tag{1}$$

Note that the system is linear but time-varying, and such systems can generally be described by a two-dimensional impulse response $h(t,\tau)$:

$$y(t)=\int_{-\infty}^{\infty}h(t,\tau)x(\tau)d\tau\tag{2}$$

In the given example we simply have

$$h(t,\tau)=\delta(t/2-\tau)\tag{3}$$

For linear time-invariant (LTI) systems, $h(t,\tau)$ only depends on the difference $t-\tau$.


Non-separable, higher-order generalized cosine windows in 2D for images

The form:

$$w(x, y) = \sum_{j = -K}^{K}\sum_{k = -K}^{K} a_{|j|,|k|}\; \cos(\pi jx + \pi ky),$$

where:

$$a_{j,k} = a_{k,j},\\
\sum_{j = -K}^{K}\sum_{k = -K}^{K} a_{|j|,|k|} = 1,\\
x \in -1..1,\\
y \in -1..1$$

yields window functions that are 2D similar. Window functions of that form are not generally separable, but some of them are, for example:

$$w(x, y) = \frac{\cos(\pi x - \pi y)}{8} + \frac{\cos(\pi x + \pi y)}{8} + \frac{\cos(\pi x)}{4} + \frac{\cos(\pi y)}{4} + \frac{1}{4}\\
= \left(\frac{\cos\left(\pi x\right)}{2} + \frac{1}{2}\right)\left(\frac{\cos\left(\pi y\right)}{2} + \frac{1}{2}\right)$$

For this window function, if the balance between the zero-frequency term and the higher terms is altered (as in going from Hann to Hamming), separability breaks.


Your signals look quite similar to vowel signals - in which a periodic pulse train is filtered by a resonant filter.

I suggest you to look into AR modelling to decompose your signal into a residual, and a set of AR coefficients. The spectrum of the residual will give you the "pulse train" component of your signal (a sharp peak every time a bubble is produced) ; while the frequency response of the all-pole filter associated with the set of AR coefficients will contain information about the sound of each bubble. The 167 Hz peak you are looking will be easy to spot on the later.


Converting Gamut3D from RGB colorspace to LAB colorspace

The so-called Gamut in the code, is nothing but a 3d color histogram. Colors from the image are collected into it, and then each unique (r,g,b) is being shown as a scatter point.

In order to show the distribution of the colors in LAB space, you can convert your image from sRGB to LAB, (Assuming that the image is in sRGB space) and then run the same procedure that you have now.

In order to convert it, use:

    cform = makecform('srgb2lab');
    labImage = applycform(cform, rgbImage);

By the way, the code in your question can be optimized into 4 lines:

    im = imread('peppers.png');
    im = reshape(im,[],3);
    im = unique(im,'rows');
    scatter3(im(:,1),im(:,2),im(:,3));


The MP3 encoder works on batches of 576 time-domain samples and converts them to 576 frequency-domain samples. That means that you get a frequency resolution of $\frac{f_\text{sample}}{576}$, whatever your sampling rate is.

Notice however how short that is at usual sampling rates, for example 44.1 kHz: that's a bare 13.something milliseconds. That makes a lot of sense – the frequencies produced by an excited string, membrane or air volume change over time after the excitation.

So, no, your approach won't work: this resolution won't work to resolve your 2 Hz!

Also, not going through the MP3 route and directly FFT'ing your audio signal can't work, either:

To get a 2 Hz resolution in a linear frequency transform, you'll need 1/2 seconds of audio, no matter how you do it!

Do the math: if you have a sampling rate of 44.1 kHz, and you want to resolve 2 Hz, you'll need to transform 44.1 kHz / 2 Hz = 22050 samples. These take 22050 / 44.1 kHz = 1/2 s to accumulate at that sample rate. It's the same for every other rate, too, because the sampling rate in the equation

$$T_{\text{recording}}=\frac{\frac{f_{\text{sample}} }{f_{\text{resolution}}}}{f_{\text{sample}}}$$

always cancels.

Now, you'll notice that a note lasting half a second in its pure form rarely happens.

Furthermore, as I tried to explain in my comments:

Your hypothesis that playing a note inserts power at but a single frequency is false.

Instruments playing a note don't produce a single tone, but a mixture of multiple harmonics dampened by a changing envelope, also exposed to a drift in frequency over time. It's really not like you can go into a FFT and say, "OK, this was this and that chord played, easy to see".

Things get even more complicated since you said you wanted to detect notes played in a chord: Since the individual oscillating elements (e.g. guitar strings) mechanically couple, you typically get further frequency components and a changed temporal behaviour.


How to convolve an arbitrary signal with a causal decaying exponential?

Indeed you have reached what can be reached, may be the following additional line can be obtained by moving the $t$ function $e^{-2t}$ out of the integral and replacing the $u(t-\tau)$ by $1$ as:

$$
y(t) = e^{-2t} \int\limits_{-\infty}^t {x(\tau)e^{2\tau}{}d\tau}
$$

In the general case you cannot simplify it any further...


Reconstruction of voice from distorted recordings

This is a difficult problem. The wall itself is simply a linear time invariant filter. Typically it's just a low pass filter and can theoretically be compensated with a matching inverse filter.

However in practice the attenuation at most frequencies is quite severe. Unless the recording room is extremely quiet and the recording extremely low noise, the remaining speech signal will be below the noise floor and hence impossible to recover.


Two signals with the same frequency in the same area can interfere?

I would say yes almost certainly - it would depend on transmit power but if they are simultaneously broadcasting on the same frequency then there will be interference as they are in the same area. This is the reason valuable parts of the spectrum must be licensed by government agencies, to prevent multiple people using them at the same time without multiplexing etc.


Expectation of product of hermitian matrix

You could define a diagonal matrix $\mathbf{D}$ with diagonal elements $d_{ii}=a_i+b_i$. Then vector $\mathbf{h}$ can be written as

$$\mathbf{h}=\mathbf{D}\mathbf{c}$$

and the expectation is

$$\mathbb{E}[\mathbf{h}\mathbf{h}^H]=\mathbf{D}\mathbb{E}[\mathbf{c}\mathbf{c}^H]\mathbf{D}^H$$

(I assume that $\mathbf{a}$ and $\mathbf{b}$ are deterministic and $\mathbf{c}$ is random, because you said you wanted the expectation in terms of $\mathbf{c}$.)


Conversion from laplace transform to z-transform

Your question makes no sense. Z transform is performed on a discrete signal/series.

Since $H(s)$ is a continuous function, you can't just calculate a Z-transform of $H(s)$ without first sampling it, to make it discrete. Also, it doesn't make much sense to do a time->spectrum transform (such as a Z-transform) on a spectral representation ($H(s)$)

I'm assuming that when you write "$Z-Transform(H(s))$", what you really want to do is to convert $H(s)\to H(z)$, meaning to calculate the Z-transform of $h[nT]$, where $h[nT]$ is $h(t)$, sampled at intervals of $T$, and $h(t)$ is the inverse Laplace transform of $H(s)$.

If I'm correct in my assumption, the transformation you are seeking is known as "star transform", which would provide a transform function $H^{*}(s)$, in terms of $e^{sT}$, which may be easily converted to $H(z)$ by way of the substitution $z=e^{sT}$.

Edit - some elaboration on the conversion process: what you need to do is calculate $H^{*}(s)$ from $H(s)$ using one of the two relations described in "Relation to Laplace transform" in the Wikipedia article, then do the substitution $z=e^{sT}$, to get $H(z)$.


Regarding MFCC feature of a speech signal

You can have as many MFCC coefficients as you want, 12 is just only a widely used number. As you probably know (or if not then please refer to the old answer), coefficients are being obtained via fitting of cosinusoids to log energies in your filter banks. Usually 12 is enough, more is not improving recognition rate too much. It is better to calculate their derivatives ($\Delta $'s and $\Delta\Delta$'s across time). Regarding names - they are simply MFCC's (first, second, and so on). Although coefficient $c_0$ is usually called the energy.
Question about range is not straightforward to answer. Usually in my applications they are in $[-12, \ 12] $. Although if you perform the Cepstral Mean Normalisation on yours, then you can expect them to have values of Gaussian Distribution with $95\%$ of them being in range of $[-2, \ 2] $.
I do not really understand that question - you must be more specific about what you are asking. Edges of what?


A random sequence of +1 and -1 is not really a BPSK signal; it's just a sequence of bits. As such, its FFT has not much meaning, at least not in the sense you're thinking of.

To create an actual BPSK signal, first denote your sequence of $N-1$ bits by $a_k$, where $k=0,\ldots,N-1$ is an integer. Second, choose a bit rate $R_p=1/T_p$. Then, choose a pulse shape, $p(t)$. The pulse shape is typically rectangular, a sinc, a raised cosine, or similar. The important thing is that the pulse is orthogonal to $T_p$-shifted versions of itself; that is, $$\int_{-\infty}^\infty p(t-mT_p)p(t-nT_p)=0$$ for all integers $m\neq n$. Now you're ready to create your BPSK signal, which is given by $$s(t)=\sum_{k=0}^{N-1} a_kp(t-kT_p).$$

Now you can find the FT of $s(t)$ (or the DFT of a properly sampled version of $s(t)$), and you can find the bandwidth and other spectral properties of your BPSK signal.


Why does an image get blurry beyond a limit of zooming?

Digital zooming is commonly done by using some form of interpolation, which commonly represents making up some form of smooth change between actual data samples.  Assuming the data was low pass filtered (or roughly near bandlimited) before sampling, the information about where any sharp (non-smooth) changes were located between samples has been lost.  Adding back any non-smoothly-changing details would be arbitrary (guessing from context, etc.), and thus possibly incorrect (non-corresponding to the optical field before sampling).

Smooth changes or interpolations across a large area of pixels looks blurry to humans, since this effect does not occur in real life vision often unless one's eye or optical instrument is out-of-focus or not focused on the feature of interest.


Sine Dipole formation using two hydrophones

As you've defined Hyd_1 and Hyd_2, you are simulating a two element hydrophone array with the array center defined at the middle of the two elements. In that case, Sine_Dipole_1 should be defined as the sum of the two elements, not the difference.
I hope this helps.


What is spatial correlation and spatial convolution?

These terms exist mainly for historical reasons. In signal processing the signal is a one-dimensional function of time.  So people talk about the time domain vs. the frequency domain.  On the other hand, in image processing you are looking at a 2D function of $x$ and $y$, and there is no notion of time.  Instead your are talking about spatial frequencies.  Hence, spatial correlation and spatial convolution.

Typically, in image processing you simply talk about convolution and correlation.  The term spatial usually shows up when 2-D convolution and correlation are introduced to people with background in signal processing.


How can we prove that Spectral density of In-Phase component and Quad-Phase component of a sample function of narrow-band noise have same definition?

Synthetic Method:

If $\{\hat X(t)\}$ and $\{\hat Y(t)\}$ are zero-mean uncorrelated  low-pass WSS processes with identical autocorrelation function $R(\tau)$ and identical power spectral density $S(f)$ enjoying the property that $S(f) = 0$ for $|f|>B$, then
$$\hat{N}(t) = \hat X(t)\cos(2\pi f_ct) - \hat Y(t)\sin(2\pi f)ct$$ is a band-pass process whose autocorrelation function is $R(\tau)\cos(2\pi f_c\tau)$ and power spectral density is
$${S}_{\hat{N}}(f) = S(f-f_c)+S(f+f_c) = S_N(f).$$
There is no pretense that $\hat{N}(t)$ is the same as $N(t)$ for any $t$ or that $\hat X(t)$ and $\hat Y(t)$ are the complex baseband representation of $N(t)$ but the point is that by looking only at the power spectral densities, one cannot tell whether the synthesized process $\{\hat N(t)\}$ is the same as, or different from, the given bandpass process $\{N(t)\}$.

Analytic Method: (Suspend disbelief and take with a large grain of NaCl)

Suppose that $\{N(t)\}$ is a zero-mean bandpass WSS process whose power spectral density $S_N(f)$ has value $0$ except when $|f| \in [f_c-B, f_c+B]$ and consider the random variables
$N(t)\cos(2\pi f_ct)$ and $-N(t)\sin(2\pi f_ct)$. They obviously have zero mean, and if $N(t)$ has variance $\sigma^2 = R_N(0)$, then these variables have variances \begin{align}\sigma^2\cos^2(2\pi f_ct) &= \frac 12\big(\sigma^2+\cos(2\pi (2f_c)t)\big)\tag{1}\\
\sigma^2\sin^2(2\pi f_ct) &= \frac 12\big(\sigma^2-\cos(2\pi (2f_c)t)\big)\tag{2}.
\end{align}
Their covariance is $$-\sigma^2\cos(2\pi f_c t)\sin(2\pi f_c t)
= -\frac 12\sigma^2\sin(2\pi (2f_c)t).\tag{3}$$

We take the viewpoint that each of the random processes $\{N(t)\cos(2\pi f_ct)\}$ and $\{-N(t)\sin(2\pi f_ct)\}$, neither of which is a WSS process, is the sum of a low-pass WSS process $\{X(t)\}$ (respectively $\{Y(t)\}$) and a "double-frequency" process which not WSS, and which we can eliminate by (ideal) low-pass filtering these two processes with bandwidth $B$. Put another way, the result of (separately) low-pass filtering the processes $\{N(t)\cos(2\pi f_ct)\}$ and $\{-N(t)\sin(2\pi f_ct)\}$ leaves us with WSS processes with variance $\frac 12\sigma^2$ each (the double-frequency stuff in $(1)$ and $(2)$ has been eliminated by the low-pass filtering), and these processes are uncorrelated (the nonzero covariance in $(3)$ is double-frequency stuff that has been eliminated by the low-pass filtering).

Multiplying by $\cos(2\pi f_ct)$ or $\sin(2\pi f_ct)$ is modulation (frequency content at frequency $f$ is moved to frequencies $f+f_c$ and $f-f_c$).  Applying this notion to the frequency band $[f_c-B, f_c+B]$, we see that any content there is moved down to be centered at $0$ and also moved to the band $[2f_c-B, 2f_c+B]$. Similarly, any content in the band $[-f_c-B, -f_c+B]$ is moved down to be centered at $0$ and also moved to the frequency band $[-f_c-B, -f_c+B]$. The latter is the stuff that gets eliminated by the low-pass filtering.  What is left is the stuff moved down to DC and this "explains" why Haykin claims that $$ S_{N_I}(f) = S_{N_Q}(f) =
\begin{cases}
 S_N(f-f_c) + S_N(f+f_c) & -B\le f \le B \\
 0 & \text{otherwise}
\end{cases}
$$


Numerically Controlled Oscillator (NCO) for phasor implementation?

The NCO is a cyclical counter that can go on indefinitely but is otherwise similar to what you suggest in that you are increment n to set the output rate.  It basically is a look up table of all the values in one complete cycle, and "wraps" on an overflow so that it will output continuous cycles with no discontinuity.

I think the NCO is ideal for what you are trying to do; given its simplicity, it's ability to run indefinitely keeping track of its own position in time, its quantified noise levels that you can set to what you need, it's fixed-point implementation (with no multipliers) and its ability to change the frequency of your phasor "on the fly" as you require.  I think this bottom lines the difference with the alternate approach you describe, which would not be as efficient (given the multiplications required and possible implementation in floating-point without careful mapping and scaling in which case you may as well go down the NCO path).

A little more theory may help you to see all the advantages (and simplicity) of an NCO.

First, referring to the diagrams below for the basic NCO Architecture:  A digital Frequency Control Word (FCW) sets the count rate to an extended precision accumulator (counter), optionally a Phase Control Word (PCW) for phase modulation can then be added to the output of the accumulator. The most signficant bits of this summation are then used as the address pointer to a Look-Up-Table (LUT) which holds the values for one complete cycle of a sine wave (you could also imagine having two pointers for sine and cosine to enable a complex I and Q output).

Now see the same block diagram with a mathematical view below that helps give further technical insight into the operation of the NCO. An accumulator (counter) is the digital counterpart to an integrator (if you don't see that right away, imagine sending all 1's into a counter, the output would be a ramp: 1,2,3,4,... just as you would expect an integrator output to be with a constant level at its input).   The input FCW which is just a digital signal, which can change will time (as you were looking for), is a waveform that represents frequency vs time. What value of frequency for each digital word you can input I will elaborate on later, but for now know that its value at any given time is directly proportional to the output frequency. The integral of frequency is phase (and if you are less familiar with that it may be easy to see that frequency is a change in phase versus a change in time, therefore $f=d\phi/ dt$; frequency is the derivative of phase, and therefore phase is the integral of frequency.) Since our FCW input to an accumulator is the digital representation of a frequency quantity, and the accumulator is a digital integrator, then the value at the output of the accumulator represents phase versus time (which is why we can add a phase offset with PCW at this point if desired) and the accumulator counts from 0 to 2$\pi$, rolling over upon overflow.

Since the accumulator output represents phase that is changing with time, and we want to generate a sinusoid output ($sin(\theta)$), then we can simply use a LUT to perform the trigonometric function. (Note: If you have plenty of extra cycles but no memory, other techniques to calculate the sine of an angle can be done, notably the CORDIC algorithm).  Beautiful right? So now how do we decide on specifics to design our NCO, and what happens when we lose all the least signficant bits in our phase word? Read on!

First, the accumulator sets the frequency resolution, and usually an extended precision accumulator is used with either 24, 32 or 48 bits typically used depending on the application. This is easy to see, imagine first FCW =1: The accumulator will step through every value, meaning the address pointer to the LUT will also step through every value in the stored sine wave, so the sine wave output will be at the slowest rate, and that rate will be as given by the "step size" in the formula below. Why step size? Because then imagine setting FCW =2, and the counter will now count by 2's and therfore go twice as fast before rolling over (and also upon rollover the counter must continue to count and that is why the NCO will continue to output the desired sine wave indefinately), put in FCW = 3 and it will count 3 times as fast etc.  Therefore,

$$F_{out}= FCW\frac{f_{clock}}{2^{accum_size}}$$

So regardless of how many bits we decide to use for the LUT, the output frequency is strictly set by this formula and nothing else.

Now to briefly explain phase truncation and primary considerations: Phase truncation is when we decide to only use the most significant bits of the accumulator output to send to the LUT, and in doing this we are truncating the phase word (rounding down). To understand the implications of this, first consider the diagram below of what would happen if we did not have any phase truncation (meaning a very very large look up table, or very course frequency step size if the accumulator is small). What this picture is showing is that an impossible implementation containing a perfectly sinusoidal analog source at the specific frequency shown (with no phase noise), sampled with a perfect 100 MHz clock with a perfect 12 bit A/D converter will produce the IDENTICAL results of the NCO output, using a noisy 100 MHz clock. In fact for the NCO with no phase truncation, all of the output frequencies in multiples of $f_{step}$ as provided by the formula above will be this precise, with quantization noise at the output being the only noise source (which you can control by setting the output word width). You can see if you imagine different cases that without phase truncation, the look up table will provide the exact output at any given point in time that is required (limited to the quantized frequency choices with FCW, but that step size can be very small with large accumulators). The waveform will be very smooth without any skips or hiccups, in other words, pure.

So this is great, consider the example with a 32 bit accumulator, and 12 bit output of the LUT; providing very fine frequency resolution with great spectral purity (and 6 dB better for every additional bit you add to the output width)...until you get to the memory requirements! An herein lies the motivation to consider phase truncation.

Phase Truncation

With phase truncation, memory requirements are significantly reduced at the expense of an additional noise source (phase noise). What we will see if that the noise will be well understood and can be planned to be well below any given requirements (as a trade with memory required).

Also to mention for memory optimization only a quarter cycle is needed since the remaining portions of the cycle can be derived from the first quarter cycle. There are many other memory optimizations as well such as interpolation between values (most common), and to mention without explanation, Hutchison Algorithm and Sunderland Algorithm, as well as the Cordic Rotator previously mentioned.

The phase noise pattern itself from phase truncation will be a sawtooth function of phase versus time, representing the truncated values that are missing.  From this, the useful relationship of SNR to phase truncation is given as in the picture below.  Here SNR is the power of the desired sine wave output relative to the power of all spurious output due to the phse truncation. This formula applies when the small angle criteria applies (when $sin(\theta) \approx \theta$) and comes from the rms value for a sawtooth function (or equivalently the standard deviation for a uniform distribution) which is $\frac{D}{\sqrt{12}}$ where D is the peak to peak height of the ramp or width of the distribution.   This formula is combined with the quantization noise contributions from the LUT for a digitized signal (using a similar formula: 6 dB/bit + 1.76 dB also derived from our $\sqrt(12)$ factor since quantization noise can be modeled as a uniform distribution!), to address all noise sources in the NCO.  To use this formula, the number of bits in the formula is the number of bits sent to the NCO (number of phase bits not truncated).

Finally we may be interested in the spurious free dynamic range (SFDR) which would be the power level of the strongest spur relative to our output signal (as opposed to the sum power of all spurs in SNR).  The power of the strongest spur due to phase truncation is simply 6.02 dB/bit, where again bit is the number of bits sent to the LUT. (This can be derived by taking the Fourier Transform of the ramp pattern which represents our phase error, again applicable to small angle approximations). All the spurs are integer harmonics of the fundamental output frequency, many of which will have digitally folded in the first Nyquist zone of our implementation, as suggested in the diagram below. Unlike the diagram, the 2nd harmonic is not necessarily the strongest spur, but helps to give context to the idea of spurs and SFDR.

Dithering

Dithering is the process of adding a small amount of noise (for example, use a LFSR generator as the PCW input) which will improve the SFDR at the expense of SNR. The overall noise power increases (due to our additional additive noise), however the spurious levels can be substantially reduced in the process.


SNR of a 16-bit DSP with 12-bit ADC, 40-bit accumulator

Let me add the following additional important consideration to dynamic range on the ADC side of the equation to the other answer provided.

The SNR as characterized with a full scale sine wave for a typical (uniform quantization noise) ADC is well established to be given by the following relationship:

$$SNR = 6.02 dB/bit + 1.76 dB + 10 Log(N)$$

(For more on how that equation is derived, see What are advantages of having higher sampling rate of a signal?)

Which demonstrates the consideration given for oversampling to dynamic range.  (And just to mention, there are other ADC typologies such as Delta-Sigma with noise shaping offering even more dynamic range improvement due to oversampling; however in any case the sampling and the distribution of the quantization noise is limited to the architectural implementation of the ADC itself).

The example below illustrates this with a 10 MHz BW signal at a digital IF of 25 MHz, sampled at 100 Msps.  This example was done with Analog Devices HMCAD1520 which is a 14 bit ADC with a SINAD (Signal to Noise and Distortion) of 72.5 dB. 72.5 dB translates to an ENOB (Effective Number of Bits) of 11.8 (using the same 6dB/bit+1.76 dB equation). This means that with a full scale sine-wave, the additive noise due to the ADC sampling process will be 72.5 dB lower than the power level of the sine-wave itself. However this noise, well approximated as a white noise source is spread evenly across the sampling spectrum. Thus as shown in the figure below, the portion of the noise that is within the bandwidth of the signal is 1/5 or -7 dB lower.

The rest of the noise, out of the band of interest, can be filtered out in the DSP, down to the limits of the datapath (unless special provisions are made to effectively increase the datapath precision as suggested in the other answer). For this filtering purpose, extended precision accumulators are particularly important, as described in more detail in other posts.  (for example, see what is the suitable design Method to the filter?)

Some of the distortion is in spurious frequencies, but referring to the datasheet for this device, the SFDR (Spurious Free Dynamic Range) is 83 dB, so in this specific case, with this oversampling rate, the dynamic range will not be limited by spurs.

So given a sine wave at full scale (to establish a consistent metric point), we can establish the ADC noise floor.  However for dynamic range, we also need to consider the headroom required for a modulated signal. For this reason I have created the following graph which is applicable to Gaussian distributed waveforms in magnitude (such as OFDM and CDMA both of which are well approximated by Gaussian distributions), and shows the optimum set point in rms level below full scale in order to maximize dynamic range for the case of a modulated waveform based on the number of bits. This is applicable throughout the digital system everywhere in the datapath, not just at the ADC, so is a very important consideration. This shows specifically the effect of clipping noise as the rms signal level approaches full scale, and the effect of quantization noise increasing relative to the signal, as the rms signal level is reduced from full scale.  The term "dBFS DC" is the level in dB below full scale of a DC signal (a sine wave at "0 dBFS" is -3dB below '0 dBFS DC"). The 0 dBFS DC mark is also applicable to full scale for a single tone in a complex datapath (with I and Q).


2 to 4 samples per symbol is common in my experience. The incentive for using fewer samples per symbol is an overall lower sample rate (and thus less processing load). It depends what point in the system you're talking about, but you also need to take the excess bandwidth of your pulse shape into account to ensure that the sample rate is high enough to avoid aliasing.

In some parts of the processing chain, you can get away with fewer samples per symbol than you might expect given the Nyquist rate and the signal's bandwidth. If the signal is perfectly synchronized, for example, then you don't need additional oversampling. In that case, you can use just one sample per symbol (basically, the symbol values themselves). You can then use that and the known pulse shape to reconstruct the signal to whatever fidelity you desire.

In the absence of perfect synchronization (for example, in the various receiver blocks that perform frequency, phase, and/or timing synchronization), you'll have some amount of oversampling. In the literature you'll find 2 samples per symbol as a desirable target rate, because it's really the smallest that you're realistically going to achieve. You can always go higher if you need to. It just results in a higher sample rate (so more computational work)


How to construct an unknown signal from sparse periodic samples?

Nonuniform Sampling

Assuming you know what the actual sampling rate is, and exactly which samples are missed, it is possible to reconstruct the signal over a time interval T as long as the bandwidth W of the signal still satisfies the Nyquist criterion: W < N/T/2, where N is the number of samples you do have.

Visualize it this way:
A signal of bandwidth W over a time interval T can be decomposed into a
linear combination of N sinc functions (sinc is shorthand for sin x/x),
where N = 2×W×T.
The sinc functions are equally spaced over the interval in question and
their widths are all the same.
At the instant where each sinc pulse reaches its peak, all of the other
sinc functions have a value of zero.

If you sample the signal at the instants corresponding to the peaks of
the sinc functions (tA0, tA1 and tA2),
each sample directly yields the amplitude of the corresponding sinc pulse
(A0, A1 and A2).
The original signal can be reconstructed by creating new sinc pulses of the
correct amplitudes and overlaying them, usually by using a digital or
analog filter with the correct brick-wall frequency response.
The reconstructed signal will be described by the equation:

Signal(t) = A0 sinc (t – tA0)
     + A1 sinc (t – tA1)
     + A2 sinc (t – tA2)

If you take N samples of the signal at other instants, each sample will be
a linear combination of all of the sinc functions.
Each sinc pulse will contribute an amount to the sample based on the
time interval between the sample and the center of the sinc pulse.
It is still possible to calculate the amplitudes of the sinc pulses, but
now it is necessary to solve a solve a system of linear equations in order
to do so.

If you have samples B1, B2 and B3 taken at times tB1,
tB2 and tB3 as shown above, you can write the
following equations:

B1 = A0 sinc (tB1 – tA0)
     + A1 sinc (tB1 – tA1)
     + A2 sinc (tB1 – tA2)

B2 = A0 sinc (tB2 – tA0)
     + A1 sinc (tB2 – tA1)
     + A2 sinc (tB2 – tA2)

B3 = A0 sinc (tB3 – tA0)
     + A1 sinc (tB3 – tA1)
     + A2 sinc (tB3 – tA2)

Given that you know B1, B2 and B3, and all of the time values, it's a
straightforward matter to solve this system for A0, A1 and A2.
Once you have these coefficients, you can plug them into the reconstruction
equation given previously to reconstruct the original signal.

Note that in the above explanation, I have implicitly assumed that the
uniform sampling interval (tA1 – tA0) equals
π, in order to simplify the diagrams and
equations.
The same argument holds for other sampling intervals (and bandwidths) as
well.

Averaging Multiple Periods

Using multiple periods of the signal to improve the reconstruction requires that you can precisely correlate those periods in the time tomain. Once you have done the reconstruction described above, it should be possible to precisely locate features such as peaks and zero-crossings, even if these features fall between samples. Use this information to time-align multiple periods so that they can be averaged together to reduce errors such as noise, quantization error or timing jitter.


Feature vector for music recommender system

For a big data, deep learning approach, you could try constructing an autoencoder from something such as WaveNet to achieve dimensionality reduction to get a feature vector. Alternatively, you could take STFT (short time Fourier transform) spectra of the songs over time and feed them into an RNN, again in autoencoder fashion. My intuition though is that, due to the immense complexity of musical genres, and current limitations of consumer hardware, any approach based on the raw audio alone is going to be very disappointing (unless you have a GPU farm).

Really grossly, you may get some results by finding the BPM/tempo, the spectral density, overall entropy, variance in entropy over time.

Actually, look at some of the feature extractors used in this Kaggle EEG contest. This may give you some rudimentary results, especially combined with XGB (extreme gradient boosting)

I believe most music recommendation services work like search engines, using an algorithm similar to PageRank to create a graph which associates artists.


Viterbi decoder K=3 using ADD-COMPARE-SELECT Operation

For each state in the Viterbi decoder, you want to keep track of the shortest path through the trellis that ends in that state. The trellis structure defines what all of the possible state transitions are.
Therefore, each time you get a new symbol observation, you loop through each state in your decoder and you:

Add the metric for each transition that ends in the state to the metric of the previous state, yielding a total metric for each state->state transition,
Compare the list of total metrics, and
Select the minimum metric as the most likely path to the current state.

The Viterbi algorithm, in this way, is really just repeated execution of this add-compare-select process, thus keeping track of the minimum-cost path to each state at any given time.


Phase response distortion due to IQ demodulation

I’ll hazard a guess on this that David’s comment is spot on.  You could help by providing some specifications for the low pass filter attenuation at the frequencies at which you are observing the phase distortion, or perhaps provide plots of your magnitude/phase responses for the input and output. If this is what you’re observing then:

1). Yes. For a DFT or FFT of a signal, there is likely to be phase distortion at low relative amplitudes based on different implementation details. This has to do with the precision/resolution of the data on your machine.

2). Sort of.  The filter could technically be reversible in the analytical sense (ie on paper) but quantization error from the limited precision of the data will make this impossible with many many applications.

3). Recover?  No, not really.  Assuming that you don’t have any errors in your implementation, and it sounds like you don’t, then that is simply the nature of the beast.  However, you are using filtfilt, which I understand works by applying the filter forwards and backwards, resulting in a net zero phase response.  As such, it would follow that your phase response should be unchanged, which I assume was your original hypothesis. You could just use the phase response from the original signal, if you have it.  This may or may not be practical for your application, and could be done more efficiently if it were an option.


Implementing inverse FFT using forward FFT

In your link I cannot find a reference to the time-reversal. However, here are some additions:

The time reversal $y(t)$ of a continuous signal $x(t)$ is given by $y(t)=x(-t)$. Similarly, for a discrete signal we have $y[n]=x[-n]$. However, here the delicate issue occurs that $n=0...N-1$, where $N$ is the signal length. Now, noting that in the finite discrete domain every signal is considered to be periodic with $N$, you get
$$y[n] = x[-n] = x[(-n)_N]$$

where $(x)_N$ is the modulo operation of $x$ by $N$. Hence, time-reversal in the discrete domain is not just reading your vector from back to front. Instead, the first element remains the same ($n=0$) and the others are read from back to front.

Now coming to the time-reversal property wtih the Fourier Transform:

$FFFF=I$ where $F$ is the (unitary) Fourier transform operator and $I$ is the identity. I.e. applying 4 times the (forward) Fourier transform yields the original signal.
Hence, logically $FFF=F^H$, where $F^H$ is the inverse Fourier Transform (yes, it is also the Hermitian of the forward transform). However, calculating inverse FFT by 3 times forward FFT is not very efficient...
Further, we know $FF=T$ where $T$ is the inflection operator $(Tx)[n]=x[-n]$, again read with the modulo interpretation.

Hence, we find $F^H=TF$, i.e. in order to caculate inverse FFT, you can do forward FFT and time-reverse the result:

f = 0.1
Fs = 10
t = np.linspace(-10,10,Fs*20)
s = np.cos(2*np.pi*f*t*(t+1))

# Take fft
U = np.fft.fft(s)

# Transform again
u_t = np.fft.fft(U)

# Reverse and normalize

s_new = np.divide(u_t[-np.arange(u_t.shape[0])],s.shape[-1])

# Finally slice for easier viewing
plt.figure(figsize=(12,4))
plt.plot(t,s_new)
plt.plot(t,s)


Why DFT-SPREAD OFDM has lower PAPR than OFDM?

It is because the time domain symbols have lower PAPR than the frequency domain symbols.

The PAPR of OFDM is due to the 2 reasons. One is due to the IDFT(IFFT) and the other is due to the symbol type. The IDFT converts the time domain symbols to frequency domain symbols that have very high PAPR. The PAPR of 16QAM is also higher than that of QPSK. However, even when these symbol types are the same, the PAPR of DFT-SPREAD OFDM is still lower than that of regular OFDM. The reason is the (N point) IDFT effect is mostly cancelled out by the (M point) DFT. If the M=N (no zero), they are completely canceled out and the symbols would be just a time domain symbols. Even though N is much larger than M, the resulting symbol has more like time domain symbols, then the PAPR is lower. LTE up-link uses this type of OFDM that is called SC-FDMA. SC means Single Carrier because the symbols look like single carrier time domain symbols even though it has an OFDM structure. They use SC-FDMA for the up-link LTE for the lower PAPR seeking the longer battery life.


What exactly can the vector/observation be? Can it be a relatively simple function, say approximation of the spectral envelope? Or is it usually a vector of R^n?

You won't go very far by summarizing the entire spectral envelope of a sound by a single real, so yes, observations are vectors - their dimensionality is in the 10 to 100 components bucket...

Typically, the features used are MFCC and their first/second order derivatives. They capture spectral envelope (in short: dimensionality reduction applied to a low-resolution spectrum computed on a perceptual scale) and have several invariance properties that make them relatively robust.

  How is it, that if you say "ooone" instead of "one", P("ooone"|M_1) will still be the highest of the three trained HMM models? Is it because if the model was trained with 'one', the model already favors the observation that happens when the sound of 'o' is said, and the state transition of x_1 -> x_1 (where x_1 is likely to emit this observation) is also favored?

Yes, this is what happens.

The cost of stretching the "o" is high, but still lower than having the emission probabilities totally wrong by matching the "o"s or the "n" that follows with other phonemes.

  Basically, is there some convincing proof/intuition, that pronouncing a certain word very slowly at certain parts will not cause it to be recognized incorrectly?

If the features used for the observation vector are robust, and if the model is trained on the same speaker as for recognition (or if suitable model adaptation measures are used), the emission probabilities can get very low when confusing phonemes.

One thing that could cause confusions would be a HMM trained on utterances of "five" and another trained on utterances of "fiiiiiiiiiiile". In this case, "fiiiiiiiiiiiive" might be recognized as "fiiiiiiiiiiiile" - the cost of repeatedly stretching the "i" might be enough to counterbalance the confusion from "v" and "l" on a handful of frames. So an abnormal utterance could be confused with another abnormal utterance for which a model is available. The thing is, abnormal utterances are "averaged out" during the training process.

  does the classification HMM operation happen at all?

For recognition, it is not relevant to know the most likely sequence of states. However, there are applications for which knowing the sequence of state is needed.

In particular, we sometimes want to exactly synchronize the recognized sequence with the original audio recording. This could be useful for subtitling, or recording a timestamped transcription of a recording. This is also used during Viterbi training, an alternative to the Baum-Welch algorithm that uses state decoding (classification) in the training loop.

  Lastly, could some please explain how continuous speech recognition differs from this?

Left-right models with a handful of states are used to describes diphones or triphones. States have hundreds to thousands mixture components (or nowadays we eschew GMMs and use other emission models). This is already a ridiculously large model space, so there are tied-states - that is to say some states in the model share the same emission probabilities with other states in the model. Anything that pools together the parameters from several parts of the model ensures that more training data will be used.

Then, the triphone models are concatenated together to build word models - using pronunciation dictionaries (lexicons). The lexicon maps a word into a sequence of phones. While there are approaches to automatically learn it, this data primarily comes from linguists.

Then word models are concatenated together to build a language model. The result is an extremely big FST, which is not even fully composed in memory from its component - but only traversed on the fly during recognition.

Of course the whole thing is not trained with Baum-Welch... We train separately each level... Tri-phone HMMs are trained on aligned speech data; lexicons are curated by linguists; language models trained separately on purely textual data...

Typically a small amount of aligned speech data (in which the transcription are aligned with the audio) is used to bootstrap the triphones model. Then this inexact model can be used to align a larger set of unaligned training data, and the process is iterated.


Digital modelling of circuits with diode (i.e. guitar distortion)

One possible tool is Wave Digital Filter analysis which is a type of physical modeling that represents signals as travelling waves. It can also be extended to non-linear elements such as diodes.

However, for distortion unit, you could instead of trying to digitalize the analog circuit try to extract a waveshaper from its behaviour. Common waveshapers are 3rd order polynomial and atan function.

EDIT : here is an intersting paper about wave digital domain modeling of guitar distortion

EDIT2 : and here a DAFx lecture about existing methods for non linearities in virtual analog


How do I detect the sharp edges in the following image?

Ok I solved this problem using a simple solution. After I computed my convex defects in my countours, I performed a point polygon test and found the distance between the point in the convex defect and the hull. And I gave a threshold after looking at the distance of the points and all the sharp points were in the threshold greater than approximately 30. So this filtered my points


Semantic Distance Measure Between Images

One of the most known image similarity metric is the SSIM. You can take a look to the next links:

https://en.wikipedia.org/wiki/Structural_similarity
https://www.imatest.com/docs/ssim/
https://www.cns.nyu.edu/~lcv/ssim/

EDIT 1:
Okay, you refer to semantic similarity. I have not worked on that issue really, and probably there are new DNN solutions to that purpose. I have not checked the state-of-the-art related to this topic.
Even so, I have made a quick test using the skimage SSIM and the difference between images to give you some idea about what happens with these metrics on several images. Find in the next images some hints about their performance for your target application:

Please, notice the values of the SSIM, MSE and STD DIFF on the images. SSIM and MSE works very well to compare quality of the same image. But not for semantic comparison.
Cheers.


If you're using the usual Decimation-In-Time Cooley-Tukey radix-2 FFT, the buffer of samples going in need to be scrambled in order by having the indices of the samples bit-reversed.  And if the FFT is a complex-input, complex-output FFT and it is given real data, yes the imaginary part is set to zero.

And, in the case of a purely real input signal, you need not bother swapping those imaginary parts in your data preparation before passing to the net FFT algorithm expecting the input to be in bit-reversed order.  But you do have to bother with the real parts of the samples.


There are lots and lots of software that can aid you in designing digital filters. MATLAB is probably the most used software, at least in the university sector. The  DSP toolbox and the signal processing toolbox probably cover all of the well-known methods for digital filter design, as already mentioned.

Alternatives to MATLAB (that are free) include: Julia, Octave, Scilab, and SciPy (Python with libraries for technical computing). There are a lot of others, but these are the ones that I know have high quality libraries/methods for filter design.

For Julia, you can use the Filters.jl package, for Octave the signal package, for Scilab the Signal Processing toolbox, and for SciPy you have scipy.signal.

A lot of filter design is done using optimization, and if you want to customize the cost-function or make other tweaks to e.g. the least-squares method or the Parks-McClellan method, you could have a look at a high-level optimization library such as CVX (works with MATLAB and Julia ++), or JuMP for Julia.

These are just the tools that I have a fair bit of familiarity with...


How to show that this filter is a HP filter

As was pointed out in the comments above, this is a highpass filter. There is no strict definition for what constitutes a lowpass, bandpass, or highpass filter. Instead, they possess certain qualities:

Lowpass filters attenuate high frequencies while passing low frequencies.
Bandpass filters attenuate low and high frequencies, while passing some band in the middle.
Highpass filters attenuate low frequencies while passing high frequencies.

Note the lack of precise terms in the descriptions above. In order to classify a filter into a category like those above, one evaluates the character of its magnitude transfer function. That requires a little manipulation:

$$
H(\omega) = 1 - e^{-j\omega}
$$

$$
\begin{align}
|H(\omega)| &= |1 - e^{-j\omega}| \\
&= \left|e^{-j\frac{\omega}{2}} \left(e^{j\frac{\omega}{2}} - e^{j\frac{\omega}{2}}\right)\right|
\end{align}
$$

Using Euler's identity, we can simplify the expression inside the parentheses:

$$
e^{j\frac{\omega}{2}} - e^{j\frac{\omega}{2}} = j2\sin\left(\frac{\omega}{2}\right)
$$

$$
|H(\omega)| = \left|e^{-j\frac{\omega}{2}} j2\sin\left(\frac{\omega}{2}\right)\right|
$$

We can drop the $j$ and $e^{-j\frac{\omega}{2}}$ terms here, because each of them have unity magnitude and therefore don't affect $|H(\omega)|$. Therefore:

$$
|H(\omega)| = \left|2\sin\left(\frac{\omega}{2}\right)\right|
$$

We can check this result by comparing it with a plot from MATLAB's Filter Visualization Tool (fvtool):

Which looks about right. This is plotted over the region $\omega \in [0, \pi]$, and shows that the filter has zero response at DC (attenuating low frequencies) and a magnitude response of $2$ at $\omega = \pi$ (passing high frequencies). That looks like a highpass filter.


Problem implementing KL expansion with square-exponential kernel: output just looks like Gaussian

I've had issue with this -- please see the errata of the reference. The eigenfunctions are not scaled in the original version of the book. This errata should solve it.


Determine constant $A$ such that $x[n] = x[n] \star x[n]$

Systematically, you could just solve the problem by finding a solution $A$ that satisfies the following two equations:
$$(A-1)^2=A-1\\A^2=A$$


Is there an audio processing library/API that contains all standard features of a digital audio workstation?

For python, I'd try out:

pydub for basic audio manipulation and arrangement (slicing, concatenation, fading, repeating, etc).
For effects processing, pyo is a good python signal processing module with a musical focus. Here's a comprehensive review, with inline clips.

Together you should be able to get good coverage of most DAW functions. There are also fantastic python audio libraries like librosa, in case you do decide to do any feature extraction or audio analysis. Good luck!


How are 12 outputs being reconciled so that a Homomorphic filter could be applied?

From my reading, the authors don't combine the 12 images in a single image: each generates a directional scratch mask, each of them being processed independently. If I read too fast, tell me. Indeed, they first remove a low-pass version, and apply a logarithmic homomorphic filter. But you could combine the binary masks with an OR to get a single image of potential scratches.

However, it would be possible with filters alone to recombine them by addition.

Suppose that you only have two filters $f_1$ and $f_2$, that are applied to an image $i$. $\mathcal{F}$ is the Fourier transform: $I=\mathcal{F}(i)$ and $F_k=\mathcal{F}(f_k)$.

The hypothesis that they do not overlap is related to their frequency behavior, with spectra $F_1$ and $F_2$. Their supports $S_{F_1}$ and $S_{F_2}$ (imagine 2D patches in the full 2D frequency domain $S_F$) do not overlap if their intersection is void: $S_{F_1} \cap S_{F_2}= \emptyset$. What is not explicit is that, often, the filters are also complementary, their union covers the whole frequency domain: $S_{F_1} \cup S_{F_2}= S_F$. Thus, the spectrum $I$ of $i$ is cut into two complementary parts, and $I = F_1\cdot I+F_2\cdot I$. Then, you could recover $i$ with
$$i = \mathcal{F}^{-1}(F_1\cdot I+F_2\cdot I) = \mathcal{F}^{-1}(F_1\cdot I)+\mathcal{F}^{-1}(F_2\cdot I)\,.$$

Equivalently, this amount to directly write, in the original domain:

$$i = f_1\star i + f_2\star i\,.$$

Visually, the frequency domain is a pie, and each filter is a slice. When you have the the slices, you can reproduce the original pie. In practice, people don't use exactly non overlapping filters in frequency, but orthogonal filters in space, that sum to unity to equally cover all frequencies.  This is a little more general as non overlapping filters, as $F_1\cdot F_2 = 0$, thus $f_1\star f_2 = \mathcal{F}^{-1}(F_1\cdot F_2) = 0$,  are a subset of orthogonal filters.


Applying a lowpass filter to a noisy square signal leads to a shift of the signal

The only possible way that any real-world filter can work is by using past values of the signal you're filtering to get the latest filtered value.  So any real-world filter will always have some delay.
So if by "correct this shift" you mean "make it go away" -- not really.
However, if you want to look at the signal and its filtered version after the fact, then yes -- calculate the filter's group delay (or just eyeball it in your data), and in your graphs shift the input over by that amount.  This is easier to do if you use a symmetric FIR filter, because it'll have a constant group delay and the group delay is just half the filter length.
If your goal is to measure the duration and magnitude, then as long as you can have some delay in the measurement then just take your filtered version and feed it into whatever duration-and-magnitude measuring algorithm you've cooked up.  Since the leading and trailing edges of the square wave are offset by the same amount, the measured duration should be just fine.


Is There Any Filtering Technique with Element Wise Product Only Instead of Convolution?

Classic filteration is indeed done using convolution.
Though I have seen broader definition of filtering as shaping the signal in its frequency domain which can be done in many other methods as well.

Of course you can create meaningful operations using element wise operations and even specifically multiplication.

Think of the case you have a noise with the following properties:

It is non negative.
It is signal dependent.

Then operation like thresholding would make sense and it is applied using element wise multiplication. This case is very similar to $ {L}_{1} $ Denoising.

Pay attention that in this case the filter depends on the data.


What are the differences in overtones for hard vs soft materials?

I am not sure if this is exactly answering your question but it may help you in your modeling efforts.  The metallic strings are going to be fairly simple to model in that the differential equations you use will be only dependent on the shape of string at any given time and the tensile strength of the string.

The nylon strings exhibit non-linear viscoelastic behavior which has a time dependent component to it.  That means that is has a memory and the local velocity of the string will be important.  By memory I mean there is residual strain built in once plucked and the time for recovery will be much greater than the frequency of motion.  I believe this is what gives it a “softer” sound.  Classically viscoelastic materials are modeled with a spring and damper in parallel.

Descretizing the string over a number of elements and using a divided differences approach to approximate the differential equations you should be able to get a theoretical sense of how these differences affect the observed signal.

Hope that is helpful for you in thinking about your problem.


Why Is the Canny Edge Detection Used Instead of Sobel / Prewitt Edge Detection Before Hough Transformation?

Canny Edge Detection is considered to be a better (In False Alarm sense) edge detection than those you mentioned.
This is, mainly, due to 2 steps:

Non Maximum Suppression - Edges candidates which are not dominant in their neighborhood aren't considered to be edges.
Hysteresis Process - While moving along the candidates, given a candidate which is in the neighborhood of an edge the threshold is lower.

Those 2 steps reduce the number of "False" edges and hence create a better starting point for farther process like Hough Transformation.


How to shift the center frequency of an IIR filter

The problem was an incorrect sampling rate issue, as a lower sampling rate than the designed rate caused the center frequency shift downwards. Today I learned something about the relationship between sampling rate and IIR coefficients (there is a proportional relationship between sampling rate and center frequency).


Difference between $\tt rand()$ and $\tt randn()$ in MATLAB

The reason for the difference is that rand() has an output range of $[0,1]$, and therefore you have a constant in both $a$ and $b$ in the first example. The other function randn() generates an output with zero mean. The constants are correlated. Try

a = -0.5 + rand(512,512);
b = -0.5 + rand(512,512);

or

a = rand(512,512);
b = rand(512,512);
a = detrend(a,'constant');
b = detrend(b,'constant');

and see if that is more like what you would expect...


The idea of autocorrelation is to provide a measure of similarity between a signal and itself at a given lag.  There are several ways to approach it, but for the purposes of pitch/tempo detection, you can think of it as a search procedure.  In other words, you step through the signal sample-by-sample and perform a correlation between your reference window and the lagged window.  The correlation at "lag 0" will be the global maximum because you're comparing the reference to a verbatim copy of itself.  As you step forward, the correlation will necessarily decrease, but in the case of a periodic signal, at some point it will begin to increase again, then reach a local maximum.  The distance between "lag 0" and that first peak gives you an estimate of your pitch/tempo.  The way I'm going to describe how to compute it in practice yields something slightly different (i.e. cyclic vs. acyclic mostly), but that's the conceptual basis for how it works.

Computing sample-by-sample correlations can be very computationally expensive at high sample rates, so typically an FFT-based approach is used.  Taking the FFT of the segment of interest, multiplying it by its complex conjugate, then taking the inverse FFT will give you the cyclic autocorrelation. In code (using numpy):

freqs = numpy.fft.rfft(signal)
autocorr = numpy.fft.irfft(freqs * numpy.conj(freqs))

The effect will be to decrease the amount of noise in the signal (which is uncorrelated with itself) relative to the periodic components (which are similar to themselves by definition).  Repeating the autocorrelation (i.e. conjugate multiplication) before taking the inverse transform will reduce the noise even more.  Consider the example of a sine wave mixed with white noise.  The following plot shows a 440hz sine wave, the same sine wave "corrupted" by noise, the cyclic autocorrelation of the noisy wave, and the double cyclic autocorrelation:

Note how the first peak of both autocorrelation signals is located exactly at end of the first cycle of the original signal.  That's the peak you're looking for in order to determine the periodicity (pitch in this case).  The first autocorrelation signal is still a little "wiggly", so in order to do peak detection, some kind of smoothing would be required.  Autocorrelating twice in the frequency domain accomplishes the same thing (and is relatively fast).  Note that by "wiggly", I mean how the signal looks when zoomed way in, not the dip that occurs in the center of the plot.  The second half of the cyclic autcorrelation will always be the mirror image of the first half, so that kind of "dip" is typical. Just to be clear about the algorithm, here's what the code would look like:

freqs = numpy.fft.rfft(signal)
auto1 = freqs * numpy.conj(freqs)
auto2 = auto1 * numpy.conj(auto1)
result = numpy.fft.irfft(auto2)

Whether you would need to do more than one autocorrelation depends on how much noise is in the signal.

Of course, there are many subtle variations on this idea, and I'm not going to get into all of them here.  The most comprehensive coverage I've seen (in the context of pitch detection) is in Digital Processing of Speech Signals by Rabiner and Schafer.

Now, as to whether autocorrelation will be sufficient for tempo detection.  The answer is yes and no.  You can get some tempo information (depending on the source signal), but it may be hard to make sense of what it means in all cases.  For example, here's a plot of two loops of a breakbeat, followed by a plot of the cyclic autocorrelation of the entire sequence:

For reference, here's the corresponding audio:

Sure enough, there's a nice spike right in the middle corresponding to the loop point, but it came from processing quite a long segment.  On top of that, if it wasn't an exact copy (e.g. if there were instrumentation with it), that spike wouldn't be as clean.  Autocorrelation will definitely be useful in tempo detection, but it probably won't be sufficient by itself for complex source material.  For example, even if you find a spike, how do you know whether it's a full measure, or quarter note, a half note, or something else?  In this case it's clear enough that it's a full measure, but that won't always be the case.  I'd suggest playing around with using AC on simpler signals until the inner workings become clear, then asking another question about tempo detection in general (as it's a "bigger" subject).


Why use both a high pass and a low pass filter in a Butterworth implementation for noise reduction?

You probably meant to reverse them:  The LPF should be at 1100 Hz, and the HPF should be at 100 Hz.  Then you're keeping everything between 100 Hz and 1100 Hz, and throwing away lower and higher frequencies.

will eliminates the low frequency samples

Also remember filters don't eliminate everything in the stopband, they drop off with frequency, so some of the content outside the bands will still exist.
It will eliminate 0 Hz, but other frequencies will only be attenuated, not eliminated.


How to Apply Window Functions in Frequency Domain?

You need to use the Convolution Theorem.

Namely, when you want to apply an LPF Kernel in one domain (Using convolution) you can use multiplication in the other domain.

The tricky part is the dimensions of the signals (Since we're dealing with Discrete signals).
Since the LPF Kernel in the convolution domain might be different then the size of the signal it is applied to, so how do we multiply?

Well, the solution is given by interpolation to the correct dimension.
This is done by adding zeros at the end of the signals.
Pay attention that the end of the signal in the frequency domain is tricky.
The end are the samples before the sampling frequency (On both sides as it is symmetric), add zeros and keep the signal symmetric (If you signals are real in the other domain).


Robert's Cross is a little tricky because it's not an odd size (2x2 rather than 3x3 or 5x5). I've done it using numpy+scipy using a padded 3x3 convolution mask.

import sys
import numpy as np
from scipy import ndimage
import Image

roberts_cross_v = np.array( [[ 0, 0, 0 ],
                             [ 0, 1, 0 ],
                             [ 0, 0,-1 ]] )

roberts_cross_h = np.array( [[ 0, 0, 0 ],
                             [ 0, 0, 1 ],
                             [ 0,-1, 0 ]] )
def load_image( infilename ) :
    img = Image.open( infilename )
    img.load()
    # note signed integer
    return np.asarray( img, dtype="int32" )

def save_image( data, outfilename ) :
    img = Image.fromarray( np.asarray( np.clip(data,0,255), dtype="uint8"), "L" )
    img.save( outfilename )

def roberts_cross( infilename, outfilename ) :
    image = load_image( infilename )

    vertical = ndimage.convolve( image, roberts_cross_v )
    horizontal = ndimage.convolve( image, roberts_cross_h )

    output_image = np.sqrt( np.square(horizontal) + np.square(vertical))

    save_image( output_image, outfilename )

infilename = sys.argv[1]
outfilename = sys.argv[2]
roberts_cross( infilename, outfilename )

From the Wikipedia entry on Robert's Cross. http://en.wikipedia.org/wiki/Roberts_Cross

My script's output.


SNR calculation when noiseless signal is not known

Assuming you're calculating the $\textrm{SNR}$ in decibels, it is defined by the following equation:

$$
\textrm{SNR}_{db} = 10\log_{10}\left(\frac{P_{signal}}{P_{noise}}\right)^2
$$

You currently have (with squares consolidated), where $P$ is the noisy signal:

$$
\textrm{SNR}_{db} = 10\log_{10}\left(\frac{P_{signal}}{P_{signal}-P}\right)^2
$$

This doesn't look right because you will end up with a negative denominator since $P>P_{signal}$, where in essence $P=P_{signal}+P_{noise}$. Although it ends up being the same thing because the squaring forces it to be positive, it seems a bit of a fudge.

I think what you meant to do was to subtract the estimated signal from the noisy signal to estimate the noise power:

$$
\textrm{SNR}_{db} = 10\log_{10}\left(\frac{P_{signal}}{P-P_{signal}}\right)^2
$$

One way of obtaining your $P_{signal}$ is to understand what your transmitter is capable of putting into the medium. It would then follow that the further away your receiver was to your transmitter, the lower the $\textrm{SNR}$. Remember, a low SNR can mean two things, either your transmitter isn't loud enough, or the background noise is too great.


Undecimated DWT vs. CWT. In what cases is one preferable over another?

Are there any other factors supporting the choice of one or another alternative?

possibility to use almost any wavelet shape (and associated sampling) or signal-matched: in favor of the CWT
possible parallelization in cycle-spinning form: in favor of SWT
tunable redundancy factor: in favor of the CWT
possibility to derive sharp signal/noise estimators for denoising: in favor of SWT
geometrical extensions (direction, shear, anisotropy) in higher dimension: in favor of the CWT


RLS is derived using the Woodbury Identity for rank-1 update.

I would go back and re-derive the RLS approach in light of Woodbury and generalize it to the block update case. I vaguely recall doing this before, but I can't recall when/what-for at the moment.

Here's the identity:

$$
(A+UCV)^{-1} = A^{-1} - A^{-1}U(C^{-1}+VA^{-1}U)^{-1}VA^{-1}
$$

[Aside:  I will try to use upper case for matrices, lower case with underbar for column vectors, and simply lower case for scalars - though I don't think we will have any scalars. I also use superscript $H$ to denote Hermitian transpose]

Now, consider your data vector and matrix at time 'k' and the resulting LS solution:

$$
\underline{c}_k = arg\min_{c_k}||\underline{y}_k - X_k\underline{c}_k||_2^2
$$

Then, at time 'k+L' - after another block of 'L' samples, we have

$$
\underline{c}_{k+L} = arg\min_{\underline{c}_{k+L}}||\underline{y}_{k+L} - X_{k+L}\underline{c}_{k+L}||_2^2
$$

Next, define some terms for the block update

$$
\underline{y}_{k+L} = \left[\begin{array}{l}\underline{y}_k\\ \underline{\tilde{y}}_{k+L}\end{array}\right]
$$
$$
{X}_{k+L} = \left[\begin{array}{l}X_k\\ {\tilde{X}}_{k+L}\end{array}\right]
$$

Now, solve for $\underline{c}_{k+L}$ using the Woodbury identity (in line 3), state, and the solution from time $k$:

\begin{eqnarray}
\underline{c}_{k+L} &=& (X^H_{k+L}X_{k+L})^{-1} X^H_{k+L}\underline{y}_{k+L}\\
&=&(X^H_{k}X_{k}+\tilde{X}^H_{k+L}\tilde{X}_{k+L})^{-1} (X^H_{k}\underline{y}_{k} + \tilde{X}^H_{k+L}\underline{\tilde{y}}_{k+L})\\
&=&[(X^H_{k}X_{k})^{-1} + (X^H_{k}X_{k})^{-1}\tilde{X}^H_{k+L}(I+\tilde{X}_{k+L}\tilde{X}^H_{k+L})^{-1}\tilde{X}_{k+L}(X^H_{k}X_{k})^{-1}][X^H_k\underline{y}_k + \tilde{X}^H_{k+L}\underline{\tilde{y}}_{k+L}]\\
&=&\underline{c}_k+W^H_{k+L}\underline{\tilde{y}}_{k+L} + [W^H_{k+L}(I+\tilde{X}_{k+L}\tilde{X}^H_{k+L})^{-1}W_{k+L}](\underline{v}_k+\tilde{X}^H_{k+L}\underline{\tilde{y}}_{k+L})
\end{eqnarray}

where

$$
W_{k+L} = \tilde{X}_{k+L}(X^H_{k}X_{k})^{-1}
$$
$$
\underline{v}_k = X^H_k\underline{y}_k
$$
$$
\underline{v}_{k+L} = \underline{v}_k + \tilde{X}^H_{k+L}\underline{\tilde{y}}_{k+L}
$$

I would keep state for the latter intermediates and for $(X^H_{k}X_{k})^{-1}$. The expensive part here is usually the calculation of $(I+\tilde{X}_{k+L}\tilde{X}^H_{k+L})^{-1}$, since it is a rank $L>1$ update.

Apologies if I have typos above. I was deriving as I typed, and eventually I couldn't see the wysiwyg output anymore. ;) Hopefully, you can see the jist of the approach, nonetheless. There certainly could be further intermediate and state optimization. I didn't go back to RLS to remind me of how it was wrung to minimize operations, but it probably goes somewhere along these lines.

Finally, given the above block update approach, the question for you is - which is preferred:  $L$ rank 1 updates (RLS) or 1 rank $L$ ("block RLS" as above) update? This depends on your situation and environment. A block approach may be better if the rank-1 updates would be done in an interpreted loop (e.g. Matlab), and one may be better than the other for accumulated precision errors.


The thing to keep in mind here is that HOG is not invariant to in-plane rotation. A change in orientation of more than 10-15 degrees will probably throw it off.  So if your head can have different orientations in the image, you would either need to train multiple detectors or use something other than HOG.

By the way, there is a function extractHOGFeatures in the Computer Vision System Toolbox, and SVM is available in the Statistics Toolbox. See this example.

Edit: from your sample images it looks like in most cases you need to detect profile faces. vision.CascadeObjectDetector in the Computer Vision System Toolbox can do that for you. For greater robustness, I would run two detectors in succession: one for profile faces and one for frontal faces.  And also, HOG may not be your best choice here. For face detection Haar or LBP features generally work better.

No matter what you do, you will have some false negatives.  So, if you are using a video camera, I would also recommend tracking the faces.  See this example. This will give you an opportunity to correct mistakes made on individual frames.

Edit2: You can also train your own detector with trainCascadeObjectDetector, and you can use trainingImageLabeler app to label the positive samples.


How would sigma-delta modulation work in software?

Upsample the signal to desired frequency. You can use linear interpolation (poor quality but simple and fast) or sinc interpolation (good quality but slow). For example of library dedicated to good quality resampling, see libsoxr: http://sourceforge.net/p/soxr/wiki/Home/
Perform dithering with noise shaping. You need to filter the dithering signal to minimize noise within 0-20000 Hz range while increasing it in ultrasonic range. For details on noise shaping, see this excellent document: http://www.beis.de/Elektronik/DeltaSigma/DeltaSigma.html especially Figure 9 which shows adding noise in feedback loop.

That's how delta-sigma DACs work and this can certainly be implemented in software.

PWM would require upsampling 65536× for 16-bit audio so it isn't used. PDM (Pulse Density Modulation) is used instead and is essentially the same as delta-sigma modulation. And it requires only 64× upsampling for Hi-Fi audio.


You know what I actually found really useful, Gabor's original paper, "Theory of Communication":

http://redwood.berkeley.edu/w/images/b/b6/Gabor.pdf

Of particular interest is how Gabor used banks of reeds, tuned to different frequencies, to visualise his filters. For me this led to a good conceptual understanding, especially of the uncertainty principle, before diving into the mathematics.


Image Segmentation - Comparing GrabCut Method vs. Level Set Method

Level set is for continuous segmentation, multiphase. But Graph cut is for discrete segmentation, only two phase


Your second-order high pass transfer function is wrong. You should use

$$H(s)=\frac{s^2}{s^2+2\zeta\omega_0 s+\omega_0^2}$$

But - as mentioned in the comments - a high damping and a sharp cut-off are incompatible requirements.


Can Morse Code be encoded using "silence" for dots and dashes?

i used to be a ham radio person (like more than 40 years ago) and still know Morse code.  it's like learning to ride a bicycle; you don't forget.  between the elements ("dits" and "dahs") of a single character, the space of silence is supposed to be the same as the ON time of a single "dit".  call that a single unit of time.  so a string of dits would be at exactly twice the frequency as a stream of dahs.  but the space of silence between characters is supposed to be three units, the same length as a dah.

so, if you encode only the onsets of each element, whether it's a did or dah, you can't tell the difference between a lot of dahs in a row in the same character (like zero or "0") or the same number of dits in separate letters (like five adjacent "E" in sequence).  from just the onsets of the beeps, a single "0" (which is "dah-da-dah-dah-dah") sounds the same as "EEEEE" (which is "dit-dit-dit-dit-dit" at the same rate).  nor could you hear the difference from "EENE" ("dit-dit-dah-dit-dit" at the same relative onset times).

if you make the space between letters long enough that there can be no confusion between separated characters and the closer elements within a character, you can figure it out from just knocking.  but, given the element and character spacing set forth by the format of Morse Code, you cannot tell "0" from "EEEEE" from "EENE" with just knocks.


Note that these definitions of effective duration and effective bandwidth are only useful for very specific signals, namely for (real-valued) low-pass signals that are even and centered around $t=0$. This implies that their Fourier transform is also real-valued, even and centered around $\omega=0$, and, consequently, the same definition of width can be used.

Now for the proof, which is straightforward. Take the definition of the Fourier transform and its inverse:

$$F(\omega)=\int_{-\infty}^{\infty}f(t)e^{-j\omega t}dt\tag{1}$$

$$f(t)=\frac{1}{2\pi}\int_{-\infty}^{\infty}F(\omega)e^{j\omega t}d\omega\tag{2}$$

From $(1)$ and $(2)$ we get

$$F(0)=\int_{-\infty}^{\infty}f(t)dt\tag{3}$$

and

$$f(0)=\frac{1}{2\pi}\int_{-\infty}^{\infty}F(\omega)d\omega\tag{4}$$

from which we get

$$T_e=\frac{F(0)}{f(0)}\tag{5}$$

$$B_e=\frac{f(0)}{F(0)}\tag{6}$$

The result follows immediately.

For information on more general definitions of duration/bandwidth and on the uncertainty principle take a look at this answer.


Sequentially get the outer contour of an Edge Image

You can do the following in order to get a connected contour. Get a few points on the contour say like 100 points or so. And then fit a cubic-spline on it.

I had done something which might be helpful to you. Below is what I had done.
The project was clustering of cars by shapes. So I had images of cars and I a user had marked the boundaries of the car. Red dots are the manually marked points. Then I had fit a closed cubic splines on these few points. The blue contour is the cubic spline. I then used this contour for my purpose

I am sure you can do something similar to get the contours. Please be aware that it is possible to get an open curve with cubic splines. Let me also tell you constructing splines from these set of co-ordinate points is simple. What you basically want to do is fit a cubic curve between each set of points.

Please ask if you need further clarification on what I think.


your output is definitely a linear combination of previous in- and output samples.
That means it's a linear time-invariant operation.
That means it can't introduce aliases.
What you see is something else than an alias, and I'll venture the guess you're interpreting sidelobes as aliases.


method to detect frequency sweep in audio

If you know the signal of interest in advance, you can pre-record it and constantly calculate correlation between recorded sample and incoming signal.

Record a sample sound from a channel to be analyzed (you can record a number of periods)
Reverse it
Use reversed signal as an impulse response for FIR filter (this also is known as "matched filter") -  filter will output a peak when signal of interest passes through it.

If you are interested in general varying tone detection (much more difficult problem), look at the chirplet transform - the sort of wavelet transform which uses varying frequency wavelets (chirplets) as its basis. People used it to count wolf howls, for example.


Trying to implement MATLAB $\tt pwelch$ in Python

This was an oversight on my part - I needed to adjust nfft to be the next power of 2 above window size.


Its very simple: The time domain signal per range gate should be windowed (e.g. hamming, blackman-harris, etc.) to avoid ringing, then a FFT per range gate should be calculated. All that is left is to arrange everything in a matrix where each row is a frequency bin and each column is a range bin. (or vice versa).


Difference between Fine and Coarse Frequency Offset in OFDM

If in doubt, read the original Schmidl&Cox Paper Robust Frequency and Timing
Synchronization for OFDM. It's actually a good read, and a good paper with respect to demonstration of state of the art, proposed algorithm, theory, and simulation. I highly recommend it.
Anyway, to understand this, you should consider OFDM as a multicarrier system. In fact, the DFT used in OFDM can be understood as Sinc filter bank:

The spacing of the sincs is actually the OFDM bandwidth divided by the size of the DFT. Notice that the important property of the sincs is that they have a zero at every integer multiple of the subcarrier distance – so that one carrier doesn't interfere with the others.
Now, assume your signal only has a small offset, let's say ca ¼ of the subcarrier spacing:

Then, Schmidl and Cox only has to find the offset so that the zeros of the received sincs align with the neighboring peaks of the "ideal" sinc-filterbank-shape.
Now, that's the fine frequency correction.
The coarse frequency correction becomes important when the frequency offset is larger than one subcarrier distance; let's look at something that is shifted by ca 3¼ of a subcarrier distance:

Then, the fine correction first aligns the receive signal so that the zeros/peaks match, but there's a remainder of full subcarrier multiple offset:

The job of the coarse frequency correction is to eliminate the remaining 3 subcarrier bandwidths of offset.


Can I compute my IIR faster given limited possible input values?

Depends on the hardware running your C++ code.  If you have a slow multiply unit and fast branching, then you might be able to replace a few slower multiply-adds with faster adds or subtracts.  But on most modern GHz CPUs, the add/sub branch penalty will likely be slower than a floating-point mul-add by +-1.0.


First of all, welcome to DSP SE.
What you see in the image you have linked is termed (spectral) leakage. When you are dealing with the Fourier series you deal with a periodic continuous function which is "decomposed" into a (possibly) infinite sum (series). Then, when you go to the Fourier transform, you have a non-periodic function (which you could possibly presume to be periodic in some interval if, for example, it starts and ends on the x-axis) which is decomposed into an infinite sum of spectral components (you have energy in every point on the spectrum).
Before moving further on, please note that this is a very brief explanation about the Fourier series and Fourier transform with a lot of details omitted and the mathematical interpretation presented here in more of a convenient way than the absolutely correct. For more information on the topic, I would strongly suggest the textbook Mathematical Methods for Engineers and Scientists 3 - Fourier Analysis, Partial Differential Equations and Variational Methods by Dr Kwong-Tin Tang (the first part of the book is relevant here).
Now, if you go from the continuous domain to the discrete domain (we will deal with the discretisation of the free variable here, which is time and not the amplitude as is the reality when you deal with digital signals) you move from the Fourier transform to the Discrete Fourier Transform (DFT). Once again, omitting the "technical" details, in DFT when the signal contains frequency (spectral) components that do not make an integer number of complete periods in the duration of the signal to be analysed, you end up with the phenomenon called leakage (see link above). This is due to the "nature" of the transform, in which the signal is assumed to be periodic. Thus, when you have spectral components that do not complete an integer number of periods in the duration of the signal, you will end up with an amplitude (for this component) other than zero at either the end or the beginning of the signal. If you try to "copy and paste" the signal before and after your original signal (in order to make it periodic for the purpose of the transform) you will realise that you end up with some discontinuity at the point where the original and the "copy-and-pasted" signal starts. In order to reconstruct this discontinuity, you have to introduce an infinite amount of spectral components, which show up as energy in frequencies around the main spectral component (in the case you only have a "pure" (co)sine wave). You can see an example of a single frequency in the picture below.

Now, another concept you will most probably encounter quite often (if not always!) when dealing with DFT is the term window. In the simple case where you apply no windowing function (bear with me a wee bit more for an explanation) to the signal is like applying a "rectangular" window where you multiply all the values of your signal with one, thus, effectively doing nothing to them.
If you apply a windowing function to your signal you effectively suppress the signal at its edges in order to avoid the appearance of discontinuities such as those visible in the second plot on the right-hand side of the image above. Thus you somehow suppress the leakage effect. Please keep in mind that the simplest way of applying a windowing function is to element-wise multiply the samples of your signal with the windowing function in the time domain.
The signal on the right-hand side of the above image after the application of a Hann window would look like in the image below

And for a "clearer" representation of the spectrum before and after the windowing you can see this image

You can see that the peak is not so well localised but the leakage has been constrained to a smaller "neighbourhood" than before. For more information on windowing and window functions, you can have a look at the Wikipedia page (link above at the word "window") and any elementary Digital Signal Processing textbook such as the Digital Signal Processing - Principles Algorithms and Applications by Proakis & Manolakis or Introduction to Signal Processing by Orfanidis (which is freely distributed).


Discrete Time Fourier Transform Pair Discrepancy

This is just normalized frequency versus un-normalized frequency.

Check the capitalization of omega. The capital omega $\Omega$, usually refers to the normalized frequency (with $-\pi$ and $\pi$ corresponding to $-f_s/2$ and $f_s/2$) and the normal omega $\omega$, to the regular angular frequency.

Your two sources just use different convention. There is no contradiction.


Covariance matrix associated with random DC level in Gaussian noise

It is important to add that $w[n]$ is uncorrelated with $A$. This means that

$$E\{Aw[n]\}=E\{A\}E\{w[n]\}=0\tag{1}$$

because we may assume that $w[n]$ is zero mean: $E\{w[n]\}=0$. In this case all cross-terms in the expectation are zero, and we're left with

$$C_{ij}=E\{A^2\}+E\{w[i-1]w[j-1]\}\tag{2}$$

Since $w[n]$ is white, you have $E\{w[i-1]w[j-1]\}=\sigma^2\delta_{ij}$. The expectation of $A^2$ is given by

$$E\{A^2\}=\sigma_A^2+\mu_A^2\tag{3}$$

where $\mu_A=E\{A\}$ is the mean of $A$.

So we finally have

$$C_{ij}=\sigma_A^2+\mu_A^2+\sigma^2\delta_{ij}\tag{4}$$

The expression in your question is only valid if $\mu_A=E\{A\}=0$, i.e., if $A$ is a zero mean random variable.


How to find minimum length of a FIR symmetric filter if I am given DTFT

HINT: How many zeros of $H(e^{j\omega})$ do you count? Does that say anything about the (minimum) order of the corresponding polynomial?


Use feedback from output to input to perpetuate the output of the state machine.  You don't need an infinite number of states to produce an infinitely long output sequence.  For example, a state machine with a single memory element and feedback can be used to implement a first order IIR filter.  An example of such a system has the recursion relationship y[n] = 0.5y[n-1] + x[n] where y is the output x is the input and 0.5y[n-1] is the past output scaled, fedback and summed with the input x[n].


Deconvolving the measurement system from the system response

Sorry for the long winded answer, but making a good acoustic measurement is hard. Here are some of the steps involved (and at some point, we'll actually get to the direct answer of your question).

First, you have to make sure that the system you are measuring is actual linear and time invariant (otherwise you can't deconvolve). This can be tricky on a PC if you use regular sound card drivers. These will route any signals through the Windows Kernel Mixer which frequently applies sample rate conversion and yanks buffers around in time in non- deterministic ways.I strongly recommend bypassing the Kernel mixer.

Second, you need to determine the length of the impulse of your system. A good estimate is the reverb time of the room that you are measuring. It's most convenient to choose a power of 2; for most rooms this will 16384 or 32768 at 44.1kHz or 48 kHz sample rate.

Third, create a periodic excitation of that length. This could either be a log sweep or (better) a pseudo random noise. The spectrum of the noise should be chosen so you get roughly constant signal to noise ratio in the frequency range of interest. That depends on the transfer function and on the background noise spectrum. If you don't know either of those yet, pink is a good start. Let's call one period of this signal x[n].

Fourth, wire up your system so that the left channel from your D/A goes into the loudspeaker AND into the left channel of your A/D. Wire the microphone into the right channel of the A/D.

Fifth, start the excitation (loop it or create a wave file with many repetitions of your noise signal). Carefully monitor all levels: make sure that the A/D sits at about 10 dB below clipping. Make sure the microphone pre-amp sits at about 10 dB below clipping. Make sure the power amp is not clipping and that the loudspeaker is not over driven.

Sixth, make sure that the room is as quiet as it can be. Close doors and windows. Shut off most things with a fan including any HVAC system. Evict all other people from the premises. If there are any ground loops, use isolation transformers and ground lifts as necessary. A good way of checking noise is to hook up a headphone amp to the mic output and listen to it through headsets. Any noise, hum, or other artifact that you can hear will also show up in the measurement.

Seventh, do the actual acquisition. Collect 12 periods with the excitation signal running. Visually inspect the results for something unusual (gaps, parts missing, drop outs, etc.)  Throw the first two periods away. Calculate the average over the other 10. Let's call the left channel y[n] (sound card) and the right channel m[n] (microphone).

Eighth, calculate the Fourier Transform of y[n]. This should be fairly flat with no zeros or areas of very low energy. This may not be the case as most sound cards have AC coupled inputs, i.e. there is some high pass filter and the value at DC might be very low.  Similarly there may also likely be an anti aliasing filter, so again you may have low energy or just noise at very high frequencies. If can you manually fix that (by adding a small amount of broadband energy), then do. If the spectrum is very flat in the frequency area of interest, you can simply replace it with a properly delayed unit impulse. If none of these will work, things get more complicated.

Ninth: deconvolute. Your transfer function can simply be calculated as $$H(\omega) = \frac{\Im \left \{ m(t) \right \}}{\Im \left \{ y(t) \right \}}$$
where $\Im \{ \}$ denotes the Fourier transform. This is direct spectral division without any zero padding or windowing. Since we are doing circular coherent acquisition we are also doing circular deconvolution.

The equation shows the problem discussed in step 8. We are dividing by a measured spectrum. Any zeros or areas of low energy or high noise in the divisor spectrum will result in garbage in the resulting transfer function. The results at the "good" frequencies will still be okay but you could not use it to, for example, calculate an impulse response. A simplified version would be to just divide by the excitation signal, i.e.
$$H(\omega) = \frac{\Im \left \{ m(t) \right \}}{\Im \left \{ x(t) \right \}}$$ and choose the excitation so that there are no zeros in the spectrum. However this will not allow you to deconvolve the D/A and A/D system response and it will also lose any absolute delay information. In many cases this is actually acceptable and many commercial measurement system do just that, so it depends on your application.

Tenth: verify that your measurement is good. Various tests should be performed:

Measure a few times and make sure that the result is the same.
Measure with the loudspeaker muted. This will give you a good estimate of the background noise spectrum. As a rule of thumb, you need at least 10 dB signal to noise ratio at all frequencies of interest.
Linearity test: measure with half the excitation gain and confirm that the resulting transfer function is the same.
A useful inline noise test is the following: Take your 10 periods and average 5 times over 2 periods each, then do a Fourier transform over 2*N samples. If your signal is noise free then all the odd bins should be zero. You can directly estimate the signal to noise ratio at any frequency as X{2*N+1}/X{2*N}


identify a specific table in document image

The simplest approach to this would be template matching via two-dimensional normalised cross correlation.

What this basically does is looking for a template: a small image representing the response letters, inside a (usually bigger) image. The result that is returned from this operation looks like an image but it essentially has peak values where the template matches the image.

This approach transferred to this problem means that one cross correlation would have to be carried out for each possible answer and each resulting image examined for "peaks".

Of course, this assumes that the letters appear at a single scale throughout the document. Otherwise, cross correlations would have to be applied (via an appropriate method) at different scales too.

There are two components to be carefully managed here:

Quality and scale of the image:

This can be roughly controlled by instructing users to align the top left corner of their camera's field of view with the paper. This will constrain the distance from the image but it still depends on the lens of the mobile phone.
Quality and scale of the template:

This depends on #1 and the hand-writing of the users. But on average, roughly equal size fonts can be used (as further below).

Here is an example based on Python, matplotlib and scikit-image:

from matplotlib import imshow, show
from skimage.io import imread
from skimage.feature import match_template

#Load the image
Q = imread("tst4.jpg")

#Load the template, here it is the letter A from the Sans font, approximately at the same size as the hand-written provided
A = imread("letter2.png")

U = match_template(Q, A, pad_input = True)

#Show the result
imshow(U);show()

(Please note, a more elaborate version of that is available via this link)

The result U looks like this:

For a Q that looks like this:

And an A that looks like this:

In the U image, please note that the red colour corresponds to high peaks (very close to 1.0). Just putting a threshold on the pixel values of U will return the centers of the 'A' symbols.

Of course, other formations seem to have high correlation as well in that image. These are "false-positives", that is, places where the algorithm THINKS there is an "A" but in reality there is not.

There are two ways by which these can be reduced (and increase the system's reliability):

Apply the template matching over a smaller area (this is now possible if the top left corner is roughly aligned with the edge of the paper)
Add more features to the classification operation than just the peak of the cross-correlation between the template and the image to make it more robust. This is probably the next level in complexity and more information can be provided if needed.

Hope this helps.