CNN - imbalanced classes, class weights vs data augmentation

Is this approach better than the mere
  augmentation or just the use of class weights ?

Note that data augmentation is the process of changing the training samples (e.g. for images, flipping them, changing their luminosity, adding noise, etc.) and adding them back into the set. It is used for enriching the diversity of training samples, thus, in this aspect it cannot be replaced with class weighting. However, it is related to over-sampling since we can both enrich and increase the size of a class via data-augmentation. For now we are only focusing on the size increasing part, i.e. data over-sampling.

Data-oversampling and class weighting are equivalent. Copying the samples of a class 3X is equivalent to assigning a 3X weight to the class. However, the weighting is better from storage and computational point of view since it avoids working with a larger data-set.

Note that based on this equivalency we can mix and match. Increasing the size of a class 6X $\equiv$ increasing the weight 6X $\equiv$ increasing the size 3X and increasing the weight 2X.

Class weights vs over-sampling in more detail

I copied my answer from here, since the solutions are almost the same but questions are the reverse of each other.

This challenge can be tackled in two places:

Data: as you mentioned, this is done by artificially increasing the number of samples for under-represented class $uc$. This produces the same effect as data-sets that are naturally balanced,
Model: this is generally done by over-penalizing the miss-classification of $uc$ compared to other classes. One place for this modification is the loss function. A frequently used loss function in classification is cross entropy. It can be modified for this purpose as follows. Let $y_{ik}$ be $1$ if $k$ is the true class of data point $i$, and be $0$ otherwise, and $y'_{ik} \in (0, 1]$ be the corresponding model estimation. The original cross-entropy can be written as:
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$
which can be weighted as
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}\color{blue}{w_{k}}y_{ik}log(y'_{ik})$$
For example, by setting $w_{uc} = 10$ and $w_{k \neq uc}=1$, we are essentially telling the model that miss-classifying $1$ member from $uc$ is as punishable as miss-classifying $10$ members from other classes. This is roughly equivalent to increasing the ratio of class $uc$ $10$ times in the training set using method (1).
As a concrete example,  suppose ratio of three classes are $c_1=10\%$, $c_2=60\%$, and $c_3=30\%$, thus, we can set $w_{c_1}=6$, $w_{c_2}=1$, and $w_{c_3}=2$, which implies $w_{c_i} \times P(c_i)=w_{c_j}\times P(c_j)$ for all class pairs $c_i$ and $c_j$.


I highly recommend you take a look at this anomaly detection survey paper. It not only provides with a great review but also categorises the different approaches focusing in different types of data or domains where these techiques have been applied.

If you go through Section 1.3, you will find a series of interesting references which go as far as 1985. So you could have a look at those to make sense of the historic evolution of these approaches from numerical, statistical or ML standpoints.

Also, since the paper is quite dated already, I suggest a reading about new methods such as deep learning methods to complement your knowledge. A nice starting point could be this deep learning survey paper, which I think is another great more recent survey paper.


How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that :
\begin{equation}
    T = P
\end{equation}

\begin{equation}
    W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI
\end{equation}

Applying a prototype vector as input :

\begin{equation}
    \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k
\end{equation}

Because they are orthogonal we have :
\begin{equation}
    \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I})
\end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation}
    W \cdot p_k= (R - Q \cdot I) \cdot p _k
\end{equation}
prototype vectors
continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's
ability to generalize or may increase the biological
plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used


What is the effect of highly correlated data on a Convolutional Neural Network?

I actually don't believe it really matters.  When applying the same idea to visual problems, you'll find that the raw RGB vs a DCT compressed version lead to similar results as seen here.  If there exist a negative, one may argue that the correlated data may have a slower learning rate.  Nevertheless, the best way to figure out the effect is to give both a try.


Different R Markdown in same document for each output type

I was able to get different inclusions and rendering with the following document:

---
output: html_document
---

```{r, results='asis', echo=FALSE}
library(knitr)
pandocto <- opts_knit$get("rmarkdown.pandoc.to")
if (pandocto == "docx") {
  cat("## Word Heading\n\nHello world")
} else if (pandocto == "html") {
  cat("## HTML Heading\n\nHello world-wide-web")
} else if (pandocto == "latex") {
  cat("## LaTeX/PDF Heading\n\nHeLlO wOrLd")
}
```

```{r}
str(opts_knit$get()
```

If you vary the head YAML with what you need, perhaps one of:

output: html_document
output: pdf_document
output: word_document

you can see the difference. The tricks are knowing that rmarkdown tells you the format to which pandoc is rendering, and the use of results='asis' in the block header to capture the output as raw-markdown (or latex, as appropriate).

I included the second block (str(...)) so you can see what else is "known" during document compilation/rendering.


When should you balance a time series dataset?

If you can change the Loss function of the algorithm, It will be very helpful and as a result you don't need to down sample your data. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)


Is there an implementation of pocket algorithm in Keras?

Yes this is usually part of the early stopping algorithm, where you supply a cross-validation data set, and a limit on number of epochs since best result so far.

In Keras, you can use an instance of the EarlyStopping class, choosing the metric that you want the best model for, and setting the patience parameter to limit the number of epochs to test after any best so far result. The instance is supplied to the fit method as a callback.

See http://parneetk.github.io/blog/neural-networks-in-keras/ for an example (last section)


You have not defined the variable predictions anywhere. You will need to get them from your classifier somehow. You have fit your nordan_tree on your training data, now you can use the fitted nordan_tree to generate the predictions, for example like this:

predictions = nordan_tree.predict(X_test)

Then your line of:

print(accuracy_score(y_test, predictions))

should work.


Be aware that posting code in images very annoying to copy/paste and it's bad for web reference ment.

This is due to the derivative of the softmax, but to me it's seems fishy.

If $S$ is the softmax vector, then the Jacobian $DS$ consists of $S_j(\delta_{ij}-S_i)$. This could explain the -=1 part, but not the /=N, and not the shape either.


how to apply similarity algorithm(or comparision) of over one million vectors with other one million vectors?

You might want to take a look at this project FAISS.

  Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.

It has nice wrappers for you to use from Python. Check the wiki for examples on how you can integrate this in your application.

It is built with scalibilty in mind, supporting indexing and comparisons of millions of vectors and it can be run either on CPU or GPU.


Difference between 1x1 Convolution and TimeDistributed(Dense())

First thing is that TimeDistributed should not be useful for your case.
It helps when having 2D data consisted of timesteps (ex. in recurrent forms) and you only have one dimension : (64, 1).

Secondly to answer, yes, your intuition is right. Conv1D with size 1 and a channel size of 1 is equivalent to a Dense of size 1.


Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features?

Tree Based models like RF and XGBoost can handle the feature space created by either of the methods you suggested above. Additionally, you can try to do feature selection (use those statistical tests) to eliminate 0 variance features and then feed the filtered dataset to your models.


Why is there a big drop off in performance in my GBM?

I would first suggest trying to plot the results during training. How do your metrics (or at least the loss) vary over the training process for training and cross-validation datasets?
The loss at each iteration is appended to your GBC object gb_model in the train_score_ attribute.

Normally, when there is such a big gap between training and test data, it indicates that you are overfitting to your training data, and that the model does not generalise well to unseen data. You could think about doing  shuffling your data, in order to balance the training/validation/test datasets - [if you are looking at a time-series problem, you should be careful as to how you do this].


How can I detect anomalies/outliers in my online streaming data on a real-time basis?

There are two well known algorithms called Isolation Forest and One-Class SVM for outlier detection. You will find implementations of these in Sckikit learn.

Doing a search for "Anomaly Detection" on github, there seem to be entries to the NAB competition available publicly eg. nareshkumar66675/Numenta. This one has a Jupyter notebook which mainly uses Scikit learn and some custom, but simple, feature engineering. They may serve your purpose. Although the author of this one has not included Licensing information, it seems simple enough to re-implement.

However, as I understand it, the NAB datasets are more "time series" detection, i.e. a signal is an anomaly if it is very different from previous / recent values. It does not have any notion of patterns in the data, as sine pulses after step pulses, and does not include learning larger patterns as the dataset grows in size.

I'm not aware of algorithms solving your specific problem, though they might well exist in the literature. The key issue in your problem is that you cannot predict if a long sequence is an anomaly until you've seen enough data. It may suffer from combinatorial explosion.

The sines and pulses of your problem can be replaced with 0s and 1s, so your problem is one of detecting patterns in strings. Genomics is concerned with patterns in DNA, so that  body of work may have what you need. (Note that is very different from Genetic algorithms)

There is an older set of algorithms called variously, Market Basket Analysis, the Apriori algorithm or Association Set Mining which has the flavor of increasing set size, but not anomaly detection. See this video explaining it. Apriori creates sets of items commonly bought together. When you have small amounts of data, you can reliably create only small patterns. As the amount of data increases you can create larger patterns.


The model performance vary between different train-test split?

What you are experiencing is not a problem, but rather an inherent attribute of all classifiers. The performance of a classifier depends on the training set, therefore the performance will vary with different training sets.

To find the best parameters for a specific classifier you will therefore want to vary training and test split (such as in crossvalidation) and choose the parameter set which achieves the best average accuracy or AUC.

Finally you will want to test the trained classifier on another dataset - evaluation set - which has not been part of the dataset used in crossvalidation.


LSTM predicting expected value and standard deviation with one-hot encoding

The standard deviation and mean of a categorical variable is not meaningful. It looks like the original data are from a range of [0, 20), and the space has been discretized. Now, instead of ranges, you have ordered categories 1 through 4. The individual numbers within each category have lost their meaning. For more details, see the accepted answer here


Disclaimer, I have not tried any of these ideas.

Predict the CI directly

Note: This method will require very large batch sizes, which may not be possible due to memory constraints.

Set the batch size to something large. You'll also need to decide what you want your confidence interval to be, 95% matches a lot of expectations but may be less numerically stable than something like 90% of 80%. We'll call this value $0 < c < 1$.

In addition to outputting a prediction $\hat{y_i}$ of the true value $y$, have your model also output a lower bound $l_i(c)$ and an upper bound $u_i(c)$ for each individual estimate.

Now, define
$$
z =
\begin{align}
& \left(\frac{1-c}{2} - \frac{\sum_{i=1}^n (1\ \text{ if }\ (y_i < l_i(c))\ \text{ else }\ 0)}{n} \right)^2 \\
+ & \left(\frac{1-c}{2} - \frac{\sum_{i=1}^n (1\ \text{ if }\ (y_i > u_i(c))\ \text{ else }\ 0)}{n} \right)^2
\end{align}
$$

In essence, $z$ is the (squared) measure of how many times the predicted lower/upper bounds are exceeded vs. how many times we expect them to be exceeded. In practice, some tweaks to exactly how $z$ is computed may be necessary, but the main idea of penalizing the model for having too lax or too restricted in its estimations of the lower/upper bounds should be preserved.

Add $z$ to your MSE loss function, you'll likely need to balance these two losses, i.e. add an additional hyperparameter $\alpha$ to your model such that $\text{TOTAL LOSS} = \text{MSE} + \alpha z$.

As an example, if you set your batch size/time-step length so that there are 500
distinct estimates (e.g. $n=500$), and set $c$ to 0.9, in theory this training procedure should encourage the model to estimate $l_i(c)$ and $u_i(c)$ such that exactly 25 actual $y_i$ values are below $l_i(c)$ and exactly 25 $y_i$ are above $u_i(c)$.

Predict the error directly

In addition to getting a prediction $\hat{y}$ of the true value $y$ and training with the MSE loss function, add another output of your model which attempts to predict the squared error $\epsilon = (\hat{y} - y)^2$ directly.

You can create the MSE of the estimate $\hat{\epsilon}$ of $\epsilon$ and combine that with your existing MSE to get the total loss for training.

You can then use this value to normalize your errors to try and get an anomaly score that is usable across different sub-populations:

$$
\text{anomaly score} = \frac{(y - \hat{y})^2}{\hat{\epsilon}}
$$
where higher scores indicate that the true error is larger than expected.


Predict the first observations of a time series when order of the model is higher

The obvious answer is to use the last 4 data points of your training dataset. Note that there is no harm or bias in doing it. The purpose of breaking the data set into training and test datasets is to estimate and forecast on different datasets.
In your model, the dependent variable is $y_5$ to $y_{183}$. On the other hand, the four explanatory variables are: $y_1-y_{179}$; $y_2-y_{180}$; $y_3-y_{181}$; and $y_4-y_{182}$.
So the test data set for each of the explanatory variable actually starts from $y_{180}$, $y_{181}$, $y_{182}$ and $y_{183}$ respectively.
Therefore, by using the last four data points of your train dataset, you are not introducing any bias in terms of revalidating the model on fitted data.


Building document classifier based on keywords, what would be the steps?

Got your email...

Built this example repository for ya:
https://github.com/dancrew32/resume_classifer/

If you go through the notebook, I outlined the process of setting up labels, different folders for different kinds of resumes, loading them up, vectorizing text, classifying using logistic regression.

https://github.com/dancrew32/resume_classifer/blob/master/Resume%20Doc%20Classifier.ipynb

Let me know if you have any questions.

P.S. Never adding a skip button ;)


How to calculate customer purchase interval and predict next purchase in python?

With Lifetimes it's very straight forward to implement.


PySpark: How do I specify dropna axis in PySpark transformation?

I  know this is a bit late, but I struggled with this also. This is my attempt at removing null columns from a Spark Dataframe.
from pyspark.sql.functions import when, isnull

colsthatarenull = df.select([(when(isnull(c), c)).alias(c) for c in df.columns]).first().asDict()
namesofnullcols = {key:val for key, val in colsthatarenull.items() if val != None}.values()
df = df.drop(*namesofnullcols)


The problem seems to come from your learning rate and the non-normalization of your data. Here your network is clearly unstable and thus gets to sky high values (10^20) which lead to NaN values. A typical learning rate for SGD is 0.001, but this is for normalized datas (inputs-outpus between 0 and 1). Here your inputs and ouputs have high values, that are amplified even more by MSE (which squares the error). So this is why there is an issue with the learning rate here, the resulting gradient is way too strong.
This is how I understand the network behaviour, that is clearly unstable with lr=0.02.
A way to solve it is greatly diminish your learning rate (lr=0.00000001 worked for me). Another way around is to Normalize your data (inputs and ouputs)
Here is your modified code, you can try changing the learning rate and see how it changes the network behaviour (from stable to unstable).
import torch
import torch.nn as nn
import torch.optim as opt
import numpy as np
import matplotlib.pyplot as plt
import math
import os

os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"  # This line may be useless

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.l1 = nn.Linear(100, 100)
        self.l2 = nn.Linear(100, 1)

    def forward(self, x):
        x = self.l1(x)
        x = self.l2(x)
        return x

mlp = MLP().float()

target = nn.MSELoss()
o = opt.SGD(mlp.parameters(), lr=0.00000001)  #Try 0.0000001 and 0.000001

w = torch.tensor(np.random.rand(100, 1) * 3).float()
x_train = torch.tensor(np.random.rand(100, 100)*100).float()
y_train = torch.mm(x_train, w) + 2

test_x = torch.tensor(np.random.rand(100, 100)*10).float()
test_y = torch.mm(test_x, w) + 2

losslist = []
losstestlist = []

for epoch in range(100):
    op = mlp(x_train)
    o.zero_grad()
    loss = target(op, y_train)
    losslist.append(math.log10(loss))
    loss.backward()
    o.step()
    test_pred = mlp(test_x)
    loss = target(test_pred, test_y)
    losstestlist.append(math.log10(loss))

plt.plot(losslist, 'r', label='train loss')
plt.plot(losstestlist, 'b', label='test loss')
plt.legend()
plt.show()

Hope this helps.


which neural network topology to learn correlations between time series?

It depends a little on what kind of correlations you're looking for. Are you expecting a correlation that is present at each time step/window, or a different level of correlation per time step/window? Are you doing a classification or a regression task? Sometimes predicting the next value involves classification, but I'll assume you're looking for regression for now.

As a starting point, try feeding each of those sequences separately as input to a recurrent neural network (start with a basic LSTM and pare it down if it's overkill). I like your suggestion of overlapping windows.

For example (pseudocode-ish):

series_1 = 1, 2, 3, ..., 100

series_2 = 5, 6, 7, ..., 200

input_1 = Input(series_1,       window_size)

input_2 = Input(series_2, window_size)

layer_1 = LSTM(input_1, input_2)

final_layer = fully_connected(layer_1)


When should we consider a dataset as imbalanced?

I think subsampling (downsampling) is a popular method to control class imbalance at the base level, meaning it fixes the root of the problem.  So for all of your examples, randomly selecting 1,000 of the majority of the class each time would work.  You could even play around with making 10 models (10 folds of 1,000 majority vs the 1,000 minority) so you will use your whole data set.  You can use this method, but again you're kind of throwing away 9,000 samples unless you try some ensemble methods.  Easy fix, but tough to get an optimal model based on your data.

The degree to which you need to control for the class imbalance is based largely on your goal.  If you care about pure classification, then imbalance would affect the 50% probability cut off for most techniques, so I would consider downsampling.  If you only care about the order of the classifications (want positives generally more higher than negatives) and use a measure such as AUC, the class imbalance will only bias your probabilities, but the relative order should be decently stable for most techniques.

Logistic regression is nice for class imbalance because as long as you have  >500 of the minority class, the estimates of the parameters will be accurate enough and the only impact will be on the intercept, which can be corrected for if that is something you might want.  Logistic regression models the probabilities rather than just classes, so you can do more manual adjustments to suit your needs.

A lot of classification techniques also have a class weight argument that will help you focus on the minority class more.  It will penalize a miss classification of a true minority class, so your overall accucracy will suffer a little bit but you will start seeing more minority classes that are correctly classified.


Pandas Groupby makes kernel die in Jupyter notebook/Python

I thought it might be because of the different types being used in the columns, but I created an example below, which works fine over mixed column types. The only real different is the size - that is why I think you are probably running out of memory.

  Working example

I use int, str and datetime objects:

In [1]: import pandas as pd

In [2]: import datetime

In [3]: df = pd.DataFrame({'Branch': 'A A A A A A A B'.split(),
                           'Buyer': 'Carl Mark Carl Carl Joe Joe Joe Carl'.split(),
                           'Quantity': [1, 3, 5, 1, 8, 1, 9, 3],
                           'Date':[datetime.datetime(2013, 1, 1, 13, 0),
                                   datetime.datetime(2013, 1, 1, 13, 5),
                                   datetime.datetime(2013, 10, 1, 20, 0),
                                   datetime.datetime(2013, 10, 2, 10, 0),
                                   datetime.datetime(2013, 10, 1, 20, 0),
                                   datetime.datetime(2013, 10, 2, 10, 0),
                                   datetime.datetime(2013, 12, 2, 12, 0),
                                   datetime.datetime(2013, 12, 2, 14, 0)]})

In [4]: df
Out[4]:
  Branch Buyer  Quantity                Date
0      A  Carl         1 2013-01-01 13:00:00
1      A  Mark         3 2013-01-01 13:05:00
2      A  Carl         5 2013-10-01 20:00:00
3      A  Carl         1 2013-10-02 10:00:00
4      A   Joe         8 2013-10-01 20:00:00
5      A   Joe         1 2013-10-02 10:00:00
6      A   Joe         9 2013-12-02 12:00:00
7      B  Carl         3 2013-12-02 14:00:00

In [5]: df.shape
Out[5]: (8, 4)

Now I just repeat the dataframe again, but add one hour to each of the datetime values, just to increase the number of groupby combinations to expect:

In [14]: df.iloc[0:8, 3] += datetime.timedelta(hours=1)

Now perform a groupby over all columns, and sum only on Quantity (it is my only numeric column).

The reuslts are as expected:

In [16]: df.groupby(["Branch", "Buyer", "Quantity", "Date"])["Quantity"].sum()
Out[16]:
Branch  Buyer  Quantity  Date
A       Carl   1         2013-01-01 13:00:00    1
                         2013-01-01 14:00:00    1
                         2013-10-02 10:00:00    1
                         2013-10-02 11:00:00    1
               5         2013-10-01 20:00:00    5
                         2013-10-01 21:00:00    5
        Joe    1         2013-10-02 10:00:00    1
                         2013-10-02 11:00:00    1
               8         2013-10-01 20:00:00    8
                         2013-10-01 21:00:00    8
               9         2013-12-02 12:00:00    9
                         2013-12-02 13:00:00    9
        Mark   3         2013-01-01 13:05:00    3
                         2013-01-01 14:05:00    3
B       Carl   3         2013-12-02 14:00:00    3
                         2013-12-02 15:00:00    3
Name: Quantity, dtype: int64

  Break your problem down

It might be difficult to break down your problem, because you need to whole data for the groupby operation. You could however save each of the groups to disk, perform the mean() computation on them separately and merge the results yourself. The name of each group is actually the combination of the groupby columns selected. This can be used to build the index of the reuslting dataframe.

It could look something like this:

for name, group in df1.groupby(['date', 'unit', 'company', 'city']):
    print("Processing groupby combination: ", name)    # This is the current groupby combination
    result = group.mean()
    _df = pd.DataFrame(index=[name], data=[result])
    _df.to_csv("path/somewhere/" + name + ".csv

You will then have a folder full of the results for each group and will have to just read them back in and combine them.

  Other methods

It is known that Pandas does not handle many operations on huge datasets very efficienty (compared to e.g. the data.table package). There is the Dask package, which essentially does Pandas things in a distributed manner, but that might be overkill (and you'll of course need more resources!)


Simple example of Parzen window (kernel density estimation)

In kernel density estimation, rectangle, triangle, or Gaussian  kernels assign weight to positions around query point $x$. For rectangle and square, the weight is steady, for triangle, weight drops linearly with distance, and for Gaussian, weight drops exponentially with distance. Here is an image for square, rectangle, and triangle kernels (one dimensional).

Here is the formula for kernel density estimation in one dimension:

$$\hat{f}_{h}(x) =\frac{1}{nh}\sum_{i=1}^{n}K\left(\frac{x-x_i}{h}\right)=\frac{1}{n}\sum_{i=1}^{n}K_h\left(x-x_i\right)$$

where scaled kernel $K_h$ is defined as:
$$K_h(x-x_i) := \frac{1}{h}K(\frac{x-x_i}{h})$$

Example

We have two training points located at $0.5$, and $0.7$ in one dimension.

If we assume query point is $x=0.5$, and $h$ is 0.3 (for rectangle and triangle, the scaled width is 0.6 from -0.3 to 0.3), for rectangle kernel we have:
$$\begin{align*}
\hat{f}_{0.3}(0.5) &=\frac{1}{2*0.3}\left(rect \left(\frac{0.5-\color{blue}{0.5}}{0.3} \right)+rect \left(\frac{0.5-\color{blue}{0.7}}{0.3} \right)\right) \\
&= \frac{1}{0.6}(0.5+0.5) = 1.67
\end{align*}$$
and for triangle kernel we have:
$$\begin{align*}
\hat{f}_{0.3}(0.5) &=\frac{1}{2*0.3}\left(tri \left(\frac{0.5-\color{blue}{0.5}}{0.3} \right)+tri \left(\frac{0.5-\color{blue}{0.7}}{0.3} \right)\right) \\
& = \frac{1}{0.6}(1+0.33) = 2.22
\end{align*}$$

For square kernel $sqr$, kernel outputs 1 for each point inside the hyper-cube, and $h$ would be the width of scaled hyper-cube (from $-h/2$ to $h/2$). For the square kernel, estimation in $d$ dimension is:
$$\hat{f}_{h}(x) =\frac{1}{nV}\sum_{i=1}^{n}sqr\left(\frac{x-x_i}{h}\right)=\frac{1}{nV}k=\frac{\frac{k}{n}}{V}$$

where $k$ is the number of points inside a hyper-cube centered at $x$ with width $h$, and $V=h^d$ is the size of hyper-cube in $d$ dimensions (length of a segment in one dimension).

The previous example for $h=0.3$ would be:
$$\begin{align*}
\hat{f}_{0.3}(0.5) &=\frac{1}{2*0.3}\left(sqr \left(\frac{0.5-\color{blue}{0.5}}{0.3} \right)+sqr \left(\frac{0.5-\color{blue}{0.7}}{0.3} \right)\right) \\
& = \frac{1}{0.6}(1+0) = \frac{\frac{1}{2}}{0.3} = 1.67
\end{align*}$$
Note that these are estimations for density not probability, thus, they are allowed to be larger than one (imagine a rectangle with width 0.5 and height 2 as pdf of a continuous random variable).


How can i decide on which features to use for clustering?

Good points by shepan6. It is also worth to mention about feature extraction techniques to lower the number of  features. You can use common feature extraction techniques such as PCA or eigen projections to transform the data into a lower dimensions. After lowering the number of features like this you can appply  techniques such as clustering


A shot is a single example available for machine learning. So "one-shot" means you're given just the one example for each class.


LASSO remaining features for different penalisation

Lambda is a tuning parameter („how much regularisation“, I think called alpha in sklearn) and you would choose lambda so that you optimise fit (e.g. by MSE). You can do this by running cross validation.

This page (for the GLMnet package in R) explains how to apply Lasso in a very instructive way (alpha is the elastic-net mixing parameter here, Lambda is the tuning parameter).

You may also look at „ Introduction to Statistical Learning“, Ch. 6.2, in which Lasso is discussed in a very good way.

There are also Python Labs for the book, which should give you a blueprint for how to use Lasso in Python (see Section 6.6.2).


Many things behave differently in high dimensional space

Along each direction for a unit cube, we have $2$ boundaries. To be less than $0.01$ from a bounday in a $d$-dimensional unit cube, it is not inside the cube of side length $1-2\times 0.001$ sharing the same centroid as the original cube.

The corresponding volume for that is $(1-2\times 0.01)^d$ where $d$ is the coresponding dimension.

$$\left(1-\left( 1-2\times 0.001\right)^2\right) \times 100 \% \approx 0.3996\%$$

$$\left(1-\left( 1-2\times 0.001\right)^{10,000}\right) \times 100 \% \approx 99.999999798\%$$

When we first computed the answer for $2$ dimension, we might carelessly conclude that most points are not closed to the boundary. The moral of the story is be careful not to generalize what we believe without proof to higher dimension.

Your second question is also known as Hypercube line picking of which its computation is non-trivial. We know that

$$\frac13 \sqrt{d} \le \Delta(d) \le \sqrt{\frac{d}{18}\left( 1+2\sqrt{1-\frac{3}{5d}}\right)}$$


I'd start here. Most basic idea is to run statistical tests to see how target variable depends on each feature. These include tests like chi-square or ANOVA.
Tree-based models can also output feature importance. Check this post. There's plenty of posts on kaggle with code. Might be worth checking those:

https://www.kaggle.com/willkoehrsen/introduction-to-manual-feature-engineering
https://www.kaggle.com/rejasupotaro/effective-feature-engineering
https://www.kaggle.com/willkoehrsen/automated-feature-engineering-tutorial

As your data set isn't so drastically large, you could push grid search and check how your model behaves for different factors of PCA.

It's hard to tell a priori whether you should drop some features. I guess trying each combination of 30 features is completely out of scope, though you might try dropping most redundant ones.

As your data contains categorical features, it might be good idea to give catboost a try. They claim it handles categorical features better than other gradient boosters. Just keep in mind, that default number of estimators is 10 times of that in xgboost. You might lower it for experiments.

First, I'd create base model with all the features. Now comes the question: which method to choose? Gradient boosters poses ability of learning the feature importance, those redundant ones will get little weight and you might not see much of an improvement, when dropping features. You might get more insight using more vanilla methods, but in the end you'll be certainly deploying gradient boosting to production, so I don't see much sense in it. I'd stick with xgboost or catboost and perform experiments using same parameters.

Please keep in mind: though some features might be highly redundant, they may still contribute some knowledge to your model.


Gradients for bias terms in backpropagation

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered.

The derivative of L (loss) w.r.t. b is the upstream derivative multiplied with the local derivate:
$$
\frac{ \partial L}{\partial \mathbf{b}} = \frac{ \partial L}{\partial Z} \frac{ \partial Z}{\partial \mathbf{b}}
$$

If we have multiple samples Z and L are both matrices. b is still a vector.

The local derivative is simply a vector of ones:
$$
\frac{ \partial Z}{\partial \mathbf{b}} = \frac{\partial}{\partial \mathbf{b}} W \times X + \mathbf{b} = \mathbf{1}
$$

That means our complete derivative is a matrix multiplication, that looks as follows (e.g. 2 samples with 3 outputs):
$$
\frac{\partial L}{\partial Z} \times \mathbf{1} =
\begin{bmatrix}
. &.  &. \\
. &.  &.
\end{bmatrix}
\begin{bmatrix}
1\\
1\\
1\\
\end{bmatrix}
$$

Note that this is the sum of the rows.

And that's where db2=np.sum(dz2, axis=0, keepdims=True) comes from. It is simply an abbreviation for the matrix multiplication of the local and the upstream derivatives.


Recommended Tutorial Videos or Books on Feature Engineering Using Python

There are several articles and tutorials online. For example,
PythonDataScienceHandbook or towardsdatascience.
Depend on your particular problem, you can also apply the Automated tools for feature engineering.
Most of these materials have python code and links for your learning.


Training Deep Nets on an Ordinary Laptop

Yes, a laptop will work just fine for getting acquainted with some deep learning projects:

You can pick a smallish deep learning problem and gain some tractable insight using a laptop so give it a try.

The Theano project has a set of tutorials on digit recognition that I've played with and moded on a laptop.

Tensorflow also has a set of tutorials.

I let some of the longer runs go overnight, but nothing was intractable.

You might also consider availing yourself of AWS or one of the other cloud services.  For 20-30 dollars you can perform some of the bigger calculations in the cloud on some sort of elastic computing node. The secondary advantage is that you can also list AWS or other cloud services as skill on your resume also :-)

Hope this helps!


Is there something like class-based object detection? Or class-based selective search?

Object detection models (such as SSD, Faster-RCNN, YOLO, R-FCN) are trained to detect specific classes.

If you wish to detect a single class, you could train a custom model on this class.


Overfitting with sklearn pipeline - reasons why?

If your dataset is giving really different results for different folds and you just have a scaler and a model in your pipeline could be for different reasons:

The splits of your validation have different difficulties for some reason. Some are just easy to predicto of you haven't choosen the proper validation and crossfold. An example of this would be that there is a temporal dependency and you are not getting it. If this was the case my reccomendation will be to choose a proper validation and crossfold.
Other possible problem would be that the features are not equally distributed, meaning that there are some outliers. If there is outliers when you scale you will have different intervals. If this was the case I would say that before doing anything more complicated doing some pre processing for the whole dataset.

Note: Doing what Kagglers do of concateneting train and test can be dangerous when you put your model into production.


Data augmentation for multiple output heads in Keras

Please refer to the source code provided at https://gist.github.com/swghosh/f728fbba5a26af93a5f58a6db979e33e which should assist you in writing custom generators (basis ImageDataGenerator) for training end to end multi-output models. In the provided example, GoogLeNet is being trained which consists of two auxiliary classifiers and thus, comprising 3 outputs in the complete model.

The output from ImageDataGenerator.flow_from_directory() have been passed into a Python function with yield statements such that the targets can be repeated thrice as per requirement.

def three_way(gen):
    for x, y in gen:
        yield x, [y, y, y]

Also, please make sure to pass a value to steps_per_epoch argument in case of model.fit() or model.fit_generator() so as to ensure that the training loop doesn't run indefinitely. (This is because the Python generator object won't be inherited from Sequence class and number of batches/steps cannot be inferred implicitly by Keras)

Also, you may consider using an image data pipeline constructed using tf.data APIs combined with Random* Layers and Preprocessing Layers which is a more modern approach for tf.keras compared to the old ImageDataGenerator. There should be a bunch of performance and usability benefits in case you transition. Let me know if you are looking for an example.

Please look at this example in case you want to implement multi-output multi-label classification model. https://www.pyimagesearch.com/2018/06/04/keras-multiple-outputs-and-multiple-losses/

I'd still advise to use tf.data.Dataset instead of raw tensors which'd be more feasible for larger datasets that do not fit into memory plus, the tf.data APIs provides for an additional bunch of flexibilities.


Sampling with replacement, specify the probabilities

Scala:

def weightedSampleWithReplacement[T](data: Array[T],
                                     weights: Array[Double],
                                     n: Int,
                                     random: Random): Array[T] = {
  val cumWeights = weights.scanLeft(0.0)(_ + _)
  val cumProbs = cumWeights.map(_ / cumWeights.last)
  Array.fill(n) {
    val r = random.nextDouble()
    data(cumProbs.indexWhere(r < _) - 1)
  }
}

Spark has an RDD.sample() method that can sample without replacement, though not with weights. You could probably adapt that method along the lines above to do this however.


What's the correct form of use the real coded genetic algorithm?

There are codifications in Genetic Algorithms which use continuous vectors instead of binary vectors, is not mandatory for all GA to work with binaries.

MATLAB also has a function called "ga" which works with these vectors.

Matlab's help on ga function


Is the decision tree the right choice to classify for this dataset?

If you know what you will decide based on the length of the tuple, then you can probably just hard-code your rules! Did the SQL version not work for you? Or are you just experimenting with decision trees?

You can of course use a decision tree. You data can be split based on the features you provide, so you can look at more than the length of the tuple e.g. look at the values of the DestinationIP and your other variables in the tuple.

It could look something like this:

from sklearn import tree
from sklearn.model_selection import train_test_split

Create a train-test split from your data, assuming you have some input tuples and the response that you want to predict:

X = your_tuple_data
y = corresponding_responses_to_predict

# Split dataset into training set and test set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1) # 70% training and 30% test

Create Decision Tree classifer object

clf = DecisionTreeClassifier()
# Train Decision Tree Classifer
clf = clf.fit(X_train,y_train)

Predict the response for test dataset

y_pred = clf.predict(X_test)

Have a look at the SciKit-Learn documentation for more information. Here is a tutorial with more ideas behind why this could work.


Finding sequence combinations that impact target variable the most

With machine learning, I could think that process analytics is what you are looking for:

I imagine you may have a dataset like this:

[step1, step2, step3]  -> [No Target]
[step1, step2]         -> [Target]
...

Business Process Analytics is the family of methods and tools that can be applied to these event streams in order to support decision-making in organizations. The analysis of process events can focus on the behavior of completed processes, evaluate currently running process instances, or focus on predicting the behavior of process instances in the future. Process Analytics

Is the most similar process (to your question) in Analytics I know


How do I use multilevel regression models?

I think what you're looking for is Survival analysis.

From Wikipedia:

  Survival analysis is a branch of statistics for analyzing the expected duration of time until one or more events happen, such as death in biological organisms and failure in mechanical systems.

In your case, you'd like to predict the time a crime will occur in a given region.

Here is a good talk about survival analysis in Python.
This is a python library dedicated to survival analysis and the one used in the video mentioned above. The lib itself has some examples which will also help you understand survival analysis as a whole.
An introduction to the concepts of Survival Analysis and its implementation in lifelines package for Python.

Hope this helps!

EDIT1:

As of the 'per group prediction': your problem is this:

  everything I have found assumes a y value at the individual data row level

There's no such thing as an algorithm to predict a target variable per group. You have to transform your data so it has a 'y' value per group, and then train some model based on that. It may be seen as a regression problem, or maybe as some kind of survival analysis. But you can't escape from the fact that you'll need one target value per group to start doing stuff.


How to prevent neural network from overfitting on small subset of features

I believe that using drop-out is the best solution for such a problem.

I also suggest ensembling methods such as RandomForest instead of a neural network because RandomForest is robust w.r.t overfitting. Also, as you mentioned, your data is not that much complex. So, it would be a good idea to use RandomForest.


How to combine different models in Keras?

You can get the output of your models with model.output or get_layer
and combine them with tf.keras.layers.concatenate


Neural network for a data with 5 inputs and 1 output of 3 different types

It sounds like you want to use a regression model (which produces a real value as the output, e.g. a value in the range [-1, 1] but possibly outside as well depending on the learning algorithm) instead of a classification model (which produces one of a set of discrete output values based on the training data, e.g. {-1, 0, 1} in your case).
If the code is changed to use MLPRegressor instead of MLPClassifier, it will provide a real value as output.
Instead of:
from sklearn.neural_network import MLPClassifier
model = MLPClassifier(...)

Use:
from sklearn.neural_network import MLPRegressor
model = MLPRegressor(...)


If you have R is quite simple

Copy the lines into a file, let's say: "mydata.json"
Be sure you have installed the rjson package

 install.packages("rjson")

Import your data

 library("rjson")
 json_data <- fromJSON(file="mydata.json")


Is the interval variable considered as a type of numerical variable or ordinal variable?

This is the formal definition of an interval (from Wikipedia):

In mathematics, a (real) interval is a set of real numbers that
contains all real numbers lying between any two numbers of the set.
For example, the set of numbers x satisfying 0 ≤ x ≤ 1 is an interval
which contains 0, 1, and all numbers in between.

This definition doesn't tell us what to do to represent an interval for ML purposes... and that's normal :)
For ML purposes we need to decide a suitable representation, and there's rarely a mathematically pure or unique way to represent a mathematical object. Your confusion comes from the fact that the sites you mention propose their own representation, i.e. they consider a particular definition of interval, presumably because it suits their goal. The second one in particular is clearly defining an "interval variable" as an object in a particular language, with constraints which have nothing to do with the above definition.
Now to answer your question: an interval is not atomic, since it's not a single value. So strictly speaking it's neither a numerical nor an ordinal.
But you're probably not very satisfied with this answer, since you still want to be able to use this kind of feature. How? As usual, it depends:

Most of the time an interval can be converted to an ordinal value (note that it's a conversion, i.e. we simplify the data to make it usable). The conversion to ordinal is typically done by numbering the possible intervals (following their order) with integers. If the variable is going to be a label in a classification task, it can even be converted to a categorical variable.
However there are cases where an ordinal would not be a good representation, in particular if the intervals are not equal and the ML method involves distances. In such cases it's probably safer to convert the interval to a numerical value, typically the mean of the interval. This way both order and distances are preserved.


Train and Test Data Split for a Tweet Classification Task

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(all_disaster['cleaned'],specific_disaster['target'], test_size=0.33, random_state=42)

Like this, you train your model on all the disasters data, and you test it on a specific disaster.


Possible to correct an actual cell state in LSTM via gradient?

Is it theoretically possible to correct the contents of the cell state, and what would it result in?

Yes it is. Using back propagation, it is possible to get the gradient of any value that affects a well-defined output. For training data, that includes all current cell outputs - in fact these are necessary to calculate as an interim step in order to get the gradients of the weights. Once you have the gradients of a cost or error function, then you can perform a step of gradient descent in order to discover a value that would result in a lower error for given training data.

In usual training scenarios you do not alter neuron outputs after they have been calculated using feed-forward, because these are not parameters of the model. Typical reasons to alter neuron values (or even the input) are in order to view what the ideal state might be in a given scenario. If your state can be visualised through some decoder - maybe even another neural network - then this could allow you to see the difference between actual internal state, and a potentially better one. That could be useful in error analysis for example.

  So why not to correct the value stored in the cell state?

That's because in usual training scenarios, you are creating a network that predicts an output value. You can calculate the right corrections for training data, but not when predicting, because the whole point of predicting is to estimate a label that you do not already have. As such, you want to alter your function parameters, and not interim values.

  It would be very useful if we carry the cell state forward, between minibatches.

Only during training. In a prediction scenario you usually have no way of calculating the necessary gradients. What you don't want is to train a system that then requires using error values and gradients that you do not have in production.

In some scenarios, such as an online system predicting next item $x_{t+1}$ in a sequence, where you could immediately train based on error after you observed the next item and before you continued the prediction sequence for $x_{t+2}$, you could possibly use the approach. I am not sure whether it would help performance, but in principle it could. If it did help, you'd have to compare the improvement versus other simpler changes such as different hyper-parameters on a network that didn't correct internal state using gradients.

In summary, it is possible your idea would work quite well in an online system with near-immediate feedback. In that case you could think of a set of weights as being "rules to update a belief state from data", and the output of hidden layer neurons as being "a current belief state". When errors occur, it does appear to make sense to update both the rules that led to the error and the current belief that resulted from earlier faulty rules. It is perhaps worth an experiment or two. The main caveat is that the two update processes (for weights and LSTM layer state) would interact and/or adapt to each other, so it may not lead to measurably different performance than just adding more LSTM cells to the layer.


Typically you would use the best model parameters and then re-run the model with the portion of the data set aside for training to come up with a new 'best' that you can run against your test set.


Machine learning approach for predicting set members

You could train an embedding model. Each element would be projected onto a location a vector space based on its co-occurrence with other elements. Then finding similar elements could be done with a nearest neighbor search.


Data Mining - Intent matching and classification of text

The questions you have listed are more or less independent from each other.
To extract location, you should be looking into the class of solutions called Named Entity Recognition. It is widely supported, and, for example, NLTK should have location extraction support for languages but English.

The second task, the line of business, seems cryptic to me, since you are solving the task, you have complete information on the full list of those lines, so probably writing any kind of detector (custom NER would work) is a way to go, however, if all the queries are formed as LOB + location, by solving the first one, you can subtract the location out of query to get the line of business.

From the examples of queries you have provided, I don't see any way for a human being to extract any affinity, so you should investigate it to understand if those queries actually carry such kind of information.

With (1) and (2) given, the fourth task should be trivial to count.


Why does the generator produce output in a different scale than the training sample?

If the last layer of your output has no activation (e.g. a linear layer, a convolution), then its range is unbounded. Supervised training may lead the network to learn the appropriate output range. This does not seem to be the case with your GAN.

When you need the output of the generator to be constrained to a specific range of value, the safest approach is to simply force it by having the last layer use an activation function that does so by construction. As you need the output to be in $[0, 1]$, an appropriate activation function for the last layer of the generator would be a Sigmod.


Bayesian combination of multi-dimensional experts?

Just so you know, in this case, even the neural network can't help you. For NNs to work, you need to have lots of learnable patterns in the data (meaning data without uncertainity/ irreducible error). If the data itself has a lot of uncertainty in the outcome( i.e. probabilistic outcomes) itself, its very hard of NNs to optimize that. (For more on this, consider modeling following problems with NNs: XOR(with probability =1) vs XOR(with probabilities <1)).

In this case, your best bet might be to use regression. Let us know how it folds out, thanks.


Can this be a case of multi-class skewness?

Just telling the accuracy does not mean anything in classification problems. The first thing you must do is to calculate your baseline, that is, what is the percentage of your majority class? For the bar plot above it is difficult to measure, can you tell us in percentage, instead of counts? In this way we can assess better your results.

Also, have you plot a confusion matrix? In this way you can see where your model are getting more wrong and try to infer why this is happening.

And yes, since you have too many classes to predict and most of them are with low representativeness this will be difficult to overcome. Maybe you can try things such as Oversampling, Undersampling techniques considering one-vs-all approach. This is just an idea, I haven't yet encountering a problem with so many classes to predict.


Use forecast weather data or actual weather data for prediction?

You could you both forecast and actual weather, then see which provide better features for forecasting? Obviously there will be a lot of correlation between the forecast and the actual weather, but you could use feature selection to choose the most important?


training one SVM model for predicting more than one response variable

e1071 is a highly popular and therefore safe solution for this purpose.

But if data starts to get bigger I would go with classyfire. It's a quite fresh package built on e1071. It is highly optimized and have some other nice feature too. In my experience it produced similar results to e1071 but with a much faster speed as it was able to utilize all 4 core of my processor.


Does reinforcement learning require the help of other learning algorithms?

You do not need additional learning algorithms to perform reinforcement learning in simple systems where you can explore all states. For those, simple iterative Q-learning can do very well - as well as a variety of similar techniques, such as Temporal Difference, SARSA. All these can be used without neural networks, provided your problem is not too big (typically under a few million state/action pairs).

The simplest form of Q-learning just stores and updates a table of <state, action> => <estimated reward> pairs. There is no deeper statistical model inside that. Q-learning relies on estimates of reward from this table in order to take an action and then updates it with a more refined estimate after each action.

Q-learning and related techniques such as Temporal Difference are sometimes called model free. However, this does not refer to the absence of a statistical model such as a neural net. Instead, it means that you do not need to have a model of the system you are learning to optimise available, such as knowing all the probabilities of results and consequences of actions in a game. In model free RL, all learning can be done simply by experiencing the system as an agent (if you do have a model then it may still be used for simulation or planning). When considering whether or not you need a neural network, then the term tabular is used for systems that work with explicit value estimates for every possible state or state/action pair. And the term function approximation is used to describe how a neural network is used in the context of RL.

For large, complex problems, which may even have infinite possible states, it is not feasible to use tabular methods, and you need good generalised value estimates based on some function of the state. In those cases, you can use a neural network to create a function approximator, that can estimate the rewards from similar states to those already seen. The neural network replaces the function of the simple table in tabular Q-Learning. However, the neural network (or other supervised ML algorithm) does not perform the learning process by itself, you still need an "outer" RL method that explores states and actions in order to provide data for the NN to learn.


How to classify a data as not matching any label

Let's phrase it another way, decomposing into two problems:

given a sound, we want to know if it's of class A
given a sound, we want to know if it's of class B

This way of putting it is valuable to some techniques, notably "one-class classification" and "PU learning" (learning from positive and unlabeled examples). These techniques are very relevant when you want to know "does my item follow a given distribution D?", which is exactly what you're looking for.

Still, if you have (or can collect) a lot of data that is neither A nor B, you may simply label it as a garbage class C and use a common classifier - however, it may actually be harmful, because you're likely assuming wrong things about class C [Li, Liu and Ng, "Negative Training Data can be Harmful to Text Classification", 2010]. It could work and is much simpler to develop, may be good enough depending on your case.


The loop for the validation data would look very similar to your training loop, but for your validation data you only have to calculate the loss and not backpropagate the error. It would look something like this:
def train(net):
        BATCH_SIZE = 64
        EPOCHS = 10
        for epoch in range(EPOCHS):
            # training loop
            model.train()
            for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
                batch_X = train_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = train_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                net.zero_grad()
                outputs = net(batch_X)
                matches = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs, batch_y)]
                acc = matches.count(True)/len(matches)
                loss = loss_function(outputs, batch_y)
                loss.backward()
                optimizer.step()

            print(f"Epoch: {epoch}. Training acc: {round(float(acc),2)} Training loss: {round(float(loss),4)}")

            # validation loop
            model.eval()
            for i in tqdm(range(0, len(val_X), BATCH_SIZE)):
                batch_X = val_X[i:i+BATCH_SIZE].view(-1,3,224,224)
                batch_y = val_y[i:i+BATCH_SIZE]

                batch_X, batch_y = batch_X.to(device), batch_y.to(device)

                outputs = net(batch_X)
                matches = [torch.argmax(i)==torch.argmax(j) for i, j in zip(outputs, batch_y)]
                acc = matches.count(True)/len(matches)
                loss = loss_function(outputs, batch_y)

            print(f"Epoch: {epoch}. Validation acc: {round(float(acc),2)} Validation loss: {round(float(loss),4)}")


This is in fact a very well suited problem for machine learning and has been explored in the past as published here. It is often called the cocktail party problem, because of the way our human brains can focus on a very particular voice pattern despite being drowned by countless other voices all around us.

I can think of two ways by which this problem can be approached the first is to transform the spectrogram to only keep the birds chirping. The second, would be to extract the voice patterns from a sample and then substract the extracted voices from the original sample. This second approach would not remove the noise from the truck.

I would go with the former technique. I would get a few thousands clips of birds chirping. Then introduce some other sounds into it. The mixed sample would be my input $X$ and the target $Y$ would be the clean bird sample. This was you can simulate many instances. However, you will need enough bird chirping data to cover the wide distribution which exists for birds making noise.

Q1

For this to work well, you will need A LOT of data. Yes, you should definitely try to get as many samples from wherever you can. You can then randomly introduce these human voices with the bird clips in order to get the dirty signals, then use these to train your model.

Q2

You most likely will not achieve 100% accuracy with existing machine learning methods. However, if you get close to 90% that is already a miracle and will truly be great. If you are using spectrograms I would first try convolutional neural network (CNN). These have been shown to produce very good results here. The same paper then proposes an autoencoder to clean up the spectrograms. I think this has a high probability of working, but will introduce smoothing to the spectrogram which may distort the birds chirping. Lastly, I would suggest using a generative adversarial network (GAN).

Q3

I would look into the cocktail party problem. You can also get some insight as to what techniques to use by looking at the extensive work that has been done on spectrograms in MRI spectroscopy (MRSI).

Q4

Like I said above I would suggest going from a $n \times n$ sample containing a mixture of sounds to a clean $n \times n$ sample with only the bird chirping. I think this strategy is the one where you will find the most existing literature and many groups have shown it to work.


What are graph Embeddings ?
"Graph Embeddings" is a hot area today in machine learning. It basically means finding "latent vector representation" of graphs which captures the topology (in very basic sense) of the graph. We can make this "vector representation" rich by also considering the vertex-vertex relationships, edge-information etc. There are roughly two levels of embeddings in the graph (of-course we can anytime define more levels by logically dividing the whole graph into subgraphs of various sizes):

Vertex Embeddings - Here you find latent vector representation of every vertex in the given graph. You can then compare the different vertices by plotting these vectors in the space and interestingly "similar" vertices are plotted closer to each other than the ones which are dissimilar or less related. This is the same work that is done in "DeepWalk" by Perozzi.
Graph Embeddings - Here you find the latent vector representation of the whole graph itself. For example, you have a group of chemical compounds for which you want to check which compounds are similar to each other, how many type of compounds are there in the group (clusters) etc. You can use these vectors and plot them in space and find all the above information. This is the work that is done in "Deep Graph Kernels" by Yanardag.

Applications -
By looking carefully, embeddings are "latent" representations  which means if a graph has a |V| * |V| adjacency matrix where |V| = 1M, its hard to use or process a 1M * 1M numbers in an algorithm. So, latent embedding of dimension 'd', where d << |V|, would make the adjacency matrix |V| * d and relatively easier to use. Another application could be - Consider a simple scenario where we want to recommend products to the people who have similar interests in a social network. By getting vertex embeddings (here it means vector representation of each person), we can find the similar ones by plotting these vectors and this makes recommendation easy. These are some applications and there are others. You can refer to a nice survey paper - Graph Embedding Techniques, a Survey.

Where from it all came ? There has been a lot of works in this area and almost all comes from the groundbreaking research in natural language processing field - "Word2Vec" by Mikolov. If you want to get started with the research on graph embeddings, I would recommend to first understand how Word2Vec works. You can find nice explanations - Word2Vec parameter learning explained and Stanford Lecture. Then you can jump to the papers that you listed. Those works can be categorized as:

Works based on "Vertex Embeddings": - DeepWalk, Node2Vec, LINE.
Works based on "Graph Embeddings": - Deep Graph Kernels, Subgraph2Vec.


You could try looking just at the IDF scores. Things like names/entities will score high, this may be desirable or undesirable depending on what you want.


Why is the decoder not a part of BERT architecture?

The need for an encoder depends on what your predictions are conditioned on, e.g.:

In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder.
In Neural Machine Translation (NMT) models, each token of the translation is predicted conditioning on the previous tokens and the source sentence. The previous tokens are received by the decoder, but the source sentence is processed by a dedicated encoder. Note that this is not necessarily this way, as there are some decoder-only NMT architectures, like this one.
In masked LMs, like BERT, each masked token prediction is conditioned on the rest of the tokens in the sentence. These are received in the encoder, therefore you don't need an decoder. This, again, is not a strict requirement, as there are other masked LM architectures, like MASS that are encoder-decoder.

In order to make predictions, BERT needs some tokens to be masked (i.e. replaced with a special [MASK] token. The output is generated non-autoregressively (every token at the output is computed at the same time, without any self-attention mask), conditioning on the non-masked tokens, which are present in the same input sequence as the masked tokens.


Using iGraph to build a Distribution Model

NetworkX does have the ability to obtain shortest paths between points for you. As for how you can encode business rules, NetworkX does allow you to manipulate graph objects easily so you could code up a function containing your rules and apply it to to the graph object.

To model the distribution shift you could give weightings to each shop that would determine how much of the monthly visits they acquire when one of their neighbors close.


What measures can I use to find correlation between categorical features and binary label?

Checking if two categorical variables are independent can be done with the Chi-Squared test of independence where we perform a hypothesis test for that. Let's say A & B are two categorical variables then our hypotheses are:

H0: A and B are independent

HA: A and B are not independent

We create a Contingency table that counts for the combination of outcomes from two variables, if the null hypothesis(H0) is correct then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the actual values are.

For Example
Suppose we have two variables in the dataset

Obesity: Obese, Not obese
Marital Status: Married, Cohabiting, Dating

We observe the following data:
Oij for a cell (i,j) is the observed count in given data

             |    dating   |   married    |  cohabiting  |  Total       |
  -----------|------------:|:------------:|:------------:|:------------:|
  Obese      |      81     |     147      |   103        |   331        |
  Not obese  |      359    |     277      |   326        |   962        |
  Total      |      440    |     424      |   429        |   1293       |

Expected Counts Calculation i.e. Expected counts if H0 was true.

Eij for a cell (i,j) as
Eij=row j total * column i total / table total

             |    dating   |   married    |  cohabiting  |
  -----------|------------:|:------------:|:------------:|
  Obese      |      113    |     109      |   110        |
  Not obese  |      327    |     316      |   319        |

X2-statistics Calculation
Statistics

Assuming independence, we would expect that the values in the cells are distributed uniformly with small deviations because of sampling variability
so we calculate the expected values under H0 and check how far the observed values are from them.

We use the standardized squared difference for that and calculate Chi-square statistics that under H0 follows χ2 distribution with df=(n−1)⋅(m−1) where n & m are the number of categories in the first & second variable respectively.

\begin{equation}
\chi^2 = \sum_ i \sum_ j \frac{ ({O_{ij} - E_{ij}})^2}{ E_{ij}}
\end{equation}

χ2 value comes out to be 30.829

We can use R to find the p-value

tbl = matrix(data=c(81, 147, 103, 359, 277, 326), nrow=2, ncol=3, byrow=T)
dimnames(tbl) = list(Obesity=c('Obese', 'Not obese'), Marital_status=c('Dating', 'Married','Cohabiting'))
chi_res = chisq.test(tbl)
chi_res
        Pearson's Chi-squared test
data:  tbl
X-squared = 30.829, df = 2, p-value = 2.021e-07

Since p-value < 0.05 we reject the null hypothesis, we can conclude that obesity and marital status are dependent.

There also exists a Crammer's V that is a measure of correlation that follows from this test.

Putting values in the formula, R code

sqrt(chisq.test(tbl)$statistic / (sum(tbl) * min(dim(tbl) - 1 )))

0.1544

So we can say there is a weak positive correlation between obesity and marital status.

I hope I am clear with the explanation.


Query similarity: how much data is used in practice?

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.

Here's how to get more than 10 results per query: https://support.google.com/customsearch/answer/1361951?hl=en

  Google Custom Search and Google Site Search return up to 10 results per query. If you want to display more than 10 results to the user, you can issue multiple requests (using the start=0, start=11 ... parameters) and display the results on a single page. In this case, Google will consider each request as a separate query, and if you are using Google Site Search, each query will count towards your limit.

There are other search engine APIs as well (e.g., Bing)


The answer to your questions depend a lot on the nature of the data represented in the time series. You should ask yourself some questions to better understand what might or might not work, e.g.:

Are the time sequences perfectly aligned?
Are two slightly shifted time series considered similar or not?
Are two time series with the same shape but different scale considered similar?

Normally, the answers to those questions are that series are not perfectly aligned and that variations in scale are also fine as long as the shape is similar. For these scenarios, the classical measure is Dynamic Time Warping (DTW). There are lower bounds for DTW that are very efficient computationally. The research of Professor Keogh might be interesting if you need theoretical foundation for it.

Also, normally euclidean distance and Manhattan distance are not very appropriate for time series due to their sensitivity to signal transformations (e.g. shifts), but actually they are often used in practice.


Detecting cats visually by means of anomaly detection

You could simplify your problem significantly by using a motion/change detection approach. For example, you could compare each image/frame with one from an early time (e.g., a minute earlier), then only consider pixels that have changed since the earlier time. You could then extract the rectangular region of change and use that as the basis for your classification or anomaly detection.

Taking this type of approach can significantly simplify your classifier and reduce your false target rate because you can ignore anything that is not roughly the size of a cat (e.g., a person or bird). You would then use the extracted change regions that were not filtered out to form the training set for your classifier (or anomaly detector).

Just be sure to get your false target rate sufficiently low before mounting a laser turret to your feline intrusion detection system.


BPE vs WordPiece Tokenization - when to use / which?

(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very fast C++ implementation of BPE. You can find fast Rust implementations of both in Hugginface's tokenizers.


Is there a cost associated with converting Koalas dataframe to Spark dataframe?

As you said, since the Koalas is aiming for processing the big data, there is no such overhead like collecting data into a single partition when ks.DataFrame(df).

However, the overheads are occurred when creating a default columns for creating the _InternalFrame which internally manages the metadata between pandas and PySpark.

Koalas is internally using immutable frame named _InternalFrame, so you can refer the /koalas/databricks/koalas/internal.py if you want to more detail.

Here is a short code examples from internal.py that creating the default index if given Spark DataFrame has no index information.

https://github.com/databricks/koalas/blob/a42af49c55c3b4cc39c62463c0bed186e7ff9f08/databricks/koalas/internal.py#L478-L491

        if index_map is None:
            assert not any(SPARK_INDEX_NAME_PATTERN.match(name) for name in spark_frame.columns), (
                "Index columns should not appear in columns of the Spark DataFrame. Avoid "
                "index column names [%s]." % SPARK_INDEX_NAME_PATTERN
            )

            # Create default index.
            spark_frame = _InternalFrame.attach_default_index(spark_frame)
            index_map = OrderedDict({SPARK_DEFAULT_INDEX_NAME: None})

        if NATURAL_ORDER_COLUMN_NAME not in spark_frame.columns:
            spark_frame = spark_frame.withColumn(
                NATURAL_ORDER_COLUMN_NAME, F.monotonically_increasing_id()
            )


The minimum number of training examples is what you have up there:

$$4(nm + n^2)$$

For more information refer to this article:
Refer to this link if you needed some visual help:
Number of parameters in an LSTM model

The number of units in each layer of the stack can vary. For example in translate.py from Tensorflow it can be configured to 1024, 512 or virtually any number. The best range can be found via cross validation. But I have seen both 1000 and 500 number of units in each layer of the stack. I personally have tested with smaller numbers as well.


My models performs better with the arbitrary random feature. How can I interpret this?

It's a good sanity check, but the fact it fails means there must be a mistake somewhere:

Hypothesis 1: mistake in the interpretation of the results. Is the performance improvement significant? If not, it might simply be due to chance. That would mean that none of the models actually uses the random feature, one happens to be slightly better by chance. However the chances that this would happen with 6 distinct classifiers are very low ($1/2^6$ to be precise)
Hypothesis 2: mistake in the generation of the the random feature. If it actually contributes to the prediction, then there must be a regular pattern so it's not truly random. Manually inspect the models (especially the decision tree one, it's the easiest to interpret) to see what happens. Then try to reproduce the result with a different random sequence and observe the model again: if the model uses the random feature in the same way then it's not really random.
Hypothesis 3: mistake in the split between training and test set. We always underestimate how easy it is to make a stupid mistake, so my money is on this one ;) More seriously, the fact that the performance is better with the random feature points to this direction: assuming it's truly random and it has mostly distinct values, an overfit model could use it as an id for an instance that was seen in the training set. Try re-sampling and/or cross-validation.


Interpreting confusion matrix and validation results in convolutional networks

These are my recommendations:

Try to train your model more, with different learning rates. Your current learning rate is so much small. At first steps put it to a big value then reduce it smoothly.
Try to change the architecture. If your task does not have the same scale as image-net surely you would need a network with simpler network architecture, consequently the number of parameters diminishes.
Changing the distribution of data is dangerous, but you can augment your entire data-set by some extent to let your network see different and more examples.
For those classes that have not been learnt you can set the class-weight to a bigger number to construct a cost function which cares more about those classes.
Because your data-set is not balanced, use other metrics to evaluate your model, like F1 score.


MLPClassifier threshold factor to eliminate test samples that are not in match with train data

SGDClassifier has desicion_function which tells the distance to the hyperplane, where the values are compared to.

This value could imply too big and too low values.


Why we use information gain over accuracy as splitting criterion in decision tree?

Decision trees are generally prone to over-fitting and accuracy doesn't generalize well to unseen data. One advantage of information gain is that -- due to the factor $-p*log(p)$ in the entropy definition -- leafs with a small number of instances are assigned less weight ($lim_{p \rightarrow 0^{+} } p*log(p) = 0$) and it favors dividing data into bigger but homogeneous groups. This approach is usually more stable and also chooses the most impactful features close to the root of the tree.

EDIT: Accuracy is usually problematic with unbalanced data. Consider this toy example:

Weather Wind    Outcome
Sunny   Weak    YES
Sunny   Weak    YES
Rainy   Weak    YES
Cloudy  Medium  YES
Rainy   Medium  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO
Rainy   Strong  NO

Weather and Wind both produce only one incorrect label hence have the same accuracy of 16/17. However, given this data, we would assume that weak winds (75% YES) are more predictive for a positive outcome than sunny weather (50% YES). That is, wind teaches us more about both outcomes. Since there are only few data points for positive outcomes we favor wind over weather, because wind is more predictive on the smaller label set which we would hope to give us a rule that is more robust to new data.

The entropy of the outcome is  $ -4/17*log_2(4/17)-14/17*log_2(14/17)) =0.72$. The entropy for weather and outcome is $14/17*(-1/14*log_2(1/14)-13/14*log_2(13/14)) = 0.31$ which leads to an information gain of $0.41$. Similarly, wind gives a higher information gain of $0.6$.


activation function for binarized neural networks

Create a custom function
As explained in Keras doc for custom Actiation function
Creating custom activations
You can also use a TensorFlow callable as an activation (in this case it should take a tensor and return a tensor of the same shape and type):
model.add(layers.Dense(64, activation=tf.nn.tanh))

def custom_bin(x):
    return tf.where(x >= 0, 1.0 , -1)

import tensorflow as tf
from tensorflow import keras
model = keras.Sequential()

model.add(keras.layers.Dense( 3, activation="linear",input_shape=(5,)))
model.add(keras.layers.Dense( 3, activation="linear"))
model.add(keras.layers.Dense( 4, activation=custom_bin))
model.add(keras.layers.Dense( 2, activation="linear"))

model.summary()

A toy NN to check the Activation output
A simple neural net with all weights = 1 and bias = 0. So, it will keep the output sign same as that of the input i.e.(+Ve, 0, -Ve).Then apply the respective Activation function in the output layer. Then predict using the 3 use-cases
def custom_bin(x):
    return tf.where(x >= 0, 1.0 , -1)

import tensorflow as tf
from tensorflow import keras

#Case I - tf.math.sign
initializer = tf.keras.initializers.Ones()
model = keras.Sequential()
model.add(keras.layers.Dense( 2, activation="linear",input_shape=(2,),kernel_initializer=initializer))
model.add(keras.layers.Dense( 1, activation=tf.math.sign,kernel_initializer=initializer))
print(model.predict([[-5,-5],[0,0],[5, 5]]))

#Case II - custom if-then-else
initializer = tf.keras.initializers.Ones()
model = keras.Sequential()
model.add(keras.layers.Dense( 2, activation="linear",input_shape=(2,),kernel_initializer=initializer))
model.add(keras.layers.Dense( 1, activation=custom_bin, kernel_initializer=initializer))
print(model.predict([[-5,-5],[0,0],[5, 5]]))

Output
[[-1.] [ 0.] [ 1.]]
[[-1.] [ 1.] [ 1.]]


If a neural network is a universal function approximator, is the irreducible error 0?

The answer to the title question is "no"; the contexts are a little different.

The bias-variance tradeoff holds in the context that $y=f(X)+\varepsilon$, with $\varepsilon$ having mean 0 and variance $\sigma^2$; and $\sigma^2$ turns out to be the irreducible error, so named because it is a lower bound on the expected squared-error.  So to answer your last question, the irreducible error can be zero, if there is no noise in the data.

That neural networks are universal function approximators refers to the actual output function itself (many linear functions with, say, ReLU activators), and not to the fitting procedure.  Notice too that the universality has restrictions: the approximation is only guaranteed on a compact domain, and the error in approximation can be pushed down by using enough neurons in the hidden layer.

So what the universality really implies, w.r.t. the bias-variance tradeoff, is that it is possible to push the bias to zero (or as close as desired, by using enough neurons).  Of course, this will generally result in a badly overfit model.

wikipedia - Bias-variance decomposition


What is the most used format to save data with type information

Parquet and Avro both support data types (strings, integers, floats, etc). These are the primary file types used for "big data" projects, although your data doesn't have to be big. Apache Spark is able to read both with ease. Having said that, I'm a big fan of keeping data in a SQL database (e.g., MySQL or Postgres) because that is what they are built for. If you can't re-use the database you're pulling from, could you make your own database locally or on a separate server? I would try using a relational database until your data exceeds 50 GB (an arbitrarily "somewhat large" size), and then I would use Avro or Parquet.


Are batch iteration and epochs different in reinforcement learning compared to supervised learning?

Your initial statements are correct.

Epoch is one-single pass over the full training set.

Most people use the word Batch as the number of samples used for one update of the weights. (The back-propagation process calculates the gradients for every single sample in the batch, but the weights update is performed a single time for the mean of the gradients over the batch).

However, originally the word batch was used to mean a training process where the batch size is equal to the overall number of samples in the training set (meaning that the weights update is performed only once per epoch). What we now refer to as "batch", was originally named "mini-batch".

Over the years, the word batch became synonymous with mini-batch (batch training is very rare these days). I think that this is the source to your confusion.


NER with LSTM - How to recognize person names that are not part of the vocabulary?

With word-level vocabularies, unknown words are normally encoded with a special token <unk>. If in the training data there were enough examples of person names, then the bidirectional LSTM may have learned to identify the person name from the context, not just the word itself. In that case, and provided that in the specific input sentence there is enough context, the model may be able to identify it. If the model were a normal LSTM instead of a bidirectional one, it would be more difficult for the model to identify it, as it would only be able to use the context to the left of the unknown word.
If the training data did not contain such examples, the it won't be able to identify the person from the context.
One option to improve the handing of this problem  would be to force this kind of examples in the training data, by replacing person names with unknown words with certain probability.
Another option would be to use a subword-level vocabulary, like a byte-pair encoding (BPE) one, eliminating altogether the out-of-vocabulary (OOV) word problem.


Dose finding slope/intercept using the formula of m,b gives best fit line always In linear regression?

The formulae you mentioned gives the coefficients of the line of best fit.The values are derived using the least squares method, where the goal is to minimize the sum of squared errors. Following is the derivation for the values of m and b.
Let the line of best fit be $$\hat{y} = m*x + b$$ We then  try to find the coefficients m and b which minimize the sum of squared errors between the actual value y and the observed value $\hat{y}$. \begin{align}
SSE &= \sum_{i=1}^{n}(y_{i}-\hat{y_{i}})^2 \\
&=\sum_{i=1}^{n}(y_{i}-m*x_{i}-b)^2
\end{align}
Taking the first derivative of SSE with respect to c and equating to zero.
\begin{align}
\frac{\partial SSE}{\partial b} &= \sum_{i=1}^{n}-2*(y_{i}-m*x_{i}-b)\\
0 &= \sum_{i=1}^{n}-2*(y_{i}-m*x_{i}-b)
\end{align}
Therefore we get c as $$ b = \bar{y} - m*\bar{x}$$ Similarly in order to find m we take the partial derivative of SSE with respect to m and equate it to zero.
\begin{align}
\frac{\partial SSE}{\partial m} &= \sum_{i=1}^{n}-2x_{i}*(y_{i}-m*x_{i}-b)\\
0 &= \sum_{i=1}^{n}-2x_{i}*(y_{i}-m*x_{i}-b)\\
0 &= \sum_{i=1}^{n}x_{i}*(y_{i}-m*x_{i}-b)\\
0 &= \sum_{i=1}^{n}x_{i}*y_{i} - \sum_{i=1}^{n}m*x_{i}^2 - \sum_{i=1}^{n}b*x_{i}
\end{align}
Substituting b and solving for m we get $$m = \frac{n\sum xy - \sum x\sum y}{n\sum x^2 - (\sum x)^2}$$


Mathematical formulation of Support Vector Machines?

Your understandings are right.

  deriving the margin to be $\frac{2}{|w|}$

we know that $w \cdot x +b = 1$

If we move from point z  in  $w \cdot x +b = 1$ to  the $w \cdot x +b = 0$ we land in a point $\lambda$. This line that we have passed or this margin between the two lines $w \cdot x +b = 1$ and $w \cdot x +b = 0$ is the margin between them which we call $\gamma$

For calculating the margin, we know that we have moved from z, in  opposite direction of w to point $\lambda$. Hence this margin $\gamma$ would be equal to $z - margin \cdot \frac{w}{|w|} = z - \gamma \cdot \frac{w}{|w|} =$ (we have moved in the opposite direction of w, we just want the direction so we normalize w to be a unit vector $\frac{w}{|w|}$)

Since this $\lambda$ point lies in the decision boundary we know that it should suit in  line $w \cdot x + b = 0$
Hence we set is in this line in place of x:

$$w \cdot x + b = 0$$
$$w \cdot (z - \gamma \cdot \frac{w}{|w|}) + b = 0$$
$$w \cdot z + b  - w \cdot \gamma \cdot \frac{w}{|w|}) = 0$$
$$w \cdot z + b  = w \cdot \gamma \cdot \frac{w}{|w|}$$
we know that $w \cdot z +b = 1$ (z is the point on $w \cdot x +b = 1)$
$$1  = w \cdot \gamma \cdot \frac{w}{|w|}$$
$$\gamma= \frac{1}{w} \cdot \frac{|w|}{w} $$
we also know that $w \cdot w = |w|^2$, hence:
$$\gamma= \frac{1}{|w|}$$
Why is in your formula 2 instead of 1? because I have calculated the margin between the middle line and the upper, not the whole margin.

  How can $y_i(w^Tx+b)\ge1\;\;\forall\;x_i$?

We want to classify the points in the +1 part as +1 and the points in the -1 part as -1, since $(w^Tx_i+b)$ is the predicted value and $y_i$ is the actual value for each point, if it is classified correctly, then the predicted and actual values should be same so their production $y_i(w^Tx+b)$ should be positive (the term >= 0 is substituded by >= 1 because it is a stronger condition)

The transpose is in order to be able to calculate the dot product. I just wanted to show the logic of dot product hence, didn't write transpose

For calculating the total distance between lines $w \cdot x + b = -1$ and $w \cdot x + b = 1$:

Either you can multiply the calculated margin by 2 Or if you want to directly find it, you can consider a point $\alpha$ in line $w \cdot x + b = -1$. then we know that the distance between these two lines is twice the value of $\gamma$, hence if we want to move from the point z to $\alpha$, the total margin (passed length) would be:
$$z - 2 \cdot \gamma \cdot \frac{w}{|w|}$$ then we can calculate the margin from here.

derived from ML course of UCSD by Prof. Sanjoy Dasgupta


Trained CNN individually on multiple images to classify them, how can I now classify a related "set" of these images that correspond to one object?

As far as I understand your problem is related to ensemble learning: You ensemble is a set of models trained on detecting the class of an object on images with the same angle on the object, and now you want to classify an object and choose the model voting of the model trained on the correct angle?
In that case, I would choose the prediction of the model which has the most unambiguous output before outputting a binary label.
So it's probably related to these topics:

Majority Voting in Ensemble Learning
Ensemble Learning in Machine Learning | Getting Started


Algorithms to generate a rating system based on history

You need to define what speeding means and how to rate the drivers for amount of such. As you note that you have speed in your dataset this should be rather trivial. Not sure at which stage you want to involve machine learning? Seems like you don't need any.
Machine learning is not the same thing as learning about cars and their drivers.


Partial autocorrelation and Autocorrelation difference

Good evening,

Let's begin with a simple example: you have a time series process, for example some process with correlation of up to lag 4.

An order (p) process is time series process where current value depends on previous p values.
You can think for examples of weekly sales in a supermarket, and your AR(4) model in this case shows that your current week sales depend on the previous 4 weeks of sales (on week 1, week 2, week 3, and week 4).

Now, let's get into ACF and PACF.

You have your AR(4) model, which basically tells you that points are correlated up to lag 4, meaning that your correlation between wk 1 and wk 5 is the same as between wk2 and wk 6 (note that the "distance" between them is 4 lags). Similarly, for any number of lags <4 this rule works the same way: corr wk 1 and wk 2 is same as wk 2 and wk 3 and so on. Assume that correlation at lag 1 (wk 1 and wk 2) is stronger than lag 2 (wk1 and wk 3), this means that as number of lags increases, the autocorrelation will also decrease. At each further lag the information from previous lags carries over.
Here is a visualization.

PACF is a completely different concept. What it primarily focuses on is finding out the correlation between two points at a particular lag. In your case, say you want to find the "independent" correlation between wk4 and wk3, this is exactly what PACF will show you.
Here is a visualization.

All together:
ACF data for wk4 will include all the information up to wk3 (wk1, wk2, wk3).
PACF data for wk4 will include "independent" (partial) correlations between wk3 and wk4 only (meaning it will by correlation between these points which wasn't explained by their mutual correlations.

To your point about AR and MA processes, it's the other way around. The lag at which ACF becomes very small is order of MA process. And the lag at which PACF becomes very small is order of AR process.

Hope this helps.


Best metric in imbalanced classification for multi-label classification

Maybe you should instead try to use some techinique to rebalance your dataset and then use a classical metric for classification problems.
Example of tchniques for rebalancing the dataset for binary classification is given in the following post:
https://stats.stackexchange.com/questions/152823/how-to-balance-my-dataset
If you still want to work on the unbalanced dataset in the following paper some s are suggested:
https://eva.fing.edu.uy/pluginfile.php/69453/mod_resource/content/1/7633-10048-1-PB.pdf


Which machine learning algorithm to choose?

If you want to use an unsupervised method i.e if your data is not labelled with classes, then something like k-means clustering may be your best bet to find patterns in the data.

Alternatively, if you want to do anomaly detection, 2 possible options are

Assuming your data is normally distributed, you could fit a Gaussian to your data and calculate the Probability Density Function (PDF). Once you have the PDF, you can set a threshold probability, below which a data point could be classified as an anomaly
If you have enough data, use a Variational Autoencoder neural network. Very roughly speaking, you train this on all the data you deem to be “normal” (the neural network learns how to reconstruct the input data in the output), and then when anomalous cases are passed to the network, it can’t reconstruct it. If the network can’t reconstruct it accurately, the data is an anomaly.


how to know what my data are uninformative and that machine learning will not work with it?

You may have heard

Correlation does not imply causation

but there is one more thats even more important

Without correlation there CAN BE causation. One example

SO just because they do not correlate does not mean that they are uninformative (think also about feature interactions)

I think you are looking for some test/theory that can give you some statement before hand. Without some HARD asssumptions thats impossible. Only way is to exhaustively try pretty different approaches and to reach desired metric scores.


I found the answer I was looking for in this Google paper:

However, in the case of ordered-classes (e.g.
aesthetic and quality estimation), cross-entropy loss lacks the
inter-class relationships between score buckets. One might
argue that ordered-classes can be represented by a real number, and consequently, can be learned through a regression
framework. Yet, it has been shown that for ordered classes,
the classification frameworks can outperform regression models [21], [31]. Hou et al. [21] show that training on datasets
with intrinsic ordering between classes can benefit from EMDbased losses. These loss functions penalize mis-classifications according to class distances


Can training label confidence be used to improve prediction accuracy?

Yes, it's possible to use this confidence data.  However, I wouldn't recommend the approach you mention.  Instead, let me suggest a different approach.  Actually, I'll suggest two.  The first one is conceptually clean; the second is probably easier to implement; and they'll probably be approximately equivalent in practice.

Adjust the loss function

You can adjust the loss function to reflect the confidence scores you have on the training data.

In particular, if you are using the cross-entropy loss, there's a particularly clean way to do this.  Let me explain some background on the cross-entropy loss, then explain how.  We think of the label on the training instance as a probability distribution on labels.  In binary classification, such a distribution can be represented as a vector $(p_0,p_1)$ where $p_0$ represents the probability that the label is 0 and $p_1$ the probability that the label is 1.  Normally, we are given "hard labels": if we know that the correct label on instance $x$ is 0, then that corresponds to the probability distribution $(1,0)$; if the correct label is 1, that's the distribution $(0,1)$.  The cross-entropy loss then compares the prediction from the classifier to this distribution.

The nice thing about the cross-entropy loss is that it generates readily to comparing any two distributions.  Thus, if you have a confidence $0.8$ that the correct label for instance $x$ is 0, then that corresponds to a probability distribution $(0.8,0.2)$.  Now, you can compute the cross-entropy of the classifier's prediction with respect to the distribution $(0.8,0.2)$, and that is the contribution to the loss from training instance $x$.  Sum this over all instances in the training set, and you get an adjusted loss function.

Now you can train a classifier by minimizing this adjusted loss function, and that will directly incorporate all of the information in your confidence scores.

Use weights

Alternatively, you can use weights to reflect the confidence information.

Some classifiers allow you to specify a weight for each instance in the training set.  The idea is that a misprediction for a particular instance is penalized proportionality to its weight, so instances with a high weight are more important to get right and instances with a low weight are less important.  Or, equivalently, the training procedure tries harder to avoid errors on instances with a high weight.

You can use weights to reflect confidence information.  Suppose you have an instanced $x$ in the training set that you think should have label 0, with confidence $0.8$.  You would add one copy of $(x,0)$ to the training set with weight $0.8$ (i.e., instance is $x$ and label is 0), and add one copy of $(x,1)$ to the training set with weight $0.2$ (i.e., instance is $x$ and label is 1).  Build up the training set in this way.  This doubles the size of your training set.  Now train a classifier, using these weights.

For classifiers that support weights, this should be easy to implement.

One can also show that it is effective and reasonable.  For instance, when using the cross-entropy loss to train a classifier, this approach using weights is equivalent to adjusting the loss function as highlighted above.  So, in that particular context, the two approaches are actually equivalent.