The problem is in the following line:

q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j]);

(h-y) has shape (20, 1), a[:,j] has shape (20), multiplying them results in a shape (20, 20), which is wrong.

Try a[:,j:j+1] instead of a[:,j] and it'll start working:

q[j][0]=q[j][0]-alpha*np.sum((h-y)*a[:,j:j+1]);

This gives the following plot:

You could adjust the shapes of your variables a bit to make it easier:

x=data[:,:1]
y=data[:,1]
a=np.hstack([np.ones((len(x), 1)), x])
q=np.ones(a.shape[1])
alpha=0.003

for i in range(10000):
    h=1/(1+np.exp(-np.dot(a,q)))
    for j in range(len(q)):
        q[j]=q[j]-alpha*np.sum((h-y)*a[:,j])

plt.plot(x,y,'x',x,h)
plt.show()

Also note that in Python you don't need ; at the end of each line.


Replace value of a column if the value of another column is a duplicate

A cleaner way would be:
df["date"] = df.groupby("name").date.transform("min")

Outputs:


Flair Embeddings - Significance of Backwards vs Forwards?

Upon further research I found that during training forward language models try to predict the next word in a sequence. Backwards language models on the other hand start at the end of a sequence and attempt to predict the proceeding word. It seems that by stacking both forward and backwards models produced from the same data-set you get better results than using a forward or backwards model alone.


Does the phenomenon of over-fitting of data varies with training algorithms?

A model is over-fitting if it makes good predictions on a test set but bad predictions on new data. This is generally a good indication that the used model is too complex.

The complexity of the model is often quantified as the number of free parameters. These are the parameters that need to be set in order to fit the data. More parameters allow more flexibility in what can be expressed but also increase the change of over-fitting.

The type of model also restricts what functions can be learned. For example, linear models can only learn linear functions (really).

The simple answer to your question is yes. For example, a SVM with little parameters might fit data well while a ANN with many parameters might over-fit on the same data.

A fair comparison however compares models with the same number of parameters. But the answer remains yes. One model might be better suited than another to fit the intrinsic structure of the data. For example, on the same data, a decision tree with 100 nodes might over-fit, a linear model might under-fit and a ANN might work perfectly. It all depends on the underlying structure that you want to model.


How about this video, which explains backpropagation intuitively?

https://www.youtube.com/watch?v=Ilg3gGewQ5U

Title: "What is backpropagation really doing? | Deep learning, chapter 3"

Author: "3Blue1Brown"


We can view nearest neighbor as a voting process where we consult our $k$ nearest neighbor.

We give the $i$-th data point a voting weight $w_i$.

In your example, each data point in class $A$ has weight $\frac1{0.95}$ and each data point in class $B$ has weight $\frac1{0.05}$. There are $4$ votes from class $A$ and $3$ votes from class $B$. We give class $A$ a score of $\frac{4}{0.95}\approx 4.21$ and class $B$ a score of $\frac{3}{0.05}=60$. Class $B$ has a higher score, hence we assign it to class $B$.


As oW_ mentioned, this can be done with the Elo rating system.

More specifically, given a record of wins and losses for some player pool, you can fit ratings for each player such the expected result of player $A$ rated $r_A$ vs player $B$ rated $r_B$ is given by:

$$
E_{A\ vs\ B} = \frac{1}{1 + 10^{(r_B-r_A)/400}}
$$

When the only possible outcomes are win or lose then this is also the probability of $A$ beating $B$.

If we define the quantity $\gamma_i = 10^{r_i/400}$ then the above can be expressed as:

$$
E_{A\ vs\ B} = \frac{\gamma_A}{\gamma_A+\gamma_B}
$$

Furthermore, Elo ratings generalize to multiplayer competitions, such that the probability of player $A$ beating players $B$, $C$, and $D$, for example, is given by:

$$
E_{A\ vs\ B,C,D} = \frac{\gamma_A}{\gamma_A+\gamma_B+\gamma_C+\gamma_D}
$$

The simplest method for computing ratings of a pool of (previously unrated) players given their game histories is to give them all the same initial rating, say $0$. The number doesn't matter because the relative rating differences (which are used to estimate win probabilities) will be the same. Next, for each result, $S$, of player $i$ vs player $j$ in the data set make the following update:

$$
r_i' \leftarrow r_i + K \times (S - E_{i,j})
$$

where $S = 1$ if $i$ beat $j$, otherwise $0$, and $K$ is a factor chosen to determine how quickly ratings change. For simplicity you can use $K=32$.

Remember to perform the update in both directions, so that if $i$ beats $j$ then $i$'s rating increases while $j$'s decreases. Use the pre-updated rating for player $i$ when computing player $j$'s rating change, so that the result is symmetric. It's also typical to use the pre-updated ratings for all rating update calculations in a single tournament. So you would keep track of all deltas for each player and apply them at the end.

To handle multi-player tournaments, simply log one win for the winner against every other player. In your case you only have the winner, so this is the best you can do. (If you had a complete ranking you could try Tom Kerrigan's Multiplayer Elo method or Microsoft's TrueSkill.)

You also need to divide $K$ by the number of players minus $1$ in your update rule:

$$
r_i' \leftarrow r_i + \frac{K}{n-1} \times (S - E_{i,j})
$$

Here is a Python library that performs the calculation I have described.

An alternative approach is the TrueSkill rating system, which handles multiplayer games nicely. It expects a ranking of everyone in the tournament, however you could try it with the winner ranked first and everyone else tied for second. Here is a Python TrueSkill library.


Get most likely topic per document in pandas dataframe using gensim

Timing your get_max function on the largest list you provided gives a time of a little over 9 us which seems quite efficient. I have, however, written another function that does the same but takes a little more than 3 us, so almost 3x as fast. For smaller lists this speedup will become even greater (~10x speedup for lists of size 1 or 2).
doc = [(0, 0.010002977),
 (1, 0.010002977),
 (2, 0.010002977),
 (3, 0.010002979),
 (4, 0.8099435),
 (5, 0.010002977),
 (6, 0.010002977),
 (7, 0.010002977),
 (8, 0.010002977),
 (9, 0.010002977),
 (10, 0.010002977),
 (11, 0.010002977),
 (12, 0.010002977),
 (13, 0.010002977),
 (14, 0.010002977),
 (15, 0.010002977),
 (16, 0.010002977),
 (17, 0.010002977),
 (18, 0.010002977),
 (19, 0.010002977)]

def get_max(doc):
    idx,l = zip(*doc)
    return idx[np.argmax(l)]

def get_max_new(doc):
    return sorted(doc, key=lambda x: x[1], reverse=True)[0][0]

%timeit get_max(doc) # 9.14 us

%timeit get_max_new(doc) # 3.53 us

There may be even more speedups that can be achieved, however that is more difficult for me to test at the moment.


How to append new numerical features to an embedding from word2vec, such that KNN on embeddings is not biased for one feature?

Since you have mixed data, use one-hot encoding for the categorical data so that they become binary data. For the numerical data, normalize them so that their ranges are within [0,1].

Next, you'll have to choose an appropriate distance measure. See the answer here:
https://stats.stackexchange.com/questions/218092/how-to-calculate-the-distance-in-knn-for-mixed-data-types


Please look into the Mask R-CNN (Faster CNN)
https://github.com/matterport/Mask_RCNN/


How is the cross-product transformation defined for binary features?

Let's do this in the opposite order of how you asked.  i.e. first:

How can I think of the cross-product transformation in general?

For me a cross-product comes from linear algebra, and it is a different transformation.  Therefore let's start by not confusing both.  The cross-product in the equation above is a transformation alright but is not related to what most of us call a cross product.

With that out of the way let's look at the formula.  Since both the index on the product and components of $\vec{x}$ and $\vec{c}$ are the same (they're all $i$), we can be rather confident that $d$ is the number of dimensions/features we are working with.  Therefore to make things a little easier to see let's work with $d = 4$:

$$
\phi_{k}(\vec{x})=\prod_{i=1}^{4} x_{i}^{c_{k i}} \quad c_{k i} \in\{0,1\}
$$

Now we can write an example vector to which we apply the transformation, say:

$$\vec{x} = [7, 5, -0.5, 0.2]$$

And an example vector which parametrizes the transformation:

$$\vec{c} = [0, 1, 1, 0]$$

And for a start let's evaluate the transformation:

$$\phi_k(\vec{x}) = 7^0 \cdot 5^1 \cdot -0.5^1 \cdot 0.2^0 = 5 \cdot -0.5 = -2.5$$

And that's it.  But now comes the important part, let's describe what this transformation just did: We took a vector in 4 dimensions, took it's 2nd and 3rd components and multiplied them together (whilst we ignored the 1st and 4th components).  In other words the transformation $\phi_k$ uses its parametrization $\vec{c}$ to decide which components of its argument to product together.

We can therefore use different parametrizations to achieve different transformations of the vector $\vec{x}$.

What this may be useful for? If you are confident that all components in the vector are $> 1$ (or all are $< 1$) then you have the combined magnitude of the selected dimensions.  Although keep reading below since this is really useful to binary features.

How does this relate to the given formula?

i.e. how the binary features can only be 1 if all (relevant) features are present.  I believe that this relates to the definition of $0^0$ and $0^1$.  Let's take a different vector:

$$\vec{x} = [0, 1, 0, 1]$$

Let's say that these are non-exclusive binary features: [male?, dog?, white?, fed?].  So we have a female dog which is not white and is currently well fed.

Now let's take the parametric vector to be:

$$\vec{c} = [0, 1, 1, 0]$$

And perform the transformation:

$$\phi_k(\vec{x}) = 0^0 \cdot 1^1 \cdot 0^1 \cdot 1^0 = 1 \cdot 1 \cdot 0 \cdot 1 = 0$$

A small note is now in order: $0^0 = 1$ is not true in all fields of mathematics, but since it is accepted in most fields and makes sense here we will use it here (and so I believe that the authors of the paper do use this assumption as such).

Wait!  So had we had the dog been white we would have had a $1$ as the result.  In other words, had $\vec{x}$ been $[0, 1, 1, 1]$ we would have gotten $1$ as the result of the transformation because the third term would be $1^1$.  Try it out!

This means that $\phi_k$ with $\vec{c} = [0, 1, 1, 0]$ will return $1$ (true) for all $\vec{x}$ that represents a white dog, does not matter if the dog is male or female or whether it is fed.  And will return $0$ for anything that is either not a dog or not white.

In other words, the parametric $\vec{c}$ defines the features that matter (the ones in the vector $\vec{c}$), and such a parametrized transformation $\phi_k$ return true if all the features that matter are true in the argument of the transformation ($\vec{x}$).

The big difference in the binary features is the fact that $0^1 = 0$, which binds the entire product to $0$.  Whilst $0^0 = 1$ and $1^1$, which results in a dimension/feature whose value does not matter for our transformation.

P.S. I prefer physics notation for vectors, a component of a vector is $x$ but a full vector is $\vec{x}$ instead of $\mathbf{x}$.  Please bear with the notation.


How to make prediction using tensorflow models?

First, you need to convert the dataframe in numpy array or tf.data dataset that the model understands. For this purpose, the tutorial provides you with a function:

# A utility method to create a tf.data dataset from a Pandas Dataframe
def df_to_dataset(dataframe, shuffle=True, batch_size=32):
  dataframe = dataframe.copy()
  labels = dataframe.pop('target')
  ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
  if shuffle:
    ds = ds.shuffle(buffer_size=len(dataframe))
  ds = ds.batch(batch_size)
  return ds

test_ds = df_to_dataset(test, shuffle=False, batch_size=batch_size)

Then you can get the predictions like this:

model.predict_proba(x=test_ds)

Take into account that the test-input format should be the same as training-input format, so if you have done any preprocessing (on_hot_encode, standardize, bucketize, etc) to the training dataset, you should do it also to the test dataset

Another way, if your dataset is ready for prediction, you can just transform it to a numpy array:

model.predict_proba(x=test[feature_names].values)


Dimensionality reduction based answer: You should consider reducing the dimensionality of your data set. There are multiple ways to reduce the dimensionality of your data set

Reduction by variance: remove variables with very low variance
Reduction by correlation: remove variables that are only weakly correlated with your criterion. For this step, it is important that you split your dataset into training and validation data set before you calculate the correlations or your data results will be contaminated with information from the validation data set.
Reduction with principal components analysis (PCA)
Reduction with decision trees by feature selection
$R^2$ based reduction: Exclude one variable of your data set and calculate the $R^2$. Then exclude another variable of your data set and calculate the $R^2$. Repeat this process for all the variables and remove the variable for which the $R^2$ decreased the least or increased the most. Repeat this process for the reduced data set. Stop when the $R^2$ change is similar for all the variables.
$p$-value based reduction: Determine the weights and remove the weights for which the $p$-value is not significant. For this procedure, you have to consider the cumulation of errors and apply Bonferroni correction.

Regression method based answer: An alternative way without reducing the dimensionality would be to use weighted least squares regression and use higher weights on the sales if you really want to predict them more accurately. If your data is highly nonlinear you could also consider using nonlinear regression to capture the nonlinearity with your model.


Quantifying binary classification uncertainty

First things first: be sure to use a validation set in order to determine the optimal threshold, do not use the final test set.
Another general point: it depends what you mean by "false alarm", but in general you should not assume that it is possible for the classifier to keep the error rate under a predefined value (unless you have already done an experiment which shows this, of course). If by "false alarm" you mean only one type of error then it is usually possible to keep one type of error under a threshold at the expense of the other type of error. For instance if the goal is to minimize false positive errors, then one can let the classifier make as many false negative errors as necessary in order to keep the false positive rate under the threshold.
Now about quantifying uncertainty:

The first approach that you propose makes sense, but as you said it's not symmetric and also it wouldn't allow to convey uncertainty in any practical way. Ideally you'd want to be able to say something like "there's an X% chance that the prediction is correct" but you cannot obtain that from a difference with the threshold.
As far as I understand your second approach goes in the right direction but I'm not sure if the details are clear. So what you could do is to determine experimentally the rate of error (either FPR or TPR depending on which side of the threshold) for different intervals of predicted $\hat{y}$. For example if $T=0.9$ you can calculate the FNR for every interval $[0.0,0.1]\ [0.1,0.2]\ ...\ [0.8,0.9]$. This way for any predicted $\hat{y}$ you can provide the corresponding error rate.


Difference Between Feature Engineering and Feature Learning

Feature engineering refers to creating new information that was not there previously, often by using domain specific knowledge or by creating new features that are transformations of others you already have, such as adding interaction terms or as you state, moving averages.
A model generally cannot 'pick up' on information it doesn't have, and that is where  finesse and creativity comes into play.

Whether you should one-hot or leave a feature as categorical depends on the modeling approach. Some, like randomForest will do fine with categorical predictors; others prefer recoding.

Intuition on these questions comes with practice and experience.  There's no substitute for trying out and comparing toy examples to see how your choices affect outcomes. You should take the time to do that, and intuition will follow.


Dealing with time series data which has multiple observations for the same timestamp

It looks like you have lost a bit of information in that dataset. You shouldn't have 4  measurement for one timestep for one variable - how do you know which of the first four rows to use for 2010-05-02?

I would suggest checking your data source, or then working out a way to explain the meaning of the four values... are they different somehow (using other information)?

How are you even creating lags on that Date index? Take the average over each day?
Depending on the package you use for your Dickey-Fuller test (and other methods), they might not be made to deal with identical timesteps as input... so could explain why the session crashes.


image based example for Bayes error/irreducible error

As an MNIST dataset, see the mispredicted images on this figure below,  you notice for some samples, we as human don't know for sure what is the true label.

Maybe there are other factors that are not visible to us to predict the correct output.

Assume I want to predict customer will leave our company, maybe there are tons of useful features to start with, yet he may simply have some other problems that we are not aware of, and it may affect his decision in leaving the company.


In image classification/recognition how are images coded?

This will depend on the type of task you want to perform:
Object recognition is a wide task than can be approached in different ways such as:

[Multi-part OR][1]
[Multi-scale OR][2]
Multi-class OR
[Multi-label OR][4]
[Multi-modal OR]

Each approach (and its combinations) uses different representations for feature vectors extracted from the images. Most typical representations try to be invariant to certain conditions on the images depending on it purpose, such as scale, rotation and illumination changes.
Some successful representations are:

Bag of Visual Words [5]
Histogram of visual words [8]
Haar-like features [2]
Statistical-based structures (Active shape models, active appearance models) [9][10]

With the rise of deep networks, surprising results are being obtained feeding the images just as the raw pixels that represent them [6]. However, you will likely need hundreds of thousands of images to proper train a network that can abstract features to further recognize objects. You can actually take a look to the highly successful VGG Convolutional Network implementation at [Find actual links here][2]

The previous approaches will typically required that you have each example labeled with target value (cat, dog, etc). In case of deep learning you are being able to learn first representation of images with unsupervised data (not labeled) and the being able to train a classifier on top of the network in a supervised way (with labeled data)

You may find most of the references below at: [Google Scholar][3]

[1]: [Xin Chen and Shen(2009)] Xiaohua Hu Xin Chen and Xiajiong Shen.
Spatial weighting for bag-of-visual-words and its application in
content-based image retrieval. Advances in Knowledge Discovery
and Data Mining, 2009

[2]: [Viola and Jones(2004)] Paul Viola and Michael J. Jones. Robust realtime
face detection. Int. J. Comput. Vision, 57(2):137–154, May 2004.
ISSN 0920-5691.

[4]: [Jia Deng and Fei-Fei(2014)] Jonathan Krause Michael Bernstein
Alex Berg Jia Deng, Olga Russakovsky and Li Fei-Fei. Scalable
multi-label annotation. ACM Conference on Human Factors in
Computing Systems, 2014.

[5]: [Xin Chen and Shen(2009)] Xiaohua Hu Xin Chen and Xiajiong Shen.
Spatial weighting for bag-of-visual-words and its application in
content-based image retrieval. Advances in Knowledge Discovery
and Data Mining, 2009.

[7]: [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex
Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet
classification with deep convolutional neural networks. In
P. Bartlett, F.c.n. Pereira, C.j.c. Burges, L. Bottou, and K.q.
Weinberger, editors, Advances in Neural Information Processing
Systems 25, pages 1106–1114. 2012.

[8]: [Lowe(2004)] David G. Lowe. Distinctive image features from scaleinvariant
keypoints. Int. J. Comput. Vision, 60(2):91–110, November
2004.

[9]: [van Ginneken(2002)] A.F. Staal J.J. ter Haar Romeny B.M. van Ginneken,
B.Frangi. Active shape model segmentation with optimal
features. Medical Imaging, IEEE Transactions on. Volume 21, 2002.

[10]: [Matthews and Baker(2004)] Iain Matthews and Simon Baker. Active
appearance models revisited. International Journal of Computer
Vision. Volume 60., 2004.


How do I determine which variables contribute to the 1st PC in PCA?

Welcome to the site. PCA is an unsupervised dimensionality reduction algorithm. It works by transforming the original feature-set into eigen-vectors that are difficult to map with the original feature set. As such, the first Principal Component (PC) contains the features with maximum variance. The subsequent PCs contain features with decreased variance to the first PC.
With this background, I invite you to read this Q on SO. It has the solution to programmatically determine the features deemed most important by PCA.
[edited]
Regarding the sign of the components, eve if you change them you do not change the variance that is contained in the first component. Moreover, when you change the signs, the weights (prcomp( ... )$rotation) also change the sign, so the interpretation stays exactly the same:
set.seed( 2020 )
df <- data.frame(1:10,rnorm(10))
pca1 <- prcomp( df )
pca2 <- princomp( df )
pca1$rotation

gives
                PC1        PC2
X1.10     0.9876877  0.1564384
rnorm.10. 0.1564384 -0.9876877

and pca2$loadigs gives,
               Comp.1 Comp.2
SS loadings       1.0    1.0
Proportion Var    0.5    0.5
Cumulative Var    0.5    1.0

Then the question arises that why the interpretation remains the same
You do the PCA regression of y on component 1. In the first version (prcomp), say the coefficient is positive: the larger the component 1, the larger the y. What does it mean when it comes to the original variables? Since the weight of the variable 1 (1:10 in df) is positive, that shows that the larger the variable 1, the larger the y.
Now use the second version (princomp). Since the component has the sign changed, the larger the y, the smaller the component 1 -- the coefficient of y< over PC1 is now negative. But so is the loading of the variable 1; that means, the larger variable 1, the smaller the component 1, the larger y -- the interpretation is the same.
The conclusion is that for each PCA component, the sign of its scores and of its loadings is arbitrary and meaningless. It can be flipped, but only if the sign of both scores and loadings is reversed at the same time.
Furthermore, the directions that the principal components act correspond to the eigenvectors of the system. If you are getting a positive or negative PC it just means that you are projecting on an eigenvector that is pointing in one direction or 180∘ away in the other direction. Regardless, the interpretation remains the same! It should also be added that the lengths of your principal components are simply the eigenvalues.


Is fitting two RandomForestClassifiers 500 trees each and average their predicted probabilities on the test set more performant than one with 1000?

With a toy dataset, I obtained slightly better results with the RF 1000 trees
import pandas as pd
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

from scipy.stats import ks_2samp

import matplotlib.pyplot as plt
plt.style.use("seaborn-whitegrid")

n_models = 2
threshold = .5

X, y = load_breast_cancer(return_X_y= True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 42, test_size = .3)

rForest1000 = RandomForestClassifier(n_estimators= 1000, random_state= 42, oob_score= True).fit(X_train, y_train)
preds1000 = rForest1000.predict_proba(X_test)[:,1]

roc_score = roc_auc_score(y_true = y_test, y_score= preds1000)

print(f"Test set score for 1K trees is : {round(roc_score, 4)}")

ls = list()
for i in range(n_models):
    model = RandomForestClassifier(n_estimators= 500, random_state= i).fit(X_train, y_train)
    ls.append(model.predict_proba(X_test)[:,1])
preds = np.array(ls).mean(axis =0)

roc_score = roc_auc_score(y_true = y_test, y_score= preds)

print(f"Test set score for 500trees X 2 avg is : {round(roc_score, 4)}")

fig, ax = plt.subplots(1,2, figsize = (12,5))

ax[0].hist(preds1000[y_test == 0], color = "darkgreen", alpha = .5)
ax[0].hist(preds1000[y_test == 1], color = "darkred", alpha = .5)

ax[0].set_title(f"Predictions distribution with 1K Trees RF\n KS: {[round(ks_2samp(preds1000[y_test == 0], preds1000[y_test == 1])[0],3)]}")

ax[1].hist(preds[y_test == 0], color = "darkgreen", alpha = .5)
ax[1].hist(preds[y_test == 1], color = "darkred", alpha = .5)

ax[1].set_title(f"Predictions distribution with X2 500 Trees RF Trees RF\n KS: {[round(ks_2samp(preds[y_test == 0], preds[y_test == 1])[0],3)]}");


Hidden Markov Model (HMM) is a method that you could try out. They have been used to model very large sequences (like gene sequences).


Is there any similarity function to compare two strings and give them a score like scipy cosine similarity for comparing arrays?

Levenshtein distance is computationally expensive and therefore slow for large datasets. For a faster method, you can use sci-kit learn's CountVectorizer or TfidfVectorizer to get frequencies of n-grams for each string. This will produce a frequency matrix, which you can then use as the input for sklearn.metrics.pairwise_distances(), which will give you a pairwise distance matrix. Note that with a distance matrix, values closer to 0 are more similar pairs (while in a cosine similarity matrix, values closer to 0 are less similar pairs).

See this blogpost for a nice tutorial on how to go about this.


Scalable way to calculate betweenness centrality for a graph in spark

Sorry, I do not think you can compute the exact betweenness centrality of nodes in a graph this size, as its complexity is $O(n\cdot m)$ where $n$ is the number of nodes, $m$ the number of links.
The good news is that you may approximate it, and in a way that may take benefit from parallel computations. Indeed, computing betweenness centrality relies on counting the number of shortest paths from any node to any other. You may (randomly) select some nodes and compute the numbers of shortest path from each of them to all others, and use the obtained number to approximate the betweenness. The more nodes you select, the best the approximate will be, but it is empirically rather good even with a small sample set.


Comparing 2 data frames, if value is present replace with 1 or else 0

Here is a way using dplyr:

require(dplyr)
X = data_frame(A=c(1,2,3,4))
Y = data_frame(B=c(5,4,6,7))

X <- X %>%
  mutate(A = A %in% Y$B,
         A = A*1.0) #converts boolean to numeric

or using base R:

X$A <- 1.0*(X$A %in% Y$B)


Learning Rate based on error of the network

Your intuition is on point, and shrinking the learning rate like this is often referred to as "annealing". But linking the learning rate to error magnitude neglects certain problematic error surface topologies. An excellent motivating example is the Rosenbrock "Banana" Function, which is often used as a test case for optimization algorithms. The "banana" is a low error valley which hides the global minimum. If an optimization path finds its way into this valley, the path to the global minimum is along a nearly flat gradient.

If you use an optimization algorithm that naively shrinks the learning rate relative to the error, you're going to get stuck as soon as you hit the valley. On the one hand: congrats! You've achieved a low error solution. But you're not necessarily anywhere near the global minimum. So how can we do better?

An approach used by modern gradient-based methods like Adagrad, RMSProp, and Adam is to separately assign learning rates to each parameter, and tie the learning rate to the magnitude of the respective parameter's update. The Stanford CS231n lecture notes explains:

  Adagrad is an adaptive learning rate method originally proposed by Duchi et al..

  # Assume the gradient dx and parameter vector x
  cache += dx**2
  x += - learning_rate * dx / (np.sqrt(cache) + eps)

  Notice that the variable cache has size equal to the size of the gradient, and keeps track of per-parameter sum of squared gradients. This is then used to normalize the parameter update step, element-wise. Notice that the weights that receive high gradients will have their effective learning rate reduced, while weights that receive small or infrequent updates will have their effective learning rate increased.


How to Get Live Data Updates & Graphing with Tableau Public

I wrote the question above, worked on the problem a little longer, and came to the following solution.

Warning: it is anything but ideal, but it works.

Put simply I wrote a small Python script using only one external library - pyautogui - the others are all native (time json requests) which retrieves my data online, updates a local file and executes the actions required to manually update the workbook in Tableau. You just need to create your workbook and the visualization you want and then run the script.

Here's the script:

import pyautogui, time
import json
import requests
def auto_tab (url_name, file_name, workbook_name):
    url = url_name
    r = requests.get(url)
    data = r.json()
    with open(file_name, 'w') as f:
        json.dump(data, f)

        pyautogui.press('win')
    pyautogui.typewrite('Tableau')
    pyautogui.press('enter')
    time.sleep(5)
    pyautogui.hotkey('alt', 'o')
    time.sleep(2)
    pyautogui.typewrite(workbook_name)
    pyautogui.press('enter')
    time.sleep(5)
    pyautogui.press('f5')
    time.sleep(2)
    pyautogui.press('enter')
    time.sleep(5)
    pyautogui.press(['alt', 'space', 'c'])
    time.sleep(2)
    pyautogui.press('enter')
    time.sleep(10)
    return print('done!')

times_to_repeat = 3

while times_to_repeat >= 0:
    auto_tab('your_url', 'your_file', 'your_workbook')
    times_to_repeat -= 1

So, like I said: it's not ideal, but it works.

Downsides:

You need to have your machine running to run the script (perhaps you can run it from a server or a VM)

Upsides:

You can live graph even if you have embedded the graph in html
It's fast
It's free


Trouble in calculating the covariance matrix

Your approach is mathematically right, your problem comes from the fact that numpy matrix multiplication defaults to inner product when providing vectors, independently from them being transposed to row or column.

Try modifying the penultimate line like this to force outer product.

def cov_naive_2(X):
    """Compute the covariance for a dataset of size (D,N)
    where D is the dimension and N is the number of data points"""
    D, N = X.shape
    covariance = np.zeros((D, D))

    for i in range(N):
        x = X[:, i]
        covariance += np.outer(x, x)  # <---- here
    return covariance / N
```


Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

Spark (naively) uses average of vectors for all words in the document as representation of the document. Check the API documentation little carefully.

"The Word2VecModel transforms each document into a vector using the average of all words in the document ; this vector can then be used as features for prediction, document similarity calculations, etc."

If you are interested specifically in vector for a word (and not document), you can check getVectors method which will return data-frame of word and vector. The API behaviour does give rise to confusion though as it assumes that everyone wants to use averaging by default.


Is it acceptable for a forecasting model to predict moving averaged version of the data?

In estimation theory, the simplest way to estimate a random variable is to predict the average of its realizations. Very easy, very simple and very naive to be useful.

Let me explain more: in your problem which is the weather, the correlation is very high between the data from the last several days and the next day, which makes your estimation a good one but in many problems, the corelation between the data from previous days and the next day is small ( the stock market). Another thing you need to keep in your mind which is the cost of doing a mistake, meaning, predicting the temperature to be the average of the last few days (75), if the actual temperature is 78, then the difference is 3 may be acceptable, in stocks, if the average from the last few days is 10 and the actual value is 10.5. This difference 0.5 may not be acceptable.

What does mathematically this mean?
If the acceptable error in you estimation is << than the variation within the data having an average based estimator is not acceptable.

Regarding the issue of transforming the data from one domain to another, it is okay if you a deterministic mapping function that can take you back to the original domain. We prefer to have one to one mapping


Predict the accuracy of Linear Regression

There are several ways to check your Linear Regression model accuracy. Usually, you may use Root mean squared error. You may train several Linear Regression models, adding or removing features to your dataset, and see which one has the lowest RMSE - the best one in your case. Also try to normalize your data before fitting into Linear Regression model.

The confusion matrix is used to check discrete results, but Linear Regression model returns predicted result as a continuous values. That is why you get the error: your dv_test data likely is integer, but y_pred is float.

You may try using classification model if it is suitable for the problem you try to solve - depends on what you try to predict. But for regression problem it would be better to use metric mentioned above.


running scikit-learn on Sypder with Python 3.5

The reason is that you are printing the object of function. Instead call the function. Suppose the name of your function is func. You are doing something like

print(func)

instead try to call it by adding parenthesis.

print(func())


Is the gradient descent the same if cost function has interaction?

The usual way to use interaction terms in linear regression is to construct new $x_n$, e.g. $x_3 = x_1 x_2$, and treat those identically as any other $x_n$. The learned parameter $b$ does not "know" the difference in how you calculated $x$, and the problem is still considered linear regression even if you create really complex functions of $x_n$ to create an input.

Taking your example, but with slightly different notation:

The model estimate for Y is $\hat{y} = a + bx_1x_2$
The value of Y you want to learn is $y$
Mean squared error for a single example is $L = \frac{1}{2}(y-\hat{y})^2$ The factor of 2 does not change this answer, and is commonly used to simplify the gradient. Typically $C$ is the mean of $L$ over all examples.

In order to learn optimal value of $b$, for one example the gradient you need is $\frac{\partial L}{\partial b}$. We can get that by expanding the loss function:

$L = \frac{1}{2}(y-\hat{y})^2$

$L = \frac{1}{2}(y^2 - 2y\hat{y} + \hat{y}^2)$

We could expand further, but typically now we calculate $\frac{\partial L}{\partial \hat{y}}$ and use the chain rule, because that has a simpler, more intuitive-looking result. Terms without $\hat{y}$ are zero:

$\frac{\partial L}{\partial \hat{y}} = \hat{y} - y$

We want $\frac{\partial L}{\partial b}$ for gradient descent

$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat{y}} \frac{\partial \hat{y}}{\partial b}$ (Chain rule)

$\frac{\partial L}{\partial b} = (\hat{y} - y)(\frac{\partial}{\partial b} a + bx_1x_2)$

Again, terms without $b$ in them are constants:

$\frac{\partial L}{\partial b} = (\hat{y} - y)(x_1x_2)$

Note that the $x_1x_2$ term is unchanged from the input. It could be any function $x_n = f(x_1, x_2, x_3 ....)$


How does an encoder-decoder network work?

It probably won't. The whole point of the training was to encode cat images and thus the network has tried to learn what information is the most necessary to keep to ensure a low reconstruction error (i.e. what separates one cat from another) and what information can it throw away (i.e. what characteristics appear in all cat images and can be discarded).

That being said, a dog image would produce a fairly decent reconstruction because most features are shared between both animals. If you try, however, to reconstruct something completely different (e.g. a car) then it would probably fail.


Survival regression with major event that won't happen

If I understand your problem correctly, I think it's possible¹, but you have to do some extra work and you may be limited with what models you can use.

First, you have a time-varying dataset, so that must be handled correctly. Your poison comes long after birth, and that's important to model. Otherwise you are biasing your model. Ex: Suppose everyone is poisoned at age 90, if they live that long. Then just the act of being poisoned is a signal of living a long time, when in reality, it will hasten the death of the subject. Give this a read too, for many more arguments why it's important to model time-varying covariates.

Okay, so I took your dataset, and I modelled it into a time-varying dataset (see below). Notice:

A subject could have more than 1 rows (in your case, everyone will have at most two rows, one for prior to poisoning, one for after)
Each row for a subject is mutually exclusive (see start and stop columns). The status is True iff they died at the end of that interval.
poisoned is a boolean flag for activating after they were poisoned.

    start     poisoned sex   stop  id  status
0       0            0   f     90   0   False
1      90            1   f     92   0    True
2       0            0   f     90   1   False
3      90            1   f    150   1   False
4       0            0   f     90   2   False
5      90            1   f     91   2    True
6       0            0   m     60   3   False
7      60            1   m    130   3    True
8       0            0   m     60   4   False
9      60            1   m    150   4   False
10      0            0   m     60   5   False
11     60            1   m     62   5    True

The above matrix has the exact same information, but presented differently.

Okay, so we now need to pick a model that can handle time-varying datasets. Let's take a step back and ask "after fitting a model on the training data, can I "transport" that to the testing data (which has no poisonings)?" I argue that, yes, this is valid, and will be more valid you have individuals who died before being poisoned. What your model will do is understand the risk factors associated with your variables given poisonings / no poisoning. Hence, when you move to your test data, the risk factors are already learned and prediction can be done. Here's an analogy: I'm studying the student exam scores on Canadian and American students, and have other variables like lunch, grade, background etc and possibly interactions between them. The objective of "fitting" is to learn the relationship of lunch on exam scores given the other variables. So when I focus my model on only Canadian students in the test data, my model is still valid.

(However, if I had no Canadian students in my training data, then the model's validity can be questioned if I wish to focus on Canadian students in the test data. This is why I hope there are subjects in your training data who died before being poisoned).

We can fit the time-varying dataset using Cox's proportional hazard model. The (linear) model's output looks like:

---
             coef exp(coef)  se(coef)     z    p  -log2(p)  lower 0.95  upper 0.95
poisoned    17.60  4.41e+07   2670.86  0.01 0.99      0.01    -5217.18     5252.38
sex         -0.82      0.44      1.25 -0.66 0.51      0.97       -3.28        1.64
---

From your description, it sounds like your test data is not time varying. That's fine, it still fits into the time-varying framework: simply each test subject has a single row. You can choose to leave the poisoned variable out, or just leave it as 0 throughout. The latter is probably easier, especially if you have interactions.

¹ I could be missing some important gotcha that renders this all useless. Generally, working with time-varying data is easy to mess up causality/data leakage.


How to load all images using image_dataset_from_directory function?

You don't actually need to apply the class labels, these don't matter. Keras will detect these automatically for you. It does this by studying the directory your data is in. Make sure you point to the parent folder where all your data should be. Your data should be in the following format:
my_data/
...BacterialSpot/
...EarlyBlight/
...Healthy/
...LateBlight/
...Tomato/

where the data source you need to point to is my_data. Here is an implementation:
train = tf.keras.preprocessing.image_dataset_from_directory(
  'my_data',
  validation_split=0.2,
  subset="training",
  image_size=(128, 128),
  batch_size=128)

val = tf.keras.preprocessing.image_dataset_from_directory(
  'my_data',
  validation_split=0.2,
  subset="validation",
  image_size=(128, 128),
  batch_size=128)

Then when you run it:
Found 3647 files belonging to 1 classes.
Using 2918 files for training.
Found 3647 files belonging to 1 classes.
Using 729 files for validation.

Keras has detected the classes automatically for you. I have used only one class in my example so you should be able to see something relating to 5 classes for yours. See an example implementation here by Google:
https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/classification.ipynb#scrollTo=iscU3UoVJBXj


How to solve this problem as Machine Learning?

Generally, You can use anomaly detection to detect a sudden change in Timeseries data. SOM is useful for detect it. Its like you want to capture an anomly in continuous distribution of data.


Good that you raise the question because there's most likely a bug.

I've done over 100 ML models and never seen test accuracy being higher than train accuracy.

Potential bug? Most likely there are much more class labels in your training than in your test set. Solution? Try in class_weight = "balanced". This most likely fixes it.

Another potential problems? Did you fix the random seed? You did not set C at a high level? E.g. 100, 1000..

PS: Nice! I see you normalize first and then split train and test! Well done. By doing so, you avoid data leakage.


Wrapping legacy scripts for reproducibility

For HPC systems, wrapping scripts in a Singularity container https://sylabs.io/docs/ could be an option.


Should boolean features be normalized and should false be -1 or 0

Based on the comment by Ricardo Cruz I tried switching the value of false from 0 to -1 and also normalization on and off.

Neither switching the value from 0 to - 1 or normalization of features had any impact on the values predicted by the SVM. In this case I was using a Gaussian kernel and Sequential Minimal Optimization for my SVM.


How to compute unseen bi-grams in a corpus (for Good-Turing Smoothing)

I just remembered that we create a table with all possible words as the header of each row and of each column. As a result, the list of all bi-grams would be all possible bi-grams formed by concatenating any 2 words.


Weights initialization in Neural Network

Three layers with one hidden layer? This sounds wrong. You have 2 layers, input -> hides -> output, that's only 2 layers, with two sets of weights.

A single layer that gets initialized with only 0 will not converge, because the derivatives will be strictly identical (this rule also applies for any such last layer in a network, the layer gets basically useless). Here, there is a second layer after that saves the day, but it's still a bad practice in general.


From the documnetation

  Encode the object as an enumerated type or categorical variable.

  This method is useful for obtaining a numeric representation of an
  array when all that matters is identifying distinct values. factorize
  is available as both a top-level function pandas.factorize(), and as a
  method Series.factorize() and Index.factorize().

The examples section goes on to show that the output of the factorize method actually returns two things:

labels - referring to the new values for each of your classes
uniques - essentially the mapping back to your original labels

In your line of code:

df['product_name'] = df['product_name'].factorize()[0]

The part at the end: [0] means you are only taking the labels, throwing away the uniques that map back to your input.

If you keep both by making the same line:

df['product_name'], mapping = df['product_name'].factorize()

You could now do the rest of your work and end up with a results column full with the factorised output, you can use this line to get the original values back from those factorized labels:

mapped_back_to_product_name = mapping.take(results)

I suggest reading the documentation to get more information on how best to use the method  :-)


How to choose a classifier after cross-validation?

You do cross-validation when you want to do any of these two things:

Model Selection
Error Estimation of a Model

Model selection can come in different scenarios:

Selecting one algorithm vs others for a particular problem/dataset
Selecting hyper-parameters of a particular algorithm for a particular problem/dataset

(please notice that if you are both selecting an algorithm - better to call it model - and also doing hyper-parameters search, you need to do Nested Cross Validation . Is Nested-CV really necessary?)

Cross-validation ensures up to some degree that the error estimate is the closest possible as generalization error for that model (although this is very hard to approximate). When observing the average error among folds you can have a good projection of the expected error for a model built on the full dataset. Also is importance to observe the variance of the prediction, this is, how much the error varies from fold to fold. If the variation is too high (considerably different values) then the model will tend to be unstable. Bootstrapping is the other method providing good approximation in this sense. I suggest to read carefully the section 7 on "Elements of Statistical Learning" Book, freely available at: ELS-Standford

As it has been mentioned before you must not take the built model in none of the folds. Instead, you have to rebuild the model with the full dataset (the one that was split into folds). If you have a separated test set, you can use it to try this final model, obtaining a similar (and must surely higher) error than the one obtained by CV. You should, however, rely on the estimated error given by the CV procedure.

After performing CV with different models (algorithm combination, etc) chose the one that performed better regarding error and its variance among folds. You will need to rebuild the model with the whole dataset. Here comes a common confusion in terms: we commongly refer to model selection, thinking that the model is the ready-to-predict model built on data, but in this case it refers to the combination of algorithm+preprocesing procedures you apply. So, to obtain the actual model you need for making predictions/classification you need to build it using the winner combination on the whole dataset.

Last thing to note is that if you are applying any kind of preprocessing the uses the class information (feature selection, LDA dimensionality reduction, etc) this must be performed in every fold, and not previously on data. This is a critical aspect. Should do the same thing if you are applying preprocessing methods that involve direct information of data (PCA, normalization, standardization, etc). You can, however, apply preprocessing that is not depend from data (deleting a variable following expert opinion, but this is kinda obvious). This video can help you in that direction: CV the right and the wrong way

Here, a final nice explanation regarding the subject: CV and model selection


Which are the strategies to counter the 80/20 dilema in Data Science projects?

Some of these pain points are unavoidable. We are data scientists, we need lots of clean, relevant data and outside sources don't generally just hand that over. Someone on your team will have to do collection, organizing, cleaning, etc. The question then is, how do we make it as painless as possible?

Know your people. You presumably have a team, and I'm sure some people on it enjoy the analysis side of things while others prefer data engineering. Identify who is a best fit from both an enthusiasm and skill perspective and delegate accordingly. If you miss on this step, the work may get done, but it will be very slow compared to an environment where people are passionate about the problem they are solving.

Know your tools. How much data are you working with? If it's small enough to fit into the working memory on your machine and just run some Python/R scripts, then just do that. No need to be fancy when a simple solution will do. Just make sure to project into the future and verify your solution won't need to significantly scale up. If you're working with very large datasets or streaming data that won't play nicely with your local machine, then look into tech like Scala/Java running on a Spark cluster.

Know your process. Identify redundant tasks and cut them out as much as possible. Data science is an iterative process and the less time you spend repeating yourself the more you can focus on the actual analysis. That means no re-cleaning of data, updating existing models with more data rather than re-training, etc.


I have been able to find an answer in Tensorflow Warrior's answer here.

In Keras, when an LSTM(return_sequences = True) layer is followed by Dense() layer, this is equivalent to LSTM(return_sequences = True) followed by TimeDistributed(Dense()).
When return_sequences is set to False, Dense is applied to the last time step only.
Number of parameters were same even when I set return_sequences = False because even though applied to all time steps, they shared the same parameters, that is after all what TimeDistributed() does.


Predicting a numerical value based on past values and categorical attributes

Without having seen the data I would say that activity sensors may have a strong seasonal behavior. If that is the case time series analysis could give you some good results. Some basic before you head of in that direction, make sure to make your time series stationary before you use it in ARIMA. See for a good tutorial here.

If you want to forecast a categorical variable you will have to make sure to create sufficient covariates based on historical values. You could then proceed with a Random Forest.

In the end I would recommend you to consider how well you should be able to explain the outcome of the model. ARIMAX models are more tractable than Random Forest since the latter use bagging and averaging. So if you need to explain why your model gives a certain forecast, ARIMAX will probably be easier to explain.

Hope this helps.


In the context of Deep Learning, what is training warmup steps

As the other answers already state: Warmup steps are just a few updates with low learning rate before / at the beginning of training. After this warmup, you use the regular learning rate (schedule) to train your model to convergence.

The idea that this helps your network to slowly adapt to the data intuitively makes sense. However, theoretically, the main reason for warmup steps is to allow adaptive optimisers (e.g. Adam, RMSProp, ...) to compute correct statistics of the gradients. Therefore, a warmup period makes little sense when training with plain SGD.

E.g. RMSProp computes a moving average of the squared gradients to get an estimate of the variance in the gradients for each parameter. For the first update, the estimated variance is just the square root of the sum of the squared gradients for the first batch. Since, in general, this will not be a good estimate, your first update could push your network in a wrong direction. To avoid this problem, you give the optimiser a few steps to estimate the variance while making as little changes as possible (low learning rate) and only when the estimate is reasonable, you use the actual (high) learning rate.


How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

You don't necessarily have to train any differently if you are happy with the result. If you aren't, undersample the majority class and/or adjust the class weights in your loss function. The thing you really have to be more careful about is the evaluation of the result because a naive metric like the accuracy would give you false hope since the imbalance allows you to achieve a high accuracy by simply always predicting the majority class or randomly predicting each class according to its probability of occurrence. It is this baseline against which you have to judge your success; like the coefficient of determination in regression, where the baseline is the sample mean.

You could use a graphical approach (ROC or precision/recall curves), or report a size-weighted average of the class positive rate instead of the accuracy. Here are some other ideas: Comparison of Evaluation Metrics in Classification Applications with Imbalanced Datasets.


Why does my Multilayer Perceptron only classify linearly?

Your error calculation looks wrong:

if output1 < 0.99:
    error += (output1)**2
if output2 > 0.01:
    error += (1-output2)**2

If I am reading this correctly, output1 = 1.0 is a correct classification, since you treat any value above 0.99 as no error at all. However, you then measure the squared error as being output1 ** 2 which means the largest error is at 0.99. This is going to confuse the network, it will treat classifications above the line as perfect 1.0 if they score > 0.99, or otherwise want to adjust weights to make the classification 0.0. The opposite is true for your scoring of points under the line.

You need to either

Swap the conditionals (change < 0.99 to > 0.01 and vice-versa)

or

Replace (output1) with (1-output1) and (1-output2) with (output2)

in order to make the logic self-consistent. Which you choose depends on whether above the line points are positive class, or it is the negative ones. If above the line is positive you should use option 2.

In addition, as comments have pointed out, you might be a little ambitious with your approach. I spot the following things that make your job harder:

Using genetic algorithm search. Whilst it is possible to train small NNs like yours using a GA, it is less efficient. GAs might be a good choice for control systems or for a-life scenarios, but are far behind cutting edge for supervised learning.
Using mean squared error metric for classification. You should use logloss, which more heavily penalises bad guesses. With a mean squared error, there is a higher chance the network will settle for a few bad misclassifications if it also gets some other reasonable ones.
Ambitious function to separate. A neural network should be able to classify your function, but that's a lot of curvature to learn for a starting problem. Reduce it to just a couple of cycles to start with should help you debug practical issues.
No input normalisation. This may lead to saturated values in the network - close to 1.0 or 0.0, making it very hard to discriminate between good and bad weights. You don't show your weight initialisations - large initial weights can also cause this effect (although using a GA may actually help here, depending on how you are mutating weights).


Feature importance over a subset of instance space instead of an entire instance space

This is a common occurrence in practice. For example, in the bioinformatics space, a protein can change a phenotype significantly over a very limited range, but outside that range has little to no effect.

If you want to where the 'range of highest effect' falls I recommend partial dependence plots which are means of ICE plots. It offers an advantage over decision trees in that you won't overfit as easily to your training dataset and can be applied to high dimensional data. Here's an example of the plot. Read more here. The following packages have some implementation of the above:

mlinsights - https://ml-insights.readthedocs.io/en/latest/
https://media.readthedocs.org/pdf/pdpbox/latest/pdpbox.pdf

Here's a decent example from the above:

If you only want to know if something is important, not where it is important. You can use something called permutation importance or Mean Decrease in Accuracy outlined very briefly in a seminal random forest paper, page 20-21: https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf

Look into the eli5 package for easy calculation.


Can R + Hadoop overcome R's memory constraints in any case?

To answer your question. I would give an analogous answer to clarify it better. Apache Spark is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run tasks quickly but also utilizes clusters computing for better distribution of the workload.

Similar to that R+ Hadoop also works in the same fashion. What R effectively does is define the tasks or Machine Learning Algorithms to use and it translates over the nodes of the Hadoop cluster to better utilize the parallelization aspect.

Since the usual working of the R language is very RAM dependent this makes the processing time of massive data exponentially high. Usually, R solves this problem by allowing multithreading to take place within the cores so it can run the task as optimized as possible. But as you can tell this provides an upper cap on how much data it can handle. What R + Hadoop does is it provides more nodes so essentially translates to more cores it can use to run the following task.

Edit:  As requested more elaboration on the last point

Simply put when looking at just R as a language which is optimally built for data analytics, it utilizes multithreading to perform the required analytical tasks more effectively. This multithreading feature lets you run different tasks as different threads simultaneously so thereby trying to achieve as much as parallelization as possible within the limited capabilities of R.

And to answer your question whether R can perform the same tasks as R+Hadoop provided if there are no time constraints

The answer is theoretically Yes practically No!. Intuitively we might think R can handle such tasks by consuming more time. But that isn't always the case. It depends more on the task.

For example, if R were doing a simple frequency Count of the number occurrences in an array of a few million records you can obviously say that even with a few threads it might be slower than R+Hadoop and finish the task after a certain amount of time. But if the task were to say something like Eigen Value Decomposition of a Matrix, such a transformation requires a lot of calculation at the element level of a matrix. Assuming that the size of that matrix is a few million x few million, this type of calculation will surely cause a memory error using just any normal application as it will bottleneck all the data it has to hold temporarily into the RAM. This bottlenecking problem might cause the whole system to fail at times but the most simple case is that of a memory error.

Using R+Hadoop solves such problems as it lets you distribute the same strategy over multiple computers with their corresponding cores. Hence it lets you handle loads much more easily than running it on the same application, this manner you can also prevent the errors which I mentioned above. This is the power of distributed computing.


Doubt on scope of text classification problem

In case you have a lot of cases where the same brand is described in different ways, e.g. "K.F.C", "KFC", "Kentucky Fried Chicken", then it might be worth using clusters based on textual similarity measures. This kind of problem is similar to record linkage.

But if this is not the case, then it would be a bad idea to try to merge brands based on their names since you would end up with groups of brands which have nothing in common except some part of their name.

At a more general level, it looks to me like the design of your task is flawed: why do you need a unique instance for every seller? A seller can have multiple brands, each from a different source, right? That would mean that there are multiple sources for each seller, how do you plan to deal with that? Are you sure you don't want to predict the source for a particular pair seller + brand instead?


Information Visualization: which parameters should a data scientist plot to best explain the work behind a Neural Network?

There are different ways to explain your findings but associating the target with the inputs provided is tough and only a few architectures like Decision Trees do it well.

If you are trying to answer the question "why is my neural network missing out on a few samples from my validation set?" then its going to be a tough but possible road ahead. There are many good resources to interpretable machine learning

If you are trying to compare your model's (CNN) performance to other
architectures on the same data set, then looking at Receiver Operator Characteristics is a good point to start.

If you're having an unbalanced data set, then the F1 Score is a good thing to have look at. Being a classification problem (which is my assumption from your question), you can indicate the performance by a variety of derived quantities.


Why does the logistic regression cost function need to be the negative of log?

Normally we want to maximize the likelihood (consequently the log likelihood). This is the reason why we call this method maximum likelihood estimation. We want to determine the parameters in such a way that the likelihood (or log likelihood) is maximized.

When we think about the loss function we want to have something that is bounded by 0 from below and is unbounded for positive values. Our goal is to minimize the cost function. Hence, we take the negative of the log likelihood and use it as our cost function.

It is important to note that this is just a convention. You could also take the log likelihood and maximize it, but in this case, we would not be able to interpret it as a cost function.


If your goal is to use them to train a supervised machine learning, the best solution is to find out which one is more efficient in predicting your output.

AGE has more information than AGE_CATEGORY. So, If I were to remove one of them I would remove AGE_CATEGORY.

In Addition, if your goal is to train tree-based models AGE_CATEGORY is not gonna be that much efficient.

You can use A/B test to find out which feature is more efficient in predicting your output.


Deep Learning model for classifying skin diseases

If you are classifying images, you should obviously use CNN for good accuracy.
The dataset you are classifying also plays a great role in determining the hyperparameters, optimizers, loss function.
So you should start with a basic CNN model that you can train with optimizers such as SGD and try tuning the hyperparameters accordingly.
If you are asking about the transfer learning models you can use RezNet or GoogLeNet


operating on a dataset with 125,497,040 records

For 127 million rows, it is better to perform data prep on DB. It will be a select + insert query and will not require whole data to be loaded in memory.

SELECT YEAR(date) AS 'year', MONTH(date) AS 'month'
 FROM data

Edit : Once you start training / validation, even then it would be better to load few batches from DB (at a time). Most frameworks support this.

Foe example : https://github.com/keras-team/keras/issues/107


Problem in creating a Numpy Dataset for Classification problem

If you're going to use an MLP classifier (probably not the best option for image classification: why not CNN?), you should feed it a "flat" array. Each image has 25 pixels (features), so your input must be of shape (n, 25), where n is the number of training observations (10 in your case).

Currently your input is of shape (10, 5, 5) (dimension 3, hence the error message). Try reshaping it like this: X = X.reshape(-1, 25).


If you are looking for pre-trained model for your solution, you must already have a dataset which needs to be trained. However to get world class model, you need to build very deep networks, for e.g. Resnet50 has 50 layers, Resnet101 has 101 layers. Think of building those from scratch. It takes ages to train them.

So advantage of pre-trained model is you take existing model with already fitted weights and attach your last layer for new targets. Now, no. of trainable parameters should not be as large as original model. You need set layers unattainable with following code

for layer in resnet50.layers:
     layer.trainable = False

This makes only the last layer trainable. Hope this helps


Inception/ResNet doing worse than SIFT in feature extraction

After examining the dataset, we found that the problem was in NUSWIDE dataset itself. Almost half of the dataset didn't have supervised labels (81 labels entered by humans). Also we weren't training on the whole dataset, rather, we were taking random samples because some methods required that the whole dataset was loaded in memory which wasn't feasible.
After knowing that, we deleted the tuples that didn't have supervised labels, extracted features and done PCA and got a comparable results to SIFT (improved, in some cases).
So my advice if you face a similar problem is:

Make sure that your data labels are valid.
Use multiple layers for feature extraction, not just the deeper ones in the network.
Using PCA instead of random sampling dataset worked very well for us.


How do I interpret the length-scale parameter of the RBF kernel?

Even though I am more familiar with the use of RBF kernel with Gaussian Processes, I think your intuition is correct since, generally speaking, a larger lengthscale means that the learnt function varies less in that direction, which is another way of saying that that feature is irrelevant for the learnt function.

So if you have to choose which feature is the most important you could say that the one with the lowest lengthscale parameter is the most relevant.

All this only applies if the characteristic scale of the features (the standard deviation for example) is already comparable (i.e. you have to standardize the features before optimizing the lengthscales if you want to retain this intuition).

That said, making assumptions about what is making you overfit is difficult from the lengthscale viewpoint. I think we could say that shorter lengthscales across many independent features are going to increase model complexity, and thus the risk of overfitting. As always there's an optimal complexity for improving out-sample performance.

Just to complete the answer, I like to track the complexity of a kernel model by evaluating the kernel matrix k(x1, x2) and computing its effective rank as in https://infoscience.epfl.ch/record/110188/files/RoyV07.pdf . A lower rank means a less complex model (because all the points look more similar in your implicit kernel space), whereas a higher effective rank allows learning very complex functions but you lose statistical power since in your kernel space all your points are "outliers", in the sense that they are apart from each other.

I hope that helps.


Roadmap to learn CNN in tensorflow from scratch

I'm assuming you're talking about medical images classification, rather than localization.

Personally I recommend Kaggle, it has an awesome forum and people share their codes and opinions there.

You can start at Digit Recognizer, it's actually the well-known MNIST dataset(hand-written numbers). There's no relationship between MNIST and medical field. However there are some common techniques and tricks as they are both image recognition/classification problems.

If you encounter problems, read the other ones' codes at the kernel section, especially those with upvotes.

In the meanwhile, comment section is a good place to learn. I learned a lot there when I started to learn CNN.


what is the difference between Bias and overfilling and under fitting?

Well, I can't express enough that there are many many many articles and questions on bias-variance, over-fitting and under-fitting.

But let me also try to explain in simple words for you.

"Bias" as a word means- favoring a thing more than another. Like we heard Amazon's alexa doesn't respond to female voices/commands. so it was biased towards men.

Now if want to make a model which is able to predict on unseen data i simply don't want it to be biased like i did't want this system to favor male's voice over female'voice. If its biased it is essentially giving me an error. Its making more assumptions about my target function(which is taking voice commands here).

Now since its assuming a lot and giving me an error, its rather under-fitting.However it actually is not only high bias but+low variance leads to under-fitting but i am keeping variance aside for now. Now, there can be many reason for high bias and low variance.I am not going into details on reasons w.r.t. training/test size.

Now, if my model is able to predict the data correctly on given test data which is a good news,it means there are no errors ,it mean there is less bias. however may be its giving me no error because it learned the "training" data so well, captured every scenario/noise. Now again comes the variance, that there is high variance here, which together with low bias shall lead to over-fitting. it means it is well able to predict on the data i gave it to train but if i feed it new data it won't be able to predict it right i.e. generalization.

Now you might wanna understand the variance also. take again the simple dictionary meaning of variance - which is "the fact or quality of being different, divergent, or inconsistent." now apply this definition in relation with target function.

Please note this is a very very vague explanation for you to follow and there are standard definitions for the concept which i suggest you to follow.

Now there are way better explanation-
https://machinelearningmastery.com/gentle-introduction-to-the-bias-variance-trade-off-in-machine-learning/
https://stats.stackexchange.com/questions/4284/intuitive-explanation-of-the-bias-variance-tradeoff

Here is a good paper-

http://www.cems.uwe.ac.uk/~irjohnso/coursenotes/uqc832/tr-bias.pdf

Most precise visualization is below-


SGD versus Adam Optimization Clarification

In many applications I've seen (e.g. GANs) $\beta_1$ is set to $0$, so $m_1=g_1$, i.e. the numerator of the update rule is the same as in SGD. This leaves two main differences, both related to the MA of the second moment:

$v_t:$ raw MA of the second moment serves as a gradient normalizer that divides the gradient by the square root of the moving average of squares of gradients
$1-\beta_2$. To reduce bias, $\sqrt{v_t}$ is also divided by $\sqrt{1-\beta_2^t}$. This follows the derivation of the expectation of the square of the gradient, $\mathbf{E}[\big(\frac{\partial E}{\partial w_t}\big)^2]$ in Section 3 of the article. Essentially $\mathbf{E}v_t = (1-\beta^t_2)\mathbf{E}[\big(\frac{\partial E}{\partial w_t}\big)^2] + \varepsilon, $ hence the exppression. Early in the training MAs are close to $0$, and division by $\sqrt{1-\beta_2}$ helps move away from it.

In probability and statistics, moments refer to the uncentered expressions of the form $\mathbf{E}X^k$, which moving averages estimates, hence the name. Normalization allows the gradient adjustment, and, hence, better parameter update


Coefficient of determination of linear regression in Azure ML studio differs significantly from regression R square in Orange 3 for same dataset why?

It could help if you have added screenshots of both models settings. However, what I can say is Orange and Azure ML Studio use different parameter settings. It might be possible that some are available in one of them but not in the other.

If we assume that you have exactly the same configuration in your two models, then the random parameter can also play a big role here. Even if you use the same seed, Azure ML and Orange, probably, don't use the same random generator algorithm. As a result, the outcome of your models is not replicable.

I would advise trying with cross-validation in both implementations and dig more on the evaluation metrics. If you need more help from the DS community, you might need to add more details as well.


What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long?

Yep I figured it out. The answer is that by default GridSearchCV's last act is to expose the API of the estimator object you passed so that you can directly call things like .predict() or .score() on the GridSearchCV object itself. It does this by retraining the estimator against the best parameters it found during cross validation. If you want to skip this step (because, for example, you're going to go on to do more development or cross-validation afterwards) then you can pass refit=False to prevent that from happening.


How to find the feature regions where each label is the most expected when using decision trees?

It seems that you want to achieve something like this :

Where you can see the instances, classes and the predicted and the cutoffs for the rules. The exemple is taken from : https://jakevdp.github.io/PythonDataScienceHandbook/05.08-random-forests.html You might want to find one that is interactive (plotly ?) so you can get the rules that interest you by hoovering your mouse above the graph.
Note that this appraoch has some problems :

It only work with two variables at a time. You migth need to plot similar graphs for all your couple of features.
It only work for simple classification trees. It might start to get more difficult to interpret the plots and the rules if your output is continuous.


You can try the below code to merge two file:
import pandas as pd
df1 = pd.read_csv(‘first.csv’)
df2 = pd.read_csv(‘second.csv’)
df = df1.merge(df2, on=‘Column1’)


How data representation affects neural networks?

It will have very little effect

The answer most will give is that it will have no effect, but adding one more feature will decrease the ratio of records to features so will slightly increase the bias and will hence make your model slightly less accurate.  Unless, of course, you have overfit your model , in which case it will make your model slightly more accurate (a good data scientist would never do this because they understand the importance of cross-validation :-).

If you normalize your data and then attempt some sort of dimensionality reduction, your algorithm will immediately eliminate the feature that you added since it is perfectly negatively (linearly) correlated with the first feature.  In this case it will have no effect.

Please also consider the following:

I always see big red flags when someone asks a very fundamental data science question with the words neural network included.  Neural networks are very powerful and receive a great deal of attention in the media and on Kaggle, but they take more data to train, are difficult to configure, and require much more computing power.  If you are just starting out, I suggest getting a foundation in linear regression, logistic regression, clustering, SVMs, decision trees, random forests, and naive Bayes before delving into artificial neural networks.  Just some food for thought.

Hope this helps!


conceptnet api documentation for /wn/ part of the URI

This is the finer-grained word sense information that was added in ConceptNet 5.7. WordNet defines "music" in the topic area of "communication". We keep this information in the word sense. The idea is to distinguish WordNet senses, but not to make the extremely fine distinctions that WordNet makes.

I didn't describe it on the "URI Hierarchy" page yet, sorry about that. It's described in the changelog for now: https://github.com/commonsense/conceptnet5/wiki/Changelog

"music" is a tricky example, because WordNet has finely-distinguished senses of it that appear to be in different WordNet topic areas, even though they all seem to relate to the same meaning. Its different senses are listed in topic areas including "communication", "act", and even "music" itself.


What momentum does is to average out the derivatives in different directions in parameter space so that the directions that oscillate much will be close to zero, but for the parameters that do not oscillate this would not be the case - averaging will not vanish.

On the other hand RMSprop works as a dampener in all directions: larger the derivatives, larger the dampening rate. This means that the fast oscillatory modes with large derivatives will be dampened more. The learning in slow directions which you want to keep may also be dampened, but the rate will be smaller.

You can find more details on RMSprop and their combination Adam on the following Coursera course:

https://www.coursera.org/learn/deep-neural-network/lecture/y0m1f/gradient-descent-with-momentum


If you use a combination of INDEX and MATCH, you can let the desired look up array vary automatically as you drag the function across your columns, which is an alternative to manually fixing the look up column in VLOOKUP.

If you wish to continue with VLOOKUP, you could potentially add column numbers to the top of your data table, and reference these numbers instead of the hard-coded 97.


Linear regression with non-symmetric cost function?

If I understand you correctly, you want to err on the side of overestimating. If so, you need an appropriate, asymmetric cost function. One simple candidate is to tweak the squared loss:

$\mathcal L: (x,\alpha) \to x^2 \left( \mathrm{sgn} x + \alpha \right)^2$

where $-1 < \alpha < 1$ is a parameter you can use to trade off the penalty of underestimation against overestimation. Positive values of $\alpha$ penalize overestimation, so you will want to set $\alpha$ negative. In python this looks like def loss(x, a): return x**2 * (numpy.sign(x) + a)**2

Next let's generate some data:

import numpy
x = numpy.arange(-10, 10, 0.1)
y = -0.1*x**2 + x + numpy.sin(x) + 0.1*numpy.random.randn(len(x))

Finally, we will do our regression in tensorflow, a machine learning library from Google that supports automated differentiation (making gradient-based optimization of such problems simpler). I will use this example as a starting point.

import tensorflow as tf

X = tf.placeholder("float") # create symbolic variables
Y = tf.placeholder("float")

w = tf.Variable(0.0, name="coeff")
b = tf.Variable(0.0, name="offset")
y_model = tf.mul(X, w) + b

cost = tf.pow(y_model-Y, 2) # use sqr error for cost function
def acost(a): return tf.pow(y_model-Y, 2) * tf.pow(tf.sign(y_model-Y) + a, 2)

train_op = tf.train.AdamOptimizer().minimize(cost)
train_op2 = tf.train.AdamOptimizer().minimize(acost(-0.5))

sess = tf.Session()
init = tf.initialize_all_variables()
sess.run(init)

for i in range(100):
    for (xi, yi) in zip(x, y):
#         sess.run(train_op, feed_dict={X: xi, Y: yi})
        sess.run(train_op2, feed_dict={X: xi, Y: yi})

print(sess.run(w), sess.run(b))

cost is the regular squared error, while acost is the aforementioned asymmetric loss function.

If you use cost you get

  1.00764 -3.32445

If you use acost you get

  1.02604 -1.07742

acost clearly tries not to underestimate. I did not check for convergence, but you get the idea.


How to know if the deep learning model is overfitting or not?

Please consider going through other questions before posting a question. This will not only help you learn more by showing you multiple perspectives to a problem, but will also help you frame a clearer question explaining why the answers to other questions don't apply to your problem.

Refer this link for your answer.


Fill the null values in the dataframe with condition

traindf.loc[traindf['Gender'] == 'female','Age'] = traindf.loc[traindf['Gender'] == 'female','Age'].fillna(value=femage)


As per my understanding, ML works on the concept of the type of Input+Output you feed during training, will expect(Input) and predict the same output.

So labeling is must for Supervised learning.

For Image localization use case, you have to trained your model with similar type of data i.e. label should have full annotation/details with respect to dimensions(x,y,height,width) and object class. Only then you can expect the Object localization(bounding box) during prediction.

Few of my posts possibly be helpful for you:

How YOLO training and prediction works for an object fall in multiple grid?

YOLO: How many bounding boxes?

Please go through these videos for more details. Object Detetction and Localization


Firstly for your question 2. The precision and recall is calculated across all images, for a single class. Not on a per-image basis and averaging across images. The "A" in "mAP" means averaging across different recall levels, as you stated in step 4.

For your question 1. Whether to report a same bounding box to both the cat and the dog class is a decision that your detection algorithm should make, not a decision that the evaluation function should make. You can imagine that input to the evaluation script to be a ranked list for each class, for example:

class [dog]                        |  class[cat]                        |  class[foo] ...
image=17, score=0.98, bbox=[...]   |  image=41, score=0.75, bbox=[...]  |  ...
image=33, score=0.60, bbox=[...]*  |  image=07, score=0.45, bbox=[...]  |  ...
image=93, score=0.55, bbox=[...]   |  image=33, score=0.40, bbox=[...]* |  ...
...                                |  ...                               |...

The evaluation script will calculate the AP for each class and it won't care whether the same bounding box (*) appears in different classes.

So for your detection algorithm, you can try different strategies and see which one achieves higher mAP.


Error plotting with datetime and value using matplotlib in python

Matplotlib plots data in lists from left to right.

Hence, in the list x[idx] which is used in ax.plot(x[idx],y[idx], color="C0"), I assume that either the first element (according to x-axis) is in the back of the list, or the last element is at the beginning of the list.


Cannot predict the prices using the date parameter

Your problem is with the date field?

In Machine Learning algorithms you have to feed the model with numerical data and date is not numerical. You can see in this posts what are the common techniques of encoding time: https://stats.stackexchange.com/questions/311494/best-practice-for-encoding-datetime-in-machine-learning

So you have either two options: encoding date time or drop it.

ds = ds.drop('Date', axis=1)


Scalable way to group users with similar titles purchased

This issue seems somewhat similar to a recommendation system problem, if I understood correctly, in which you have several titles you want to recommend to users based on their previous interactions.

Maybe you could read on that? You can search for algorithms such as Matrix Factorization/Collaborative Filtering.

A good resource for that: https://course.fast.ai/videos/?lesson=4


Reinforcement Learning on real time data over a web server

Yes - You need to define an Agent, Environment, State, and Rewards:

Agent - The current week process.
Environment - Server.
State - Current data.
Rewards - Predicted future values are the same observed values.

Then implement in JavaScript. One JavaScript framework is REINFORCEjs


Difference between a target and a label in machine learning

Target: final output you are trying to predict, also know as y. It can be categorical (sick vs non-sick) or continuous (price of a house).

Label: true outcome of the target. In supervised learning the target labels are known for the trainining dataset but not for the test.

Label is more common within classification problems than within regression ones. Nonetheless, they are often used interchangeably without great precision.


You always need to flatten your pictures when connecting the input to a Dense layer in Keras (Note that this is not the case for CNNs or RNNs). The reason is when the dense layer is built, based on the Dense layer code, the input dim is the last element you pass in the inputs (input_dim = input_shape[-1]). Therefore, although you are passing an input of (28,28) keras thinks that the shape is only 28. This also explains why the input of (,784) indeed works.

You can check the Dense layer code here


Geospatial Social Network Analysis Visualization

Your dataset can be viewed as a directed graph. The party's location (latitude and longitude) can be denoted as a node and the directed edge can denote who referred whom. Once the dataset can be viewed as this, the problem boils down to joining co-ordinates with lines.


Where in the workflow should we deal with missing data?

My strong opinion regarding automated tasks like imputation (but, here I can include also scaling, centering, feature selection, etc) is to avoid in any way do such things without carefully inspecting your data.

Of course, after deciding what kind of imputation to apply it can be automated (under the assumption that the new data has the same shape/problems).

So, before anything, take a wise decision. I often wasted time trying to automate this things, destroying my data. I will give you some examples:
- a marketplace encoded as N/A, which I missed and considered to be North/America
- numbers like -999.0, because the data producer could not find a better replacement for missing data
- number like 0 for blood pressure or body temperature, instead of missing data (it is hard to imagine a living human with 0 blood pressure)
- multiple placeholders for missing data, due to the fact that the data was collected from various sources

After that you need to understand what kind of imputation would resemble better the information from your data for a given task. This is often much harder to do it right than it seems.

After all those things, my advice is to delay your imputation task to an upper layer where you have tools to reproduce on new data and to inspect if the assumptions for the new data are not violated (if it is possible).


Tensorflow MirroredStrategy() looks like it may only be working on one GPU?

I'm seeing the same thing, here I enabled just the first 2 GPU's.
If I force it to use just 1 GPU the memory utilization drops but the amount of time per epoch is unchanged.


In addition to what you have already done, in my notebook I added two lines:

import os
os.environ["PATH"] += os.pathsep + 'C:/Program Files (x86)/Graphviz2.38/bin/'

(Source)


When you have 32 feature maps with height and width equal to 100 and the depth of each equal to one it means that you have 32 planes, a common jargon among vision people, with 100 by 100 entries. You can set the height and width of the next layer and they can be arbitrary. You can also set the number of feature maps but the depth of each feature map would be equal to the number of feature maps of the previous layer. So it should be 16 * 9 * 9 * 32 if you set height and width equal to 9 and the number of feature maps to 16. As you can see in 16 * 9 * 9 * 32, 32 is located at the end, this is called channels last. You can not set the depth, because it should be equal to the number of channels, features maps, of the previous layer.

16 * 9 * 9 * 32 means that you have 16 feature maps of dimension 9 * 9 * 32, so the output of each feature map would be the member-wise product of all the outputs of the feature maps of the previous layer and each of 9 * 9 * 32 kernels. Consequently the result would be 16 planes.

I highly recommend you taking a look at here.


I wanted to know what is the answer to the question 'how is your policy expressed'?

Typically when this is asked, what the asker wants to know is something like:
How do you represent the action choice in the output?
For example, if there are N discrete action choices, and your policy is stochastic, you could say something like: "The policy is expressed as a vector size N, with each value being the probability of choosing a matching action." You may also want to enumerate the actions - to say which index of the vector corresponds to which specific action. That depends on what need the person communicating with you has for understanding your policy.
Plus maybe:
How do you represent the state in the input?
The same thing here, give the state representation required to make the policy function work according to your implementation. Again, whether you should supply this depends on why the person asking you wants to know.

Am I supposed to give an analytical function?

With the right level of detail, that should be fine. The input and output domains plus maybe the index map (of which input and output features are in which positions). Note that because you are asked for how the policy is expressed, you would usually stop at the point that you have defined the probability distribution over the action space.


How to find similarity/distance matrix with mixed Continuous and Categorical data?

Similarity measures are subjective and so are they ways to combine them.
You should decide what is your subjective definition of similarity and then find a way to combine them that fit your definition.

In general, I like to reduce similarity problems into classification problems. Given the dataset of items you have, create a new dataset of item pairs. The concept should be whether the two items in a pair are similar. Each similarity measure you have is a feature of the pair. Note that now you are in the good old classification framework. You can evaluate the similarity measures by computing the mutual information/accuracy/your chosen metric given the concept.

In your case, I would build the dataset this way. First choose pairs of items on which both your measures agree on. Choose pairs that are close by both the Euclidian distance and the cosine distance or pairs that are far by both measures. Note that since the similarity measures agree on these pair, combining them usually  leads to agreement too and not that important. However, you should use such pairs so they will stabilize your weighting function. Other than that, you can easily generate these pair, which is always an advantage.

Next, you should choose pair on which your similarity function disagree. Choose some that are considered close by the first and far by the second and vice versa. Manually label the pairs in order to decide whether they are close or not.

Once you built the dataset, you can present your subjective definition of similarity as a set of pairs and distances. Now run any classifier (in your case a linear regression might fit what you look for) and use the result as the combined similarity measure.


Regression performance with Feature Selection

Generally speaking, some ML algorithms include regularization (such as Ridge/Lasso Regression, Random Forest, CatBoost,...) and others don't (k-NN, Gaussian Naive Bayesian,...). Using one that includes regularization may not change the performance when removing less important features.
This applies to your first question. If you use Lasso regularization for example, it will push the weights of "less important" features to zero. So removing them should not change the performance. Using Ridge regularization is a bit different.
So when you say:

MSE increased gradually while I am removing features step by step

There could be multiple reasons for that:

the features you removed contained information and were not "less important" for the regression (see *Note/warning)
even the less important features contain information
you did not retrain your model after removing the "less important" features

Regarding your second question, $R^2$ is related to $MSE$ (it is well explains in this post), so if the $MSE$ increases, $R^2$ should increase as well.
*Note/warning: a feature is often said to be "less important" using one algorithm (such as CatBoost which often uses Gini or entropy). It does not always mean this feature will be "less important" when using it with another algorithm (such as regression).


Replace missing values of numerical features with unique numbers

This depends on the exact learning algorithm which is used, but most of the time numerical values are treated... numerically.

It's easy to see with the case of linear regression: let's say a feature has values between 0 and 100, but with a few missing values. Now replace missing values with -999: the coefficient learned for this feature will be completely different and based on  meaningless indications, so the performance will decrease.

This idea might work in the case where the numerical values are 'binned' into several classes, i.e. transformed into ordinal/categorical variables.

But the best way to use such values is to use an algorithm and framework which can deal with missing values in the first place.


Using standard deviation as a metric for model selection

Does the validation score's standard deviation have a correlation with overfitting / error

Yes definitely: a high variance shows that the model is not stable across different training sets, which indicates a high risk of overfitting.

  and can this be used in my scoring?

Using the std dev directly in the scoring itself, I'm not sure. I would consider this more like a kind of qualitative indication. However if two different models have  similar performance but one with higher variance than the other, it's usually a good idea to select the latter.

  currently the stdev is around 5% that seems ALOT.

It always depends on the specifics of the data, but yes I'd say that 5% is quite high.

  I propose that if I reduce the stdev of the validation scores of each k fold then this should decrease variance on out of sample predictions?

I don't understand what you mean here.

In general the standard way to reduce overfitting is to increase the ratio between number of instances and number/complexity of the features:

Add more instances in the training set if possible (but it's rarely possible)
Apply feature selection to reduce the number of features
Simplify the most complex features; that depends on the task, but typically removing values which appear very rarely can help.


Industry Standard Process for Fraud/Outlier/Anomaly Detection

You should refer this survey paper on Anomaly Detection (from University of Minnesota).

Please let me know if this helps you.


On assiging weights for unbalanced classes

What is the proper way to assign weights? Should I assign weights on the full dataset and then split into train and test?

No, your test set doesn't have to be weighted and shouldn't be, because it should reflect the real distribution of the data. So you should split first and weight only the instances in the training set, since this is during the training stage that these weights are taken into account.


How to increase number of outliers in a dataset?

I'm assuming you want to create a point that, each column by itself appears normal, but when looking at all the columns appears as if it's an outlier (thus you'd need some sort of outlier detection). Thus the method of generating an outlier would require looking at all the dimensions in relation to each other. And since we didn't assume normality here, generating is not straightforward.

I would recommend first using some kind of outlier detection method from here on the original dataset, (Somethind like an Isolation Forest would work)

Then you can generate random numbers, (or use the numbers you generated) to test if they are outliers or not. This should be easy to do by hand since you only want 7 points and each point only has 4 dimensions. Also an additional tip would be to test the numbers using one of the methods that returns a score instead of a 0,1 prediction so that you can make sure it's not too obvious of an outlier (since you didn't want that).

Lastly, if you generated points, some sort of sanity check would be to append those points to the dataset, apply PCA to reduce the dimensions down to 2, plot the PCA result with a separate colour for the appended outlier points. And you can check by eye if the outliers are far apart but not too far apart from your dataset.

Hope this helps and gives you some ideas.


How to fairly rank the best items having two set of grades?

Procedure-1:

Convert them into percentage and then take average of outcome of both.

so for example for person A, judge-1 rated 5 out of 7 and judge-2 rated 7 out of 10,

converting judge-1 score to percentage i.e., $5/7 *100$ = 71%(rounded 71.45)

converting judge-2 score to percentage i.e., $7/10 *100$ = 70%

Now take average of both i.e., $71+70 / 2$ =  70.5%, now you can convert it back on to a scale from $1-10$ :  This person A's overall rating is 7(rounded 7.05 ) out of 10.

Procedure-2:

You can normalize the rating of both the judges by using this formula,

$$z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)}$$

where $x=(x_1,...,x_n)$ and $z_i$ is now your $i^{th}$ normalized data.  As a proof of concept (although you did not ask for it)

Procedure-3:

You can convert the 0-7 rating to 0-10 and then take average of both.

Once you get the outcome then you can rank them accordingly.

Do let me know if you have any doubts.