Ordinal data deals with categories, which themselves are ordered by meaning somehow, but we cannot strictly talk about the distance between the categories. In the example below, you can see that the ordinal data runs from awesome to terrible, with 5 categories explaining an entire ordinal range:

Having a look at the paper, it seems the define a uniform distribution $U(a, b)$ (probably continuous), which they then divide into some random categories, let's say 5, to follow the table above. So if we set the bound of the distribution, $a = 0$ and $b = 1$, we might randomly create five categories, such as:

Category :  a      b
----------------------
Awesome  : 0.00 - 0.09
Great    : 0.10 - 0.44
OK       : 0.45 - 0.46
Bad      : 0.47 - 0.59
Terrible : 0.60 - 1.00

We can see that they are ordinal, because they numerical values do signify some kind of meaningful order, but the size of the categories is not necessarily uniform. The authors generated them randomly.


Training loss = 0, training accuracy =1, validation and test around 85%

To answer your last question - think of the model as your brain trying to give a maths test. Training data is what you encountered during homework/exercise and validating/testing data is what you encounter in the final examinations(most likely unseen data). To think of yourself as proficient in mathematics, you'd want your brain to be able to perform best on these unseen data.
Following the above reasoning, you'd want the snapshot of the best performing model in terms of epoch.
Why validation you ask? The answer can be found in the existence of separate validation and test set. The validation set is used as part of the training for model selection. Whereas the test set is used at the end for the final analysis of the model before being deployed for real-world data. Think of it as taking mock tests before taking the actual examination. You can combine the two and label them as validation but after that, you will have no reference for their performance as real-world data.
Here's an excellent answer that talks more about the difference between validation and test data.


Does it make sense to build a ROC for a decision tree where there are multiple threshold you can adjust?

The ROC Curve has no relation with the way your model works, but instead, with its outputs. If the target is binary and your model outputs anything in between 0 to 1 (e.g. [0, 0.2, 0.4, ..., 1] or continuous probabilities), there's a sense in building the ROC Curve. If instead, the only outputs of your model are 0 or 1, the ROC Curve would be kind of useless, and computing simpler metrics like Precision and Recall would make more sense.


How does scikit-learn decision function method work?

Both approaches seem to do the same:

safe_sparse_dot(X, self.coef_.T, dense_output=True) + self.intercept_

seems to compute a dot product of the coefficients (you called these $w_1, w_2$, and combined them to $\theta$), and the point ($x_1, x_2$, combined to $\mathbf{X}$). The intercept $b$ is added. So the formulas should do exactly the same, once in mathematical, once in Python notation: they compute the distance to the separating hyperplane, called the margin.

You also wanted to know

  how it works in the case of SGD Classifier

As that also fits a linear classifier, the computation should be the same (even though the hyperplane might have different parameters).


Simulation - identify the input parameters that impact the most the output

With my background on Data Analysis and Nonlinear Regression, your question seems to be very general and blurry. I guess what you are looking for is called Sensitivity Analysis in some contexts.

You need to understand that parameters may have different orders of impact on different sub-spaces of variables (and Parameters) space.

You may also want to generate multiple sub-models based on this sub-spaces, so in each sub-model only a few of your parameters have the main effect and the rest could be neglected.

Again you need to make your question a little more detailed. for example tell us why you want to do this analysis. because your final goal totally impacts the solution.


Normally we imagine a neural network propagating from left (input) to right (output), but what is the layer above/below the current layer? Is the input the bottom layer or the top layer?

Yes, a "below" layer means an "earlier" layer, i.e. closer to the input. For CNNs it is quite common to speak of lower layers, lower level feature maps etc.

And for RNNs you can even take this literally as they are usually visualized bottom-to-top instead of left-to-right (e.g. see figures here).


Performance of four GTX 1080 Ti's versus one Tesla V100 for Deep Neural Network training

This link has more realistic benchmarks of 1080ti vs Titan V (same as the V100 but less RAM), using pytorch. Its not nearly the best case scenario of 10x speedups, and seems to be a more modest 20% to 50% boost. This is fairly expected, nVidias grandiose performance claims tend to be only applicable to fairly restricted benchmarks that don't scale well to real world usage. You'd probably get a lot more out of 4 1080ti cards than a single V100 unless you REALLY need a large amount of RAM in a single card.


Interactive labeling/annotating of time series data

Update: we have updated TRAINSET to include the ability to upload multiple series as well as apply multiple labels! See demo in GIF below.
We had this same problem again and again at Geocene, so we came up with this open-source web app called TRAINSET. You can use TRAINSET to brush labels onto time series data. You import data in a defined CSV format, then label the data, and export a labeled CSV. You can also import a pre-labeled CSV if you're really just trying to refine labels. You can use the hosted version of TRAINSET at https://trainset.geocene.com or you can deploy it yourself by following the readme at https://github.com/geocene/trainset


In your specific case you only have 2 clusters, however this is not necessarily always going to be the case. I would allow for more flexibility.

I assume from your sample code that you are following what is shown in the docs. Following from what they are doing you should have the following

feature_cols = ['age','workclass','fnlwgt','education','education num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country']
X = train[feature_cols]
y = train['label']

# split X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30,random_state=10)

X_train_scale = scale(X_train)
X_test_scale = scale(X_test)

reduced_data = PCA(n_components=2).fit_transform(X_train_scale)
reduced_data_test = PCA(n_components=2).fit_transform(X_test_scale)

from pylab import *
xx, yy = zip(*reduced_data)
scatter(xx,yy)
show()

# Compute DBSCAN
db = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)
core_samples_mask = np.zeros_like(db.labels_, dtype=bool)
core_samples_mask[db.core_sample_indices_] = True
labels = db.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)

I just changed your variable name dbscan to db to make it easier to go through the code from the source. From here you can plot all the different clusters that were identified by the DBSCAN method. You should also keep the mask which retains the labels as a list in order for us to access them easily when we plot.

We will identify what are the unique labels which DBSCAN identified and we will map a color to each of these labels.

# Plot result
import matplotlib.pyplot as plt

# Black removed and is used for noise instead.
unique_labels = set(labels)
colors = [plt.cm.Spectral(each)
          for each in np.linspace(0, 1, len(unique_labels))]

For each cluster as determined by the unique labels we will plot all the values associated with it as

for k, col in zip(unique_labels, colors):
    # -1 is an identifier for noise
    if k == -1:
        # Black used for noise.
        col = [0, 0, 0, 1]

    # Find out what instances belong to this cluster, k
    class_member_mask = (labels == k)

    # Pull out these instances
    xy = X[class_member_mask]

    # Plot all of these instances
    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
             markeredgecolor='k', markersize=6)

plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()


Split text into phrases of a person and an operator

I think the task you're looking for is called "Punctuation restoration" or "Punctuation prediction". An example of this is described in this paper, or this other paper.
Unfortunately I am not aware of available per-trained model that you might apply to your dataset. Honestly I think that it is unlikely that something trained specifically for costumer service calls exists, unless you want to annotate your own dataset and train a model out of it.


In that great post, Joel Spolsky use Monte-Carlo simulation to give an estimation of the end of a project.

Thinking about a similar concept whould be to replace the notion of velocity by efficiency.

$$efficiency = \frac{quote}{hours} $$

The more discrete the scales of the quotes, the better.

It should work even with a few weeks of data.


Fitting Lasso model repeatedly on same train dataset

In principle, that shouldn't happen.  If you rerun Lasso on the same data set, with the same hyperparameters, after dropping attributes whose coefficient was set to zero in your first Lasso run, you should get the same result.  If you see something different, something has gone wrong.

You'll need to investigate deeper to see if you can figure out why that is happening.  Perhaps there is a bug in your code, or perhaps you are not providing the right settings to your implementation of Lasso, or perhaps the optimization algorithm is not finding the exact optimum solution, or perhaps something else has gone wrong.  It's hard to say with the level of detail you have provided.

If you're using different hyperparameters the second time you run Lasso, anything could happen.  Don't do that; that doesn't make sense.


How to handle One Hot Encoded columns with changing categories in supervised ML Problem?

The main idea of Machine Learning is that your training data is similar to test data. If that hypothesis does not hold, then big chances of failure are guaranteed.
Given that, you still can handle unseen categories even thought the solution is not good and more with One Hot Encoding
In the category encoders library you have the following hyperparameter with the following options

handle_unknown: str
options are ‘error’, ‘return_nan’, ‘value’, and ‘indicator’. The default is ‘value’. Warning: if indicator is used, an extra column will be added in if the transform matrix has unknown categories. This can cause unexpected changes in dimension in some cases.

But, given your problem and your data I am not sure if you will ever be much more successful than doing some descriptive statistics.


What machine learning model should be used to predict coincidence factor

From what you are saying I guess you have both categorical and continuous variables, One suggestion would be to check the distributions of your features to see if, more or less, the criteria for a linear regression are met.
If you have too many variables, which I still haven't picked up, you could even apply a PCA and then a regression.


References/tutorials about data mining and machine learning

I recommend "An Introduction to Statistical Learning" (ISL) by Gareth James, Daniela Witten, Trevor Hastie and Rob Tibshirani. The book is available online.
The book covers a number of topics in ML and for each topic there is a "lab" in R (code is also available for Python, in fact there are several pages on Github covering the labs).
ISL is not too technical but gives a sound introduction. In case you want to advance on some topics, you can switch to the advanced book "Elements of Statistical Learning" (ELS).


Memory error on using data generator in keras

It seems like the root cause for this is that zca_whitening is set to True. According to the answer from the keras-collaborator rragundez in this related Github issue, there is no work-around for it, except disabling ZCA whitening:

  This is known problem with the ZCA method. There is no real immediate way around it. The problem is with the calculation of the sigma matrix which is a dot product.

The other alternative to disabling it would be to use images with a smaller resolution. But I suppose that's rarely a practical solution.


Numpy element wise comparison for a particular value in two arrays

There are two ways I'll show you (there are probably a lot more using NumPy):

First method: chaining operations

You can use "masking" followed by the comparison and finally a sum operation:

We want all values in a from the indices where b is equal to 1:

part1 = a[b == 1]

Now we want all places where part1 is equal to 1

part2 = part1[part1 == 1]

now we are left with all the places where a and b are equal to 1, so we can simply sum them up:

result = part2.sum()

Method 2: built in numpy.where

This is much shorted and probably faster to compute. NumPy has a nice function that returns the indices where your criteria are met in some arrays:

condition_1 = (a == 1)
condition_2 = (b == 1)

Now we can combine the operation by saying "and" - the binary operator version: &.

part1 = numpy.where(condition_1 & condition_2)

To get your desired output, we can take the length of the resulting set of indices:

result = len(part1)

Read the documentation about numpy.where to see the other things it can do for you!


Should I seperate the data for a k-means model?

Yes you should definitely split these two up. When calculating the TF-TDF matix, it will give different terms to objects of the same entity, because of the difference in language. This will affect your clustering results.


Improvement in the evaluation metric by rebalancing depends on the type of classifier algorithm. Certain algorithms are very sensitive to rate imbalances, thus adjusting the respective levels of support will change the algorithm's performance.

One way to empirically verify that only under-sampling is driving the improvement is to run an experiment - gradually increase the level of under-sampling and plot the result on the F1 score.


How to prune the features using recursive feature elimination?

StratifiedKFold tries to keep balance between representation of classes in each fold. You have 5 folds and one or more of your classes have less than 5 instances in the data i.e. they can not be spread among 5 folds. So you can not use it.
That was the minor problem. ML-wise you have very few data points so the right way to cross-validate is leave-one-out.


requesting password while git push in jupyter notebook

Use one of the following command:

git push https://username:password@myrepository.biz/file.git --all

git push https://username:password@myrepository.biz/file.git master

Found in:
https://stackoverflow.com/a/29776651/10929995


Why do we determine the values of λ in regularization as ln λ, such as ln λ=-18 instead of for example λ=0.3?

I can just suppose, that it's because then we could consider regularization together with a Log Likelihood function, for instance, and that is why it's more convenient to have such a representation. It's easier to make computations, using log, for example, when we want to minimize some functions (ln a + ln b = ln(a*b), (ln a)' = 1/a etc)


How to measure the information of covariates in a ML task?

There exists a notion of entropy in information theory. It is in fact closely linked to covariance, and especially the covariance matrix determinant (I don't recall the actual mathematical link but I think this should be something that Google can find).

From then, you could use some metrics based on the dataset correlation matrix. You could use its determinant (low determinant means highly correlated data), but it's not always easy to analyse (2 perfectly correlated features will drop the determinant value to 0). The distribution of correlation matrix eigenvalues will surely give more detailed insight (multiple low eigenvalues will mean lots of correlated features).

Another powerful and less mathematical solution would be to compute PCA over the dataset. Give yourself a threshold of explained variance (for instance 95%), and count how many principal components you need to reach this explained variance. A low number of such would mean low "quality" data.


How to setup and run Conda on Google Colab

Found a way to get Miniconda working in Google colab. For now, use source activate, not conda activate in the 2nd cell. Still working out the bugs with using conda to activate.
Full Notebook demo here:
https://donaldsrepo.github.io/Notebooks/GoogleColabCondaCreateEnv.html
github with demo notebook:
https://github.com/donaldsrepo/SampleNotebooks/blob/master/CondaCreateEnv/GoogleColabCondaCreateEnv.ipynb
Google uses Python 3.6.9 by default. I created an environment using Python 3.6.10.
Steps to follow:

Install miniconda
Create a new environment
Run some python code to test the environment is being used

Some sample code:
# try to get the bare minimum to get a new conda env working
conda_path = ''
try:
    conda_path = !which conda
finally:
    print('')

if (len(conda_path) == 0):
    print('installing miniconda')
    !wget https://repo.continuum.io/miniconda/Miniconda3-4.5.4-Linux-x86_64.sh && bash Miniconda3-4.5.4-Linux-x86_64.sh -bfp /usr/local
    !conda update conda -y -q
    !source /usr/local/etc/profile.d/conda.sh
    !conda init
    !conda install -n root _license -y -q
else:
    print('found miniconda')

conda_envs = !conda env list
res = [i for i in conda_envs if 'test36' in i]
if (len(res) == 0):
    print('not found test36 env', len(res))
    !conda create -y -q --name test36 python=3.6
else:
    print('found test36 env', len(res))

Next cell:
%%bash
source activate test36

python
import sys
# maybe only need this the first time we run this notebook
sys.path.append('/usr/local/lib/python3.6/site-packages')

print("Python version")
print(sys.version)

Output:
Python version
3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21)
[GCC 7.3.0]

Note that version 3.6.10 is the one installed in my personal conda environment. You can also see your new environment here:

A few things to point out:

If you open a new notebook, your new environment does not exist. Seems to be stored in a transient location, similar to docker.


What are the implications for training a Tree Ensemble with highly biased datasets?

I would recommend training on more balanced subsets of your data. Training random forest on sets of randomly selected positive example with a similar number of negative samples. In particular if the discriminative features exhibit a lot of variance this will be fairly effective and avoid over-fitting. However in stratification it is important to find balance as over-fitting can become a problem regardless. I would suggest seeing how the model does with the whole data set then progressively increasing the ratio of positive to negative samples approaching an even ratio, and selecting for the one that maximizes your performance metric on some representative hold out data.

This paper seems fairly relevant http://statistics.berkeley.edu/sites/default/files/tech-reports/666.pdf it talks about a weighted Random Forest which more heavily penalizes misclassification of the minority class.


Why don't we use space filling curves for high-dimensional nearest neighbor search?

Preserving "locality" is a desirable property in space-filling curves, especially in this context.  But we cannot hope to actually preserve the ordering of the nearest neighbors for all points.  (The discrete versions of Hilbert's curve put adjacent cells nearly on opposite ends of the curve; in wikipedia's illustrations, see the top-central region.)

This idea might still be useful, as an sort of k approximately-nearest neighbor algorithm; but I'm not convinced that producing an appropriate (discrete version of) a space-filling curve for a dataset and querying new points onto that curve will actually be substantially faster than ball-tree.  Another problem is that the error in the approximations of "nearest" are dependent on the location along the curve, so that which curve you select will have a substantial impact on the correctness of the approximations for different sample points.


Predicting which apps users may be interested in

Broadly speaking, what you want is a recommender system. If you consider only the user-app association data (app installs, likes, etc.), and no user or app metadata, then you should look at collaborative filtering.

More specifically, a simpler technique you can consider is item-similarity based recommendation (users who install app A, also install app B). A slightly more complex method involves factorization of the partial user-app association matrix and then inferring missing points in the matrix. This paper describes one such factorization technique used in the Netflix Prize movie recommendation challenge.


How to apply the gradient of softmax in backprop

After further working on this, I figured out that:

The homework implementation combines softmax with cross entropy loss as a matter of choice, while my choice of keeping softmax separate as an activation function is also valid.
The homework implementation is indeed missing the derivative of softmax for the backprop pass.
The gradient of softmax with respect to its inputs is really the partial of each output with respect to each input:

So for the vector (gradient) form:

Which in my vectorized numpy code is simply:

self.data * (1. - self.data)

Where self.data is the softmax of the input, previously computed from the forward pass.


Pandas: How can I update dataframe values?

df_a = pandas.DataFrame(a)

df_b = pandas.DataFrame(b)

c = pandas.concat([df_a, df_b], ignore_index=True).drop_duplicates(subset=['field'], keep='last')


In CS231n lecture, can't the linear classifier be softmax itself?

The notation they're using is a bit funny. $f$ is just the dot product between the input and weights, $w^Tx_i$. The loss function is what differentiates those classifiers. Each loss function takes that dot product as an input and uses it in different ways. All of those loss functions compare the output of $f$ in some way with an associated $y$ value: you are computing your loss relative to a target variable to understand how well your model is performing, and then you can compute the gradient of the loss function to improve those predictions. If you ignore the loss function, you're just left with linear regression predictions (and even linear regression needs a loss function, MSE, to fit the weights) which aren't being evaluated relative to anything.


Real time topic identification of news article

You are looking for either "online" or "streaming" topic modeling.  A hierarchical Dirichlet processes can automatically choose the number of topics. Here is a paper on streaming/online Latent Dirichlet Allocation model (LDA). A Python implementation of streaming/online can be found here. If your scale warrants it, Apache Spark has an implementation of online LDA also.


Creating bar plot with frequency distribution based

What you are looking for is a histogram. It plots the distribution of a given series of values. So, you can just extract the values that you want to plot from your dataframe and then use matplotlib's histogram function. Here is the code:

import matplotlib.pyplot as plt
to_plot = data.loc[df['Size']==1]['Value']
plt.hist(to_plot)
plt.show()

This is the output I get, note that I haven't done any formatting, this is just the raw output.

The above option is more flexible. But in your case you can also use the following:

import matplotlib.pyplot as plt
to_plot = data.loc[df['Size']==1]['Value']
to_plot.value_counts().plot.bar()


Fashion Clothing images Dataset with image description

You can check Fashion10000 dataset released by Delft University of Technology here


How to implement LSTM using Doc2Vec vectors to get representation?

Inputs of LSTMs are vectors and it does really matter what the vectors are, in signal processing there are signal windows, in NLP, these are usually word embeddings, here, they are document representations obtained from Doc2vec. Judging from the scheme, they probably use Doc2vec to obtain sentence embeddings. It is rather unusual, but definitely possible.

If the input vectors carry enough information, LSTM will certainly learn a reasonable representation.


What are the ways to partition a large file that does not fit into memory so it can later be fed as training data?

Yes, of cause. But, it's insignificant, because Spark and Hadoop are better.

This is my idea. Suppose that your memory can take in 100,000 examples. So splitting your data set to files with size lower than 100,000.

The key and most complex step is how to train classify with those data. Good luck, For Gradient descent series optimization algorithms (GB, SGD and so on), most algorithms (SVM, GBDT, Bayes, LR, deeplearn and so on) support this. You could
 load one file to RAM and fed them to classifier until to find the best parameter.

My code is very simple. Before each iteration, re-shuffling the order of simples and re-splitting data set will boost the classifier.

import numpy as np

X = np.random.random((100, 2))
y = [1 if x[0] > x[1] else 0 for x in X]

from sklearn.linear_model import LogisticRegression

lr_cly = LogisticRegression()

def stop_train(X_s, y_s, threshold):
    scores = [gnb.score(X, y) for X, y in zip(X_s, y_s)]
    return np.mean(scores) > threshold

def iter_train(cly, X, y, threshold=0.99, max_iter=10):
    X_s = [X[:50, :], X[50:, :]]
    y_s = [y[:50], y[50:]]

    iter_times = 0
    while iter_times <= max_iter:
        print "--------------"
        for X, y in zip(X_s, y_s):
            cly.fit(X, y)
            print cly.score(X, y)
        if stop_train(X_s, y_s, threshold):
            break
        iter_times += 1

iter_train(lr_cly, X, y)


Should input images be normalized to -1 to 1 or 0 to 1

As the other answers previously said, in practice it doesn't have much difference which of the two you choose. However, theoretically it's better to scale your input to $[-1, 1]$ than $[0, 1]$ and I'd argue that it's even better to standardize your input (i.e. $μ=0$, $σ=1$).

Let me explain why:

Deep neural networks, especially in their early days, had much trouble with backpropagation, as they suffered from vanishing/exploding gradients. A popular way to combat this is by initializing the weights of the network in smarter ways. Two initialization techniques have proven to be the most popular: Glorot initialization (sometimes referred to as Xavier initialization) and He initialization, both of which are variations of the same idea.

In order to derive their intialization they make a few assumptions, one of which is that the input features have a zero mean. You can read this post for simplified explanation, but I'd suggest reading the two papers mentioned above.


How should I sample from a mixture distribution?

The mixture distribution can be obtained in the following way.
Let $f(x)=w_1p_1(x) + w_2p_2(x) + ... + w_np_n(x)$, where $p_i$ are density functions and $w_i>0$. Note that $f(x)$ is a density function if the sum of all weights is one.
Then, we use the following two-stage process.
Stage 1. Draw a random variable $X$ (selector, if I remember correctly), such that $P(X=i)=w_i$ for $i=1,2,...,n$.
Stage 2. Return a random variable drawn accordingly to $p_X$, where $X$ is the index obtained in the stage 1.


Faster R-CNN wrapper for the number of RPNs in the layer dimensions?

OK so the trick was in the RoI Pooling Layer: it is initialized with dimensions (1,512,7,7) and outputs the tensor (#RoIs,512,7,7), which in turn is an input in a fully-connected layer fc6 that treats the input as batch size #RoIs and hence processes them in the normal way, like it would have processed any batch.


Confusion matrix and ROC AUC curves are not in sync

The ROC curve is built by taking different decision thresholds, and should be built using the predict_proba of your estimator.  In particular, in your multiclass example, the ROC is using the values 0,1,2 as a rank-ordering!  So there are four thresholds, the one between 0 and 1 being the most important here: there, you declare all of the samples the model predicts as being class 0 to be negative, and all others as positive, giving you the complementary point in the ROC-space.

Instead, you should have

fpr,tpr, thres =  roc_curve(y_test, model.predict_proba(x_test)[:,0], pos_label=0)
# or
fpr,tpr, thres =  roc_curve(y_test==0, model.predict_proba(x_test)[:,0])


Keras metrics.binary_crossentropy computes the cross entropy averaged across all inputs (pseudocode):

original_dim = 3
x = [1,1,0]
x_decoded = [0.2393,0.7484,-1.1399]

average_BCE = binary_crossentropy(x, x_decoded)
print(average_BCE)
>>>0.1186

For this part of autoencoder loss we need the sum, not the average over all squared differences between input and output pixels, which is equivalent to  average_crossentropy_of_pixels * num_pixels (original_dim)

print(original_dim * average_BCE)
>>>0.3559

Another way of writing this part would (which I think is more illustrative of what's happening, but probably less performant in Keras land):

xent_loss = K.sum(K.square(x - sigmoid(x_decoded)))
print(xent_loss)
>>>0.3559

Regarding the second part, since the first operation you are really doing is subtraction, it implied the tensors for input and output are the same size. This can be found if you check the implementation code - which is better than the docs in this case - go to line 3056:
https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py


When training accuracy increases while validation accuracy remains constant or decreases, then the model is most likely overfitting, or it may be saturated.

You can try the following methods to increase accuracy:

Momentum
Regularization techniques: Early Stopping, Dropout, Weight Decay, etc
Synthetic Data and related techniques such as SMOTE
Ensembles


Not sure where you are in terms of prior knowledge, but this blog post might get you started.


How to perform feature engineering on unknown features?

You do not need domain knowledge (the knowledge of what your data mean) in order to do feature engineering (finding more expressive ways of framing your data).

As Tu N. explained, you can find "quick and dirty" combinations of features that could be helpful pretty easily. Given an output $y$ and an individual feature $x$, you can take the following transforms, $x' \in \{e^x, \log(x), x^2, x^3, \tanh(x)\}$. A quick check of the usefulness of the transformation is if the correlation between $\{y,x'\}$ is higher than the correlation between $\{y,x\}$.

Warning on correlation: Correlation does not show everything and depending on the model you are using (highly non-linear such as NN or RF) and interaction with other variables, a change in correlation could mean nothing.

However, if you are using a simple linear model like logistic regression, it is an OK indicator of perfomance. The best way to evaluate such a transformation, however, as noted by Fokhruz Zaman, would be to build a model with and without your transformed feature, and see how the validation error (on your Cross-Validation folds) evolves.

It is rather easy to spot single-feature transformations this way. Those apply to a lot of data, where a more expressive relation between your input and output could be on a different scale. For exemple the relationship between Income and "Happiness" appears to be logarithmic, but you would never record the log of a participant income directly.

Finding combinations of feature is more difficult. For a start, if you want to test every addition of 2 features, and you have $D$ features, you have an order of $D^2$ transformations to test. In order to find such transformations, you can apply a nonlinear model (such as NN or RF) to the problem and try to see what it is that it is learning. If you can identify what an intermediate layer in a NN is doing, you can pre-compute its result and add it as a new feature. It will not need to compute it again, and it will probably try to learn something new.

It can be hard to interpret the internal representation of a NN, or even interpret feature importance in a Random Forest. An easier, and probably more suited method for this purpose, model would be Boosting with decision trees. There are a lot of libraries implementing Boosting, and if you are into Kaggle competition as your post seem to imply, XGBoost seems used by a lot of participants, so you might find some help/tutorials on what I'm going to describe.

First, run your boosting algorithm using only stumps, 1-level decision trees. Stumps are very weak, but Boosting makes it a reasonnable model. This will act as your baseline. Depending on the library you are using, you should be able to display pretty easily which are the most used features, and you should plot them against the response (or do an histogram if the response is categorical) to identify some pattern. This might give you an intuition on what would be a good single feature transformation.

Next, run the Boosting algorithm with 2-level decision trees. This model is a lot more complex than the previous one; if two variables taken together have more power than taken individually, this model should outperform your previous one (again, not in term of training error, but on validation error!). Based on this, you should be able to extract which variable are often used together, and this should lead you to potential multi-feature transformations.

On related material, I would advise the following videos as they are easy to follow

Data Agnosticism: Feature Engineering Without Domain Expertise; SciPy 2013 Presentation: An example of how "simple" feature engineering can be.
Peter Prettenhofer - Gradient Boosted Regression Trees in scikit-learn: The speaker touches quicly on analysing feature importance at the end.


Equivariance vs Invariance in Convolutional Neural Networks

Several answers have been provided to the role of equivariance or invariance question "What is the difference between “equivariant to translation” and “invariant to translation”.
Depending on how local they are, scalar and Convolution operators tend to be equivariant, max/min or range are more invariant, and subsampling/pooling can somewhat link those behavior.


How is the linear regression cost function evolved?

What I remember is that they give you more insight in future lectures, but the main reason for that is the maximum likelihood with which you try to increase the chance of making the current data set by setting the parameters. It is a good choice for setting the parameters, but its weakness is that it may overfit to your training data. You may want to take a look at Maximum Likelihood Estimation.

The cost function which is discussed there is called minimum square errors. It is found by maximum likelihood. This means that you want to increase the chance of making your training set. In other words, you want to increase the chance of $P(D|\theta)$ where $D$ can be considered as your training set. Due to the fact that your data should be iid, you can write the previous probability as $\pi p(x_i|\theta)$. You then apply some simplifications and you finally find that cost function. You can take a look at MSE as Maximum Likelihood for exact justification.


What is the best tutorial to quickly learning machine learning in R

There is the Wikibook Data Mining Algorithms In R  and some useful examples at rdatamining.com


How to include user features in a recommender system?

This is referred to as side information. This is used to enhance the recommender system.
A good library for collaborative filtering (and beginner friendly) is turicreate. Have a look at this link. o summarise, the traditional, basic matrix factorisation will encode user i and items j respectively as vectors $u_i$ and $v_j$ so that the predicted score that a user would give to the unseen item is:
$$
score(i,j) = u_i^T  v_j
$$
but you can have a more complex model that will also take into account idiosyncratic characteristics of both items and users:
$$
score(i,j) = u_i^T v_j + a^T x_i + b^T y_j
$$
which increases the capacity of the model.


Is there a difference between np.matrix(np.array([0,0])) and np.matrix([0,0])?

No, they're absolutely the same.

In this case there is absolutely no difference apart from perhaps a trivial amount of processing time.  This is all open source code so we can just read it:

The relevant part of numpy for us here is the matrix constructor (yes, np.matrix is a python class below the hood).  In a summary from the NumPy code we see:

class matrix(N.ndarray):
# ...
    def __new__(subtype, data, dtype=None, copy=True):
    # ...
        if isinstance(data, N.ndarray):
            if dtype is None:
                intype = data.dtype
            else:
                intype = N.dtype(dtype)
            new = data.view(subtype)
            if intype != data.dtype:
                return new.astype(intype)
            if copy: return new.copy()
            else: return new
        # ...
        arr = N.array(data, dtype=dtype, copy=copy)
        ndim = arr.ndim
        shape = arr.shape
        # some extra checks
        ret = N.ndarray.__new__(subtype, shape, arr.dtype,
                                buffer=arr,
                                order=order)
        return ret

What we give as the data argument is literally what we give to to np.matrix().  Therefore we can draw for the two cases:

np.matrix([0, 0])

The python interpreter builds two integers: 0 and 0.
The python interpreter builds a list from the pointers to the two integers.
The python interpreter evaluates the matrix constructor with the list as data.
The if in the constructor is not executed, instead an np.array is build from the list.
Inside the array constructor data types are checked.
The final array is returned (the second array constructor perform much less work because it is passed buffer=)

np.matrix(np.array([0, 0]))

The python interpreter builds two integers: 0 and 0.
The python interpreter builds a list from the pointers to the two integers.
The python interpreter evaluates the array constructor.
The resulting array is passed as data to the matrix constructor, and the if is executed.
Within the if the data type is taken from the existing array.
The array is copied an returned.

Both ways execute pretty much the same number of constructors and lines of code.  One could argue that copying the array (the copy= argument) may be a slow operation.  Yet, given the fact that to have enough data for array.copy() to be slow one would first need to construct a full python list of that size, the copy() time is negligible compared to the list construction.  In other words, both methods need to construct the list - because python will always evaluate arguments before passing them - which is the slowest part of the execution of this code.

As for the return value, they're absolutely and completely the same.  Most of the code within the constructor summarized (and linked) above is to make sure that you get the same return if you give equivalent input.

P.S. (Unrelated Note) If one starts reading data from a file (or any other external source) the picture changes.  If one reads directly into an array without going through the python list phase, that method is bound to be much faster.  The processing bottleneck is the python list, if one can avoid that things will go faster.


Does MLPClassifier (sklearn) support different activations for different layers?

One can see from the code (look at uses of self.activation) that the same function is used for all the hidden layers.
You might want to consider the comments to this question for alternative approaches, generally being a move away from sklearn and towards a deep learning framework.


How to do stepwise regression using sklearn?

Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and scikit-learn deliberately avoids inferential approach to model learning (significance testing etc). Moreover, pure OLS is only one of numerous regression algorithms, and from the scikit-learn point of view it is neither very important, nor one of the best.

There are, however, some pieces of advice for those who still need a good way for feature selection with linear models:

Use inherently sparse models like ElasticNet or Lasso.
Normalize your features with StandardScaler, and then order your features just by model.coef_. For perfectly independent covariates it is equivalent to sorting by p-values. The class sklearn.feature_selection.RFE will do it for you, and RFECV will even evaluate the optimal number of features.
Use an implementation of forward selection by adjusted $R^2$ that works with statsmodels.
Do brute-force forward or backward selection to maximize your favorite metric on cross-validation (it could take approximately quadratic time in number of covariates). A scikit-learn compatible mlxtend package supports this approach for any estimator and any metric.
If you still want vanilla stepwise regression, it is easier to base it on statsmodels, since this package calculates p-values for you. A basic forward-backward selection could look like this:

```

from sklearn.datasets import load_boston
import pandas as pd
import numpy as np
import statsmodels.api as sm

data = load_boston()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

def stepwise_selection(X, y,
                       initial_list=[],
                       threshold_in=0.01,
                       threshold_out = 0.05,
                       verbose=True):
    """ Perform a forward-backward feature selection
    based on p-value from statsmodels.api.OLS
    Arguments:
        X - pandas.DataFrame with candidate features
        y - list-like with the target
        initial_list - list of features to start with (column names of X)
        threshold_in - include a feature if its p-value < threshold_in
        threshold_out - exclude a feature if its p-value > threshold_out
        verbose - whether to print the sequence of inclusions and exclusions
    Returns: list of selected features
    Always set threshold_in < threshold_out to avoid infinite looping.
    See https://en.wikipedia.org/wiki/Stepwise_regression for the details
    """
    included = list(initial_list)
    while True:
        changed=False
        # forward step
        excluded = list(set(X.columns)-set(included))
        new_pval = pd.Series(index=excluded)
        for new_column in excluded:
            model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()
            new_pval[new_column] = model.pvalues[new_column]
        best_pval = new_pval.min()
        if best_pval < threshold_in:
            best_feature = new_pval.argmin()
            included.append(best_feature)
            changed=True
            if verbose:
                print('Add  {:30} with p-value {:.6}'.format(best_feature, best_pval))

        # backward step
        model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()
        # use all coefs except intercept
        pvalues = model.pvalues.iloc[1:]
        worst_pval = pvalues.max() # null if pvalues is empty
        if worst_pval > threshold_out:
            changed=True
            worst_feature = pvalues.argmax()
            included.remove(worst_feature)
            if verbose:
                print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))
        if not changed:
            break
    return included

result = stepwise_selection(X, y)

print('resulting features:')
print(result)

This example would print the following output:

Add  LSTAT                          with p-value 5.0811e-88
Add  RM                             with p-value 3.47226e-27
Add  PTRATIO                        with p-value 1.64466e-14
Add  DIS                            with p-value 1.66847e-05
Add  NOX                            with p-value 5.48815e-08
Add  CHAS                           with p-value 0.000265473
Add  B                              with p-value 0.000771946
Add  ZN                             with p-value 0.00465162
resulting features:
['LSTAT', 'RM', 'PTRATIO', 'DIS', 'NOX', 'CHAS', 'B', 'ZN']


Increasing tolerance will result in "higher root mean squared error value" most of the time. Increasing tolerance is telling the model it is okay to stop earlier with higher error and not continue the search for a possible more optimal solution using smaller updates.


What model are you using?

Ideally, one should always scale / normalize data before feeding them into some ML / statistical model. By using Z-scores you should be able to control for skewness of your variables.


Looking for second opinion on checking that attribute in data frame is unique using r

multiple options are available in R.Apart from your mentioned function.

dplyr

duplicated


Data scaling before PCA: how to deal with categorical values?

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, borrowing from its explanation:

  All your variables are numeric: use principal component analysis (prince.PCA)
  You have a contingency table: use correspondence analysis (prince.CA)
  You have more than 2 variables and they are all categorical: use multiple correspondence analysis (prince.MCA)
  You have groups of categorical or numerical variables: use multiple factor analysis (prince.MFA)
  You have both categorical and numerical variables: use factor analysis of mixed data (prince.FAMD)

Check also this question/answers in stats.stackexchange or discussion in researchgate.


We are trying to select the optimal $c$, here $d$ is fixed and hence $P(d)$ and $\frac1{P(d)}$ is just a positive constant.

Multiplying an objective function with a positive constant doesn't change the optimal solution, hence we can drop $P(d)$.


TypeError: float() argument must be a string or a number, not 'function'

a generic way to convert pandas column to numeric dtype:

df['col_name'] = pd.to_numeric(df['col_name'], errors='coerce')

this will replace all values, that couldn't be converted to numeric values with NaN's (Not a Number).

PS you may want to analyze how and why do you have a reference to a lambda function in your cells. Fixing the source of that problem would be a more appropriate solution.


Keras load pre-trained weights. Shape mismatch

I have solved the kind of issue as follows. Hope the solution would be helpful.
1. Delete "by_name=True"
# -model.load_weights(weights_path, by_name=True)
model.load_weights(weights_path)

2. Change the number of class
While it throws another throwing "ValueError: Shapes (1536, 1000) and (1536, 1001) are incompatible", I change num_classes from 1000 to 1001. And then it shows the correct model summary.
# -num_classes = 1000
num_classes = 1001


To clarify, I feel like the original question references by OP probably isn't be best for a SO-type format, but I will certainly represent python in this particular case.

Let me just start by saying that regardless of your data size, python shouldn't be your limiting factor. In fact, there are just a couple main issues that you're going to run into dealing with large datasets:

Reading data into memory - This is by far the most common issue faced in the world of big data. Basically, you can't read in more data than you have memory (RAM) for. The best way to fix this is by making atomic operations on your data instead of trying to read everything in at once.
Storing data - This is actually just another form of the earlier issue, by the time to get up to about 1TB, you start having to look elsewhere for storage. AWS S3 is the most common resource, and python has the fantastic boto library to facilitate leading with large pieces of data.
Network latency - Moving data around between different services is going to be your bottleneck. There's not a huge amount you can do to fix this, other than trying to pick co-located resources and plugging into the wall.


logistic regression algorithm fails to work

Consider that you are doing vector operation, change your cost function to the following:

(1 / m) * sum(((-y) .* (log(h)) - ((1 - y) .* log((1-h)))));

and your gradient to the following:

grad = (1./m) * (x' * (h - y))

Although the latter is just for precedence reassuring.

Based on the discussion in the chat, although the code calculates the cost in a wrong way, the reason the cost does not decrease is that the data is not linearly separable. Logistic regression is a simple algorithm which classifies successfully linearly separable data. Take a look at here.


It is possible.

It is called Multi-output regression. Allows you to predict more that one variable:

Multioutput regression


StanfordTokenizer will be deprecated in version 3.2.5 Warning

@imoutidi, I also encountered the same deprecation warning.

After digging around a bit, it looks like the new/replacement package can be imported with the following:

from nltk.tag.stanford import CoreNLPNERTagger

However, when trying to run the tag() method I end up getting an unexpected HTTP connection refused error. I haven't figured out if this is an environment configuration issue (missing .jar path) or an issue with the NLTK codebase. I tried temporarily turning off my local firewall and that didn't help, so I'm guessing it is one of the other two possibilities.

I hope the python import statement is at least some help.


How to label "other" while labeling image for object detection/classification?

This highly depends on your test data. Suppose you have trained your data which all contains some kind of food. If you give it a hand at test time, it will try to find the most similarity it has with the current labels and outputs the one which is more similar than the others. The point is that your test environment is really important. If you are going to test your app near the sea, your training data set should contain data on that condition. If you are going to use it during the night, your dataset should be taken during the night. In your case, you have a simple task, for the current dataset add an entry with a zero which means you are trying to add a new class. After that, try to add new pictures that are not images of food and add one in the new entry for these images. But consider this important point. The number of images in each class should have the same distribution as the test data.


For gradient descent, batch updates calculate the weights based on all examples. This process continues until convergence which is typically measured by a derivative of zero. The resulting weights might end up in a local minimum or saddle point because both have a derivative of zero.

Stochastic gradient descent takes a random sampling of examples, introducing randomness into the updates. This randomness increases the possibilities of escaping a local minimum or saddle point. The updates might randomly overestimate the area with a derivative of zero and find a new section of the error surface that leads to a better optimum.


Learning with dirichlet prior - probabilistic graphical models exercise

Your formula is correct, but the final computing is wrong.
It should be: $\frac{10+87}{270+2000} \times \frac{10+100}{270+2001}$


Is there any way to create column based on some previous column in PANDAS dataframe?

Let's call the dataframe "manager" with column names "Emp" and "Man". First you get the people who are always head mananagers:

head_man = set(manager["Man"]) - set(manager["Emp"])

This returns

{'E001', 'E241'}

and then use the following function to find the head managers of the rest:

def manager_of(emp):
  if emp in head_man:
    return emp
  else:
    return manager_of(manager[manager["Emp"] == emp]["Man"].values[0])

Complete Code:

import pandas as pd

head_man = set(hrms_data["Manager ID"]) - set(hrms_data["Employee ID"])

def manager_of(emp):
  if emp in head_man:
    return emp
  else:
    return manager_of(hrms_data[hrms_data["Employee ID"] == emp]["Manager ID"].values[0])

for index, row in hrms_data.iterrows():
  hrms_data.loc[index,"Head of Manager"] = manager_of(row["Employee ID"])

Edit: hrms_data["Head of Manager"] = hrms_data["Employee ID"].map(manager_of) instead of the for loop is better.


I guess it is more about "What would be the most informative (and easy readable) way to visualize one specific problem?". Having a specific question in mind you can explore galleries of the libraries with their examples, for instance,  seaborn. You could also explore another kernels to see how people visualized it. For some question there are pretty trivial solutions. Other questions can be visualized in very many different ways but what counts is how readable and informative your plot is.

In case of titanic imagine three questions that you want to visualize.

What if survival rate depends somehow on the age and/or sex?

For this you might want to choose a density or histogram with having two curves on the plot: for survival and non-survival. Maybe you want to split the plot into two plots for female and male.

What if survival rate depends on a Pclass? Or maybe Emarked as well? And maybe sex plays also an important role?

I am a fan of factor plots as you can introduce lots of information into it and it is quite understandable from the first sight. So, you see that 1st and 2nd class had a higher survival rate comparing to the 3rd class. (The next question can arise: For each embark what is the relative proportion of the pclass? What would be the better visualization?)

What if a survival rate depends on the pclass and age? Additionally you could see what is the age distribution of each class.

Violin plots might be good for that.


Confusion about Decoder labels for training seq-to-seq models

It can be both.

If you input the desired label and predict the next desired label, it's called teacher forcing.

But using only this technique might hurt the performance at test time. So using the actual predicted output from the last time step is also a good idea.

It's possible to do both :
For each batch, with X% chance you use teacher forcing, otherwise you don't.


Why Generator error increases after some epoch

In your question, it's pretty hard to answer specifically what is going on as you don't share any details of what you're trying to do. However, I'll try to address the general case:

In GANs (and other adversarial models) an increase of the loss functions on the generative architecture could be considered preferable because it would be consistent with the discriminator being better at discriminating.

As the discriminator is trained, it actually changes the loss landscape of the generator, which in turn increases the loss of said generator. This is probably why.

Another reason could be that the discriminator is getting too good and the generator is getting stuck, which often can be fixed by adjusting learning rates in the optimizers or increase the number of iterations of the adversarials. In this particular case I'd try decreasing the discriminator learning rate or increase the number of steps the generator is trained.

I'd suggest reading the original Generative Adversarial Nets paper and perhaps also the Improved Techniques for Training GANs paper.


Semantic Segmentation using Keras: loss function and mask

let me see if I can help.

(1) I would definitely recommend binary crossentropy for your loss function.

(2) Your labels should be "masks", which are images (the same size as your input images) where your "0-class" pixels are 0's and your "1-class" pixels are 1's.  This is basically a black and white image where black and white represent the 2 different classes. The output of the UNET network will be a single channel image (the same dimensions as the input) where your "0-class" is colored black and your "1-class" is colored white.

Here is an example:

The output mask in this image is what your label ("mask") should look like.
If you train your network right, it should also be the output of UNET.  Train for about 40 epochs and you will start to see results.


is it wrong to use average='weighted' when having only 2 classes?

As you already know, a precision score (or recall, or f-score) is for a single class, and in the function the argument pos_label says which class.
Now I'm going to guess that when pos_label is not provided and instead average is provided the function probably calculates the metric for every class and then returns the average of these values.
A weighted average can be calculated with any number of classes, and since no pre-defined weights are provided we can reasonably assume that the function takes the proportion of the two classes as weights. So the result of:
metrics.recall_score(test_sentiments,predicted_sentiments,average='weighted')

is probably the weigthed average (by proportion of instances) of:
metrics.recall_score(test_sentiments,predicted_sentiments,pos_label='positive')
metrics.recall_score(test_sentiments,predicted_sentiments,pos_label='negative')

If I'm not mistaken, this is equivalent to the micro-average recall.


Extracting popular keyword terms or topics from blog posts based on post usage

Seems right. However, establishing causality will not be as simple as extracting keywords and noticing the differences. And I would suggest not to divide the posts, instead club them together run LDA, extract the keywords, then analyse the differences. By separating you are introducing quite a huge bias into your model.


How do data scientists test the claimed accuracy of a model?

There will already be a historical dataset of Fraudulent transactions (E.g.: If we take Credit Card as an example, all transactions that were disputed and dispute was accepted).

Sales team will include pre-sales developers that can help run a subset of such data through model (After various paperwork such as NDA, Legal review that a specific attribute like Gender/ Race can be used for this purpose). This will usually require :

Extracting data from source systems or warehouse or data lake
Transforming data for the model's input/output format
Quantitative + Subjective validation by Operations team that handles such transactions

Getting 90% accuracy is pretty straight-forward. Since most of the transactions are not fraudulent, any model that predicts all transactions as genuine will meet this bar.

So step #3 (validation) will include agreement on appropriate measure for validation of the model (Such as F1 score)


Confusion Matrix - Get Items FP/FN/TP/TN - Python

Considering you have two lists y_actual and y_pred ( I assume you made a typo error on x_test and x_pred as in your code), you can pass the two lists to this function to parse them

def perf_measure(y_actual, y_pred):
    TP = 0
    FP = 0
    TN = 0
    FN = 0

    for i in range(len(y_pred)):
        if y_actual[i]==y_pred[i]==1:
           TP += 1
        if y_pred[i]==1 and y_actual[i]!=y_pred[i]:
           FP += 1
        if y_actual[i]==y_pred[i]==0:
           TN += 1
        if y_pred[i]==0 and y_actual[i]!=y_pred[i]:
           FN += 1

return(TP, FP, TN, FN)

Alternatively, if confusion matrix is a 2x2 matrix (named cm), you can use

TP = cm[0][0]
FP = cm[0][1]
FN = cm[1][0]
TN = cm[1][1]


Why does logistic regression in Spark and R return different models for the same data?

A quick glance at the docs for LogisticRegressionWithLBFGS indicates that it uses feature scaling and L2-Regularization by default. I suspect that R's glm is returning a maximum likelihood estimate of the model while Spark's LogisticRegressionWithLBFGS is returning a regularized model estimate. Note how the estimated model weights of the Spark model are all smaller in magnitude than those in the R model.

I'm not sure whether or not glm in R is implementing feature scaling, but this would also contribute to different model values.


unique predictions for "multi-label multi-output" classification task

2 Questions.

I think this prediction task is a multi-label multi-output classification, right?

Yes. But you could consider splitting these joint classification tasks into seperate ones (i.e. for swimming whats the placement of A,B,C,D etc) IF you data allows it.

Are there any algorithms, which provide unique forecasts (for each class)?

Think in terms of probabilities. The strongest (most discriminatory one) is the one that has the highest-not-so-much-overlapping probabilities to different outputs. So taking these probabilities and "rounding" (possibly forcing some predictions if model is highly uncertain---meaning, for example, that he predicts 25% for every placement, here some other heuristics should be used. For example create a new variable that says how did competitor fare in other competitions etc) that up will give unique classes.


Evaluating recommendations quality and accuracy

Unraveling the causality involved in this type of recommender system is complicated, but tractable.

Given that you have a good experimental methodology, and your evaluation numbers are generated on numbers not used for training (among other concerns), it sounds like you have the best estimate that you can get (or are in the neighborhood) using past data.

The next step would be to form a new experiment, in which you (ideally) randomly provide the recommendation for one group and not provide it for another group, which otherwise are the same. You then measure the outcomes for both groups and compare them statistically to determine the presence of an effect and its size.

Proposing that study is your next step if you're in an organization, or performing it if you have the agency to do it on your own.


How to create a custom loss function from sklearn metrics in Keras?

Use the cross-entropy loss function.  That is already built into Keras, and while you haven't defined exactly what you mean by "mutual information metric", if it's what I suspect you mean, it is equivalent to the cross-entropy loss.


On the top of my head:

with regular clustering techniques you could try to use text-specific distance/similarity measures instead of only considering distinct words as elements. There are hybrid string similarity measures such as SoftTFIDF which take into account character-based and word-based similarity.
use lemmas instead of words in order to facilitate matching of the same concept
for more specific NLP methods you could look at topic modeling and/or word sense induction techniques. The two follow similar ideas in different ways: the former is closer semantically to your case, the latter is meant to work with small context windows similar to your short descriptions size. I'm not sure what is the state of the art nowadays but Latent Semantic Analysis was the standard not so long ago. Afaik these techniques are meant for a quite high amount of data, but they might be worth trying.


Is PCA considered a machine learning algorithm

PCA is actually just a rotation. Seriously, that's all: it's a clever way to spin the data around onto a new basis. This basis has properties that make it useful as a pre-processing step for several procedures.

The basis is orthonormal. This is incredibly useful if your features exhibit multicolinearity (two or more features are linearly dependent): applying PCA is guaranteed to give you a basis where this is no longer a problem. This procedure is known as principal component regression
The basis vectors are meaningful with respect to the spread of the data: they are the eigenvectors of the covariance matrix. This second property gives rise to PCA's famous utility as a dimensionality reduction technique: after rotating the data, projecting the data onto a subset of the basis vectors associated with a significant portion of the total variance yields a lower dimensional representation that (often) retains (most of) the (interesting) structural properties of the data.

So: is it a learning algorithm? This is sort of a philosophical question. What makes something a learning algorithm? Certainly PCA isn't a "supervised" learning algorithm since we can do it with or without a target variable, and we generally associate "unsupervised" techniques with clustering.

Yes, PCA is a preprocessing procedure. But before you write it off completely as not "learning" something, I'd like you to consider the following: PCA can be calculated by literally taking the eigenvectors of the covariance matrix, but this is not how it's generally done in practice. A numerically equivalent and more computationally efficient procedure is to just take the SVD of the data. Therefore, PCA is just a specific application of SVD, so asking if PCA is a learning algorithm is really asking if SVD is a learning algorithm.

Now, although you may feel comfortable writing off PCA as not a learning algorithm, here's why you should be less comfortable doing the same with SVD: it is a surprisingly powerful method for topic modeling and collaborative filtering. The properties of SVD that make it useful for these applications are exactly the same properties that make it useful for dimensionality reduction (i.e. PCA).

SVD is a generalization of the eigendecomposition, and that too is extremely powerful even as a constrained version of SVD. You can perform community detection on a graph by looking at the eigenvectors of the adjacency matrix, or determine the steady-state probabilities of a markov model by looking at the eigenvectors of the transition matrix, which coincidentally is also essentially how PageRank is calculated.

Under the hood, PCA is performing a simple linear algebra operation. But, this is exactly the same operation that underlies a lot of applications that most people wouldn't question applying the label "machine learning" to. This class of algorithms is called Matrix Factorization, and even extends to sophisticated techniques like word2vec: indeed, you can actually get word2vec-like results by literally just applying PCA to a word co-occrrence matrix. Generalizing again, another word for the results of PCA is an embedding. Word2vec is probably the most famous example of an embedding, but constructing embeddings (as intermediaries) is also an important component of the encoder-decoder architecture used in RNNs and GANs, which are the bleeding edge of ML research right now.

So back to your question: is PCA a "machine learning algorithm?" Well, if it's not, you should be prepared to say the same about collaborative filtering, topic modeling, community detection, network centrality, and embedding models as well.

Just because it's simple linear algebra doesn't mean it isn't also magic.


Is there a mistake in Lecture 5 of Stanford CS234 available on youtube?

It's the notation that might be a bit confusing. Take a look at David Silver slides: pages 10-15. He has a complete derivation. Do not forget that the term $r + \gamma V(s';w)$ is the target. She mentions that in the video and the fact that you are actually doing supervised learning with target provided by a bootstrapped value (you try to minimize the Bellman error).
In other words: you are trying to minimize the error between a target value and the estimation from your model. You do not know the target value so you estimate it with bootstrapping. Then you have $SE=(y - \hat{y}(w))^2$. The target y is considered known (as in supervised learning) so eventually you are searching for the weights $w$ that will make the output $\hat{y}$ of your model close to y. What I wrote here applies for batch training (minimizing the mean square error MSE).
So eventually yes the quantity $V(s';w)$ is considered constant and will have derivative 0.


Extract key phrases from a single document

A related keyword to your case can be Single Document Keyword Extraction. A good paper about this is:

  We present a new keyword extraction algorithm that
  applies to a single document without using a corpus.
  Frequent terms are extracted first, then a set of cooccurrence
  between each term and the frequent terms,
  i.e., occurrences in the same sentences, is generated.
  Co-occurrence distribution shows importance of a term
  in the document as follows. If probability distribution of
  co-occurrence between term a and the frequent terms is
  biased to a particular subset of frequent terms, then term
  a is likely to be a keyword. The degree of biases of distribution
  is measured by the $\chi^2$-measure. Our algorithm
  shows comparable performance to tfidf without using a
  corpus.

You can find the paper here.

In sum, this paper gives a rank on keywords based on the defined $\chi^2$-measure.


Is data augmentation changing the train/test sets distribution?

I would argue that augmentation does not alter your train/test sets distribution.

I say this because I consider data augmentation to be part of your training pipeline and not part of your train set.

The pipeline below is one that I use often when someone asks about data augmentation. At the moment the data is split into train/test(/validation) sets, each set should have the same, or at least very similar distributions. This is the moment you need to worry about by picking a good splitting strategy.

Then, whatever you do with the train set, including data augmentation is part of your training pipeline. You will do data augmentation, like mentioned by others, do improve you model robustness. The validation and test set, do not need to be passed through your augmentation process.

          +-> training set ---> data augmentation --+
          |                                         |
          |                                         +-> model training --+
          |                                         |                    |
all data -+-> validation set -----------------------+                    |
          |                                                              +-> model testing
          |                                                              |
          |                                                              |
          +-> test set --------------------------------------------------+


Tidying Time Intervals for Plotting a Histogram in R

This was a x-post. There is a great solution over here, where I originally asked the question.

  If I understand your problem, you'll want something like this:

bottom <- seq(1750, 2010, 5)
library(dplyr)
new_df <- mlt.mx.info %>%
  arrange(Year) %>%
  mutate(year2 = as.numeric(substr(Year, 6, 9))) %>%
  mutate(new_year = paste0(bottom[findInterval(year2, bottom)], "-",(bottom[findInterval(year2, bottom) + 1] - 1)))
View(new_df)

  So what this does, it creates bins, and outputs a new column
  (new_year) that is the bottom of the bin. So everything from 1750-1754
  will correspond to a new value of 1750-1754 (in string form; the
  original is an integer type, not sure how to fix that). Does this do
  what you want? Double check the results, but it looks right to me.
  @goodtimeslim


They are defined that way. Have a look a the paper cited in the docstring. The sse is defined in eq. 3, and the scSE is defined in the text in 2.3.


Tools for project management Data Science

Data science project planning is carried out through any of the scrum application such as Jira, Asana or even using Github projects. Gitlab also offers efficient tools for the same. Once you are done with planning, you can use open-source projects such as Git-lfs and DVC to version control your datasets. The notebooks that you prepare could also be version controlled in Github or Gitlab. You can easily find courses and tutorials to dynamically set up your boards and workflows in Github as well as Gitlab.


Deep Learning for high contrast images with small differences

I have solved my problem! I found a relevant article which describes an architecture suitable for my problem. I have changed it a bit however. The article: https://www.researchgate.net/publication/312328148_Binary_shape_classification_using_Convolutional_Neural_Networks.

The model in keras:

modelBS = Sequential()

modelBS.add(Conv2D(150, (3, 3), input_shape=(ImgSize,ImgSize,1), activation='relu', padding='same'))
modelBS.add(MaxPooling2D(pool_size=(5, 5), strides=(5, 5)))

modelBS.add(Conv2D(150, (5, 5), activation='relu', padding='valid'))
modelBS.add(Conv2D(150, (2, 2), activation='relu', padding='valid'))

modelBS.add(Flatten())
modelBS.add(Dense(rp, activation='softmax'))


TL;DR

Yes, with overfitting all data becomes (non-linearly) separable (as long as the points don't precisely overlap).

Explanation

The problem with your argument is that you are using circles on a 2D plane, which is very difficult to learn. However, I think your argument can be made stronger with a decision-tree.

(0.2, 3.1)? --> yes -> star
            \-> no  -> (1.2, 4.5)? --> yes -> circle
                                   \-> no  -> (x1, x2)? --> yes ...
                                                        \-> no  ...

Decision trees are well accepted models, but note that they are non-linear models. With that, it is easy to argue that all data becomes separable.

However, the issue lies with overfitting. Because models like that become unstable with data points that were unseen before. So, just because the training data is separable, it doesn't mean that the models generated from it become any useful.


Guidance needed with dimension reduction for clustering - some numerical, lots of categorical data

There are many ways to get rid of redundant dimensions. The choice wheter to do it or not depends on what kind of problem you want so solve and what kind of algorithm you plan to choose.


We usually say a model is overfitting, not a layer or a neuron. Basically, there is not a standard way to set the number of convolutional layers. Iff you see your model is not learning well enough, increase the number of filters. Iff it is overfitting, decrease the number of filters or use the common approaches like drop out to overcome the problem. There are standard ways for interpreting the number of neurons is dense layers. Take a look at How to set the number of neurons and layers in neural networks.

If by activation normalisation you mean batch normalisation, there is an answer. Batch normalisation is a technique for avoiding covariat shift. It does not allow the distribution of activations to change. It is needed for very deep networks and it does help. If you have a very deep network, use it and you will see that it is very helpful most of the time. Simple networks may be able not to have them. These are usual facts but you may be able not to use them in dense layers, something like AlexNet, and still have a good result.


How should I retrain my neural network with new images?

I guess you can try both the solutions.

First try fine-tuning. Then, if the results aren't improving, retrain with all the data.


Neural network for variable length data classification

Each input sequence should be padded to the same length. The most common method is to find the longest sequence and then add zeros to all shorter sequences. Most Deep Learning frameworks will have a built-in function to do this.

Consistently sized-input data allows common neural networks models (e.g., Convolutional Neural Network and Recurrent Neural Network) to be fit in a straight-forward manner.


Discover open-source deep learning code and pretrained models at Model Zoo

These are pre-trained sources available in the Github.

Handwritten Text Recognition with TensorFlow

More
Handwriting OCR
Handwritten Text Recognition (OCR) with MXNet Gluon

Some Helpful Resources:

Handwriting recognition and language modeling with MXNet Gluon
Handwriting OCR: Line segmentation with Gluon


How do I obtain the weight and variance of a k-means cluster?

It is valid to use the k-means to initialize the EM for Mixture of Gaussian modeling. As you said, the mean of each component will be the average of all samples belong to the same cluster (it depends on the used clustering algorithm, some times the centroid is not the average of the cluster but is one of the samples). for the weight you can use the following: the weight of cluster x = the number of samples belong to cluster x divided by the total number of samples. thus, the cluster with the highest number of samples is the cluster with the highest weight. for the variance: just find the variance of all samples belong to the same cluster.


Balancing XGboost still skews towards the majority class

That is completely normal. You should remember that the model will basically learn a statistical function given by your data (Intuitively), and since your data is skewed, it will learn by the majority class.

To overcome that, you can treat the imbalance characteristic of data set using two types of approaches: sampling and cost-sensitive learning. Basically, in sampling techniques we have undersampling and oversampling, that is, we can reduce our data set to balance data or increase it with artificial methods. Some examples:

Random Oversampling: essentially, this technique increases minority class by randomly sampling it.
SMOTE: random oversampling has a overfitting problem as we are just replicating data considerably. To overcome that, smote works, in a simplified way, interpolating points on the feature space for nearest neighbors in minority classes.

For undersampling techniques:

Random Undersampling: this method simple undersample the majority class. We can loss information, but if points are close, it can give good results
NearMiss: In a simplified way, this model selects the closest points to the minority class.

In cost-sensitive learning techniques, we use cost matrix 'punishing' models that give wrongs score to a specific class. For example, let's say you want to give better results for class C1, you can create a matrix where the misclassification of C1 is higher than other classes.

In python we have two excellents libraries to deal with this problems:

https://imbalanced-learn.readthedocs.io/en/stable/api.html
https://wwwen.uni.lu/snt/research/sigcom/computer_vision_lab/costcla_a_cost_sensitive_classification_library_in_python

They have excellent tutorials on the topic so you can get a deeper overview of methods discussed and others ones. I hope it helps.


But in policy iteration also we are have to output a softmax vector related to each actions

This is not strictly true. A softmax vector is one possible way to represent a policy, and works for discrete action spaces. The difference between policy gradient and value function approaches here is in how you use the output. For a value function you would find the maximum output, and choose that (perhaps $\epsilon$-greedily), and it should be an estimate of the value of taking that action. For a policy function, you would use the output as probability to choose each action, and you do not know the value of taking that action.

  So I don't understand how this can use to work with continuous action space ?

With policy gradient methods, the policy can be any function of your parameters $\theta$ which:

Outputs a probability distribution
Can be differentiated with respect to $\theta$

So for instance your policy function can be

$$\pi_{\theta}(s) = \mathcal{N}(\mu(s,\theta), \sigma(s,\theta))$$

where $\mu$ and $\sigma$ can be functions you implement with e.g. a neural network. The output of the network is a description of the Normal distribution for the action value $a$ given a state value $s$. The policy requires you to sample from the normal distribution defined by those values (the NN doesn't do that sampling, you typically have to add that in code).

  Why are policy gradient methods preferred over value function approximation in continuous action domains?

Whilst it is still possible to estimate the value of a state/action pair in a continuous action space, this does not help you choose an action. Consider how you might implement an $\epsilon$-greedy policy using action value approximation: It would require performing an optimisation over the action space for each and every action choice, in order to find the estimated optimal action. This is possible, but likely to be very slow/inefficient (also there is a risk of finding local maximum).

Working directly with policies that emit probability distributions can avoid this problem, provided those distributions are easy to sample from. Hence you will often see things like policies that control parameters of the Normal distribution or similar, because it is known how to easily sample from those distributions.


Pandas dataframe with multiple observations per model

# create the df
df = pd.DataFrame({'model':['our','theirs'],'metric':['acc','sen'],'value':[np.array([1,2,3]),np.array([4,5,6])]})
# unzip the array into three columns
df[['ex1','ex2','ex3']]=pd.DataFrame(df['value'].to_list(),index=df.index)
# melt df to long df
df.melt(id_vars=['model','metric'],value_vars=['ex1','ex2','ex3'])

You may need to sort the dataframe as you like.


How to formulate reward of an rl agent with two objectives

In general it is not possible to simultaneously optimise two separate objective functions. Your approach of adding weights (your coefficients) to each objective, then summing the scaled objectives, is a standard way of resolving that.

As your penalties are on different scales and in different units, it is your task as the engineer setting the objective to provide the conversion to a single scale. That is what the coefficients represent - you can even think of them as $points/Joule$ for energy and $points/(\Delta K)$ for temperature difference.

Sometimes analysis will show you that there is a natural combined scale. For instance, in business settings it may be possible to frame compromises as financial costs, e.g. your coefficients might be $\text{GBP}/Joule$ for energy and $\text{GBP}/(\Delta K)$ for temperature difference. Then you have one clear objective to minimise cost or maximise profit.

If that is not possible - a financial cost for exceeding temperature bounds may be hard if this is about human comfort in a building - deeper analysis might lead to thinking about longer-term outcomes. Perhaps your initial rewards are too focused on immediate numerical issues (that appear easy to collect, but don't represent your true goals), and a re-framing of the problem could work. For instance, perhaps it is more reasonable that both the temperature and energy costs stay within strict bounds over a year of varying external temperatures and system workload, with a scaling penalty depending on how badly these are exceeded.


How can I avoid requiring global information for performing regression on meter variables?

Your "missing value" formula mentions $v_{i-1}$, and you seem to be suggesting that you'd prefer not to use that. Ok, fair enough. So use an estimate, $\hat{v}_{i-1}$, which is based on weighted regression of $v_{i-1}$ and several preceding values. Give greater weights to more recent values.

You may wish to implement a classifier that labels each reading as "good" or "bogus". Part of that is straightforward, as you already have a "missing" label for some of your samples. The classifier would additionally discard readings that swing implausibly high, or that show physically impossible behavior like having negative derivative.

You don't mention the generative source of bogus readings. If they are due to noise events on the digital channel used for obtaining each reading, then they don't affect the data source and will soon average out. Here is another technique. Suppose you're willing to suffer higher readout latency in exchange for numeric accuracy. Delaying by one sample would let you compute $\hat{v}_{i}$ as median of three values: $v_{i-1}$, $v_{i}$, $v_{i+1}$. Similarly, you could delay by two and find median of five values, or by three with median of seven values.


Can 2 different OOV words get the same vector in FastText?

TL;DR
Theoretically it is possible, but it is unlikely.
1) Uncommon subwords
word1 = 'iiii'
word2 = 'jjjj'

word1_subwords = ['<ii', 'iii', 'iii', 'ii>']
word2_subwords = ['<jj', 'jjj', 'jjj', 'jj>']

In this example, there are basically 6 subwords: ['<ii', '<jj', 'iii', 'jjj', 'ii>', 'jj>'], but these are not common subwords in general. So, there is a possibility that the embedding for all the subwords is the same (e.g. [0,0,...,0,0]), making their sum all the same.
2) Homographs
word1 = 'lie' # meaning: tell something untruthful
word2 = 'lie' # meaning: to rest on a horizontal position

In this example, there are two homograph words. These are different words but they have the same spelling. Since FastText only take syntax into account, they will have the same subword embedding sum.


Two steps optimization of a credit card limit

You can model it as a binary logistic regression. The target is the probability of default. One of the features is credit card limit.
Logistic regression allows an odds ratio to be calculated. The odd ratio will give the effect of increasing credit card limit on wether a default is predicted to happen.
Logistic regression allows a threshold to be set to "minimize the probability of default".


NLP varying amount of features and BoW as feature concatenating to feedforward NN

Let me take a crack at your questions:

The article specifies the features are concatenated. How does a concatenation layer work internally? Does it concatenate all the values in a single variable, in a very literal sense? how does that work computationally?

The concantenation of information in this context is, concatenation of vector represententations of the text. You could concatenate using a concatenation layer as described here. This is a very common approach followed where you want to feed your network information by taking various contexts depending on the problem you need to solve.

How can a Bag of Words be a feature, when its a key-value pair? Or is it also just all concatenated into one variable. Which again, how can that work computationally?

Bag of words, is typically a vector representation of the context. The above answer should help you. You could take a look at how to combine embeddings here as well

The text specifies multiple words are used as a single feature; e.g. Left context: five words before the entity. Is this again concatenating the embedding / vectors?

Yes, you are spot on. You combine the embedding vectors that you generate using skipgram or cbow, or you could even use one-hot encoded vectors.

Entity end: last three words of the entity (they can be duplicated with the previous feature if they overlap, or padded if there are not that many) does this mean a variable amount of features as input to the NN (or concatenation layer) or is this more intended as a configuration? Less context available so fewer amount of 'hard' coded input features?

Its always good to pad and use well defined dimensional vectors. Helps you structure your architecture better. You could use masking layer to ensure that the network ignores the paddings.

I hope this helps.


How can Time Series Analysis be done with Categorical Variables

By definition time-series ARIMA models assume that, given a numerical observation at time $t-k$, the value of the numerical variable $X$ at time $t$ can be approximated as
$$
X_t = \sum_{j=1}^p a_j X_{t-j} + \varepsilon_t + c
$$
where $\varepsilon$ is a white noise error term and the $a_j$ are parameters to be determined. The idea is that the numerical variable $X$ at time $t$ only depends on some of its values at previous times; as you can see, by construction the above works for numerical variables only. Then one introduces some more conditions about moving averages and deviations to be verified and is able to prove that, under such conditions, the form of the coefficients $a_j$ can be determined.

The standard way to deal with categorical variables in these cases is to use one-hot encoding, namely you introduce dummy variables for each level of your category and fit against the dummy being 1 or 0, according to whether such category is present or not at time $t-k$. A similar question was asked here and you might want to have a look.

Another thing that you may want to ask yourself is whether time series is really what you are looking for, rather than just any classification model that provides a prediction given a set of categories and a seasonal variables.


Variance in statistics vs machine learning

Variance in machine learning appears in many places, but in all cases it is of course simply an application of the mathematical definition. For instance, people often worry about the variance of stochastic gradients $\mathbb{V}[\nabla \mathcal{L}]$ (especially when using score function/likelihood ratio gradient estimators), which refers to how much the gradient (treated as a random variable) varies, either over time OR as a function of the minibatch. If you took the same model, with many different minibatches, ideally you'd hope that the variance of the resulting gradient estimate would be low (sometimes true with large enough batch sizes).

For the specific case you are talking about

  variance is a measure of learning the training data too well/capturing the noise in the data/oversensitivity to the small local fluctuations of the data

This is the variance in model performance with respect to changes in the training data.

Here's a simple way to look at this. Let $\mathcal{D}=(\mathcal{X},\mathcal{Y})$ be the space of possible datasets and $P_D$ some distribution on that space (so $D\sim P_D$ where $D=(X,Y)\in\mathcal{D} $ is a random training dataset).
A specific training example is $(x,y)\in D$ with $x\in {X}$ and $y\in {Y}$.
We write $x,y\sim D$ to mean $D\sim P_D$ and $x,y\sim\mathcal{U}(D)$ where $\mathcal{U}$ denotes the discrete uniform distribution.
Now, let $f_\theta : X \rightarrow Y $ be a learned function parametrized by $\theta\in \Theta$.
Suppose $\mathcal{L}: \Theta\times \mathcal{D}\rightarrow \mathbb{R}$ is some loss function defined on the space of learned functions. Note that it is dependent on the dataset! For example, one possible definition of $\mathcal{L}$ could be
$$ \widetilde{\mathcal{L}}(\theta,D) = \frac{1}{|D|}\sum_{(x,y)\in D} L_s(f_\theta(x), y) $$
where $L_s : Y\times Y\rightarrow\mathbb{R}$ is some loss function for specific examples. This is one possible form, but a specific one is not necessary.

What is the variance then? We can consider $\mathcal{L}$ as a (random) function of $D$ by first assuming that we can compute $\theta$ given $D$, presumably by some stochastic training algorithm $A$, so that $\theta \sim A(D)$. Presumably, this looks something like stochastic gradient descent. Then our expected loss is
$$ \mu_\mathcal{L} =
\mathbb{E}_{D\sim P_D}\left[ \mathbb{E}_{\theta \sim A(D)}\left[ \mathcal{L}(\theta, D) \right] \right]
$$
meaning the variance in the loss as a function of the training set is given by
$$
\mathbb{V}[\mathcal{L}] = \mathbb{E}_{D\sim P_D}\left[ \mathbb{E}_{\theta \sim A(D)}\left[ \left(\mathcal{L}(\theta, D) - \mu_\mathcal{L}\right)^2 \right] \right]
$$
Essentially, for a fixed (random) training procedure, we compute the variance in the model performance as the training set varies.

This is pretty oversimplified and honestly non-standard, so
if you are interested in the formalities of this kind of thing, look into Probably Approximately Correct (PAC) Learning Theory and Empirical Risk Minimization, where this is formulated properly (meaning, rigorously). :)

The comments above discuss (very appropriately) an important special case of this, namely the Bias-Variance Decomposition for squared error.
A common simplification for this case is to assume that $P_D$ samples from a fixed set of paired values $(X\in\mathcal{X}, Y\in\mathcal{Y}=\mathbb{R})$, related by
$$y = h(x) + \xi, \;\;\; x\in X, \, y\in \mathbb{R},\, \xi\sim\eta(\mu_\xi,\sigma_\xi^2)$$
where $\mathbb{E}[\xi]=\mu_\xi$, $\mathbb{V}[\xi]=\sigma_\xi^2$, and $\xi$ independent from $x,y$.
We still therefore denote $D\sim P_D$ to mean sampling a random training set (meaning, draw $n=|D|$ random points $x_i\sim X$ and then perform  $y_i=h(x_i) + \xi_i$).
Note that $\theta$ (and $f_\theta$) a random variable, dependent on the random variable $D$, computed by some (potentially stochastic) procedure like
$$\theta^*(D) \leftarrow \arg\min_\theta \mathcal{L}(\theta,D) =
\arg\min_\theta \frac{1}{|D|}\sum_{(x,y)\in D} (y - f_\theta(x))^p + R(\theta) , $$
where $p\geq 1$ and $R$ is a regularizer, for instance.

Our expected squared error on a single fixed pair $(x,y)$ is then
\begin{align}
\mathbb{E}_{D\sim P_D}\left[ (y - f_\theta(x))^2 \right]
&= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 + 2h(x)\xi + \xi^2 \right]\\
   &\;\;\;\;\;\;\;- 2 \left( \mathbb{E}_{D\sim P_D}\left[ h(x)f_\theta(x) \right] +
               \mathbb{E}_{D\sim P_D}\left[ \xi f_\theta(x) \right] \right) \\
    &\;\;\;\;\;\;\;+ \mathbb{E}_{D\sim P_D}\left[ f_\theta(x)^2 \right] \\
&= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 - 2h(x) f_\theta(x) + f_\theta(x)^2 \right] \\
    &\;\;\;\;\;\;\;+ \underbrace{\sigma_\xi^2}_{\mathcal{V}_\xi} + \underbrace{\mu^2_\xi + 2\mu_\xi\mathbb{E}_{D\sim P_D}\left[ f_\theta(x) - h(x) \right]}_{\mathcal{B}_\xi} \\
&= \mathbb{E}_{D\sim P_D}\left[ h(x)^2 - 2h(x) f_\theta(x) + f_\theta(x)^2 \right] + \mathcal{V}_\xi + \mathcal{B}_\xi \\
&= \underbrace{\mathbb{E}_{D\sim P_D}[h(x)]^2}_{h(x)^2} + \underbrace{\mathbb{V}_{D\sim P_D}[h(x)]}_0 - 2h(x)\mathbb{E}_{D\sim P_D}[f_\theta(x)] \\ &\;\;\;\;\;\;\;+ \mathbb{E}_{D\sim P_D}[f_\theta(x)]^2 + \underbrace{\mathbb{V}_{D\sim P_D}[f_\theta(x)]}_{\,\mathcal{V}_f} + \mathcal{V}_\xi + \mathcal{B}_\xi \\
&=
\underbrace{\left( h(x) - \mathbb{E}_{D\sim P_D}[{f_\theta(x)}] \right)^2}_\mathcal{B_f}+
\mathcal{V}_f +
\mathcal{V}_\xi + \mathcal{B}_\xi \\
&= \mathcal{V}_f + \mathcal{B}_f +
\mathcal{V}_\xi + \mathcal{B}_\xi
\end{align}
where we used $\mathbb{V}[r] = \mathbb{E}[r^2] - \mathbb{E}[r]^2$ for any random variable $r$, $\mathbb{E}[uv]=\mathbb{E}[u]\mathbb{E}[v]$ for independent $u$ and $v$, and the fact that $h(x)$ is deterministic (meaning $\mathbb{E}[h(x)]=h(x)$ and $\mathbb{V}[h(x)]=0$), since $x$ is fixed and we are varying the epxectation over $D$ (which only affects $\theta$).
Note that the "classical" bias-variance decomposition (as on Wikipedia) is a special case when $\mu_\xi = 0$.

In any case, we get that the expected error of our regressor on a fixed input (where we consider the expectation over possible datasets) is a sum of four terms (two variance and two bias):

The bias $\mathcal{B}_\xi$ and variance $\mathcal{V}_\xi$ caused by the inherent noise $\xi$ in the relation between $x$ and $y$.
The bias $\mathcal{B}_f$ which is how far the average prediction of $f_\theta$ is from the true value.
The variance $\mathcal{V}_f$ which is how "noisy" the output of $f_\theta$ is. Note this does not depend on the true value, nor on the noise in the true relation between $x$ and $y$. It is the inherent noise in the learning processes, measuring how much a change in the training set alters the resulting model.

So anyway, I got a little sidetracked, but all of these derivations were merely just to show that the variance in ML is in fact literally a simple mathematical variance. :)


Why is hard for neural machine translation model to learn rare words?

Rare words are not a problem only for NMT, they are a problem for MT in general. The reason is simple: in order to accurately translate a word in any particular context, the model needs to see as many examples as possible during the training stage. By definition the training data contains very few occurrences of rare words (especially hapax words which occur only once), so the model doesn't have enough information to learn their translation properly.


Which neural network to choose for classification from text/speech?

Welcome to the site. I'm a little disturbed by the other two answers you received here. It sounds like you are skipping a whole lot of steps and wanting to jump right into modeling - that's a massive mistake!

You are a scientist! Your role is to create the most fair, unbiased environment possible to let the data speak to you (not the other way around!). What worked before (LSTM) may or may not be the best approach to this completely new data set. Therefore, after doing your EDA phase, you should keep an "open field" view to the multiple models that you will examine and test prior to making any decisions about which model to proceed with. The answer may not even be a neural network, it may be a whole different approach.

Please, be responsible your data science practice. You cannot jump into modeling right away. Let the data speak to you.


Does "feature importance" depend on the model type?

When it comes to feature importance I always go with a model-agnostic measure, as you well mention if you have two different models, they will interpret importance in different terms (Linear models as the coefficient and Tree-based models as the information gain/impurity decrease on each feature.
So you already mention one measure that does not depends on the model, rather on the metric you are interested in; Permutation importance does not care about what model you are using, but the impact that a feature has on the global performance.
This reference might give you a better idea of the advantages of using permutation importance over tree-based models importance Permutation Importance vs Random Forest Feature Importance


how to do Time Based splitting of Amazon fine food reviews dataset

Those dates are on timestamp. try to convert them useing:

from datetime import datetime
datetime.fromtimestamp(1540574790)

where "1540574790" is a moment ago for example.. for more, check https://en.wikipedia.org/wiki/Unix_time