Why could an overfitted CNN model have a higher validation accuracy?

I will try to answer your question "Why can the overfitted model have a higher validation accuracy?" with an example, then try to answer the remaining questions (TLDR at bottom):

Say we have 2 classes with a 1:1 ratio in both the training and testing sets (each 4 instances).

Perhaps our training/validation curves are just like yours (I highlighted some points for reference).

Going from point A to point B ($\overline{AB}$), we do not observe overfitting:

validation accuracy increases
validation loss decreases

$\therefore$ We might conclude an inverse relationship between validation accuracy/loss.

However, $\overline{BC}$ shows overfitting:

validation accuracy is nearly constant
validation loss increases

This appears to contradict the relationship we observed with the $\overline{AB}$ example!

The reason behind this phenomenon is simple: we cannot assume a linear dependence between the accuracy and loss functions.

Let us define our accuracy and loss functions respectively:

$accuracy:\quad 1-\frac{1}{N}\sum^N_{i=1}|[X_i]-\hat{X}_i|$

$loss \space (MSE):\quad \frac{1}{N}\sum^N_{i=1}(X_i-\hat{X}_i)^2$

$\{X \space | \space 0<X<1\},\qquad\{\hat{X} \space | \space \{0,1\}\}$

We can see that these are linearly independent (which you could verify by calculating their Wronskian). How might this affect the relationship between training/loss curves?

Let us plug-in some numbers to see. Columns {A,B,C} represent the model output vectors for these three epochs and the corresponding ground truth is in $\hat{X}$ column.

\begin{array}{|c|c|c|c|}
\hline
A   & B   & C & \hat{X}\\ \hline
0.6 & 0.3 & 0.3 & 0\\ \hline
0.2 & 0.2 & 0.2 & 0 \\ \hline
0.4 & 0.4 & 0.1 & 1 \\ \hline
0.8 & 0.8 & 0.8 & 1 \\ \hline
\end{array}

\begin{array}{|c|c|c|}
\hline
epoch & accuracy \space calculation & accuracy & loss \space calculation & loss \\ \hline
A & 1-\frac{1}{4} * (1 + 0 + 1 + 0) & 0.50 & \frac{1}{4} * (0.6^2 + 0.2^2 + 0.6^2 + 0.2^2) & 0.2000\\ \hline
B & 1-\frac{1}{4} * (0 + 0 + 1 + 0) & 0.75 & \frac{1}{4} * (0.3^2 + 0.2^2 + 0.6^2 + 0.2^2) & 0.1325 \\ \hline
C & 1-\frac{1}{4} * (0 + 0 + 1 + 0) & 0.75 & \frac{1}{4} * (0.3^2 + 0.2^2 + 0.9^2 + 0.2^2) & 0.2450 \\ \hline
\end{array}

Therefore, $C_{loss} > A_{loss},\quad$even though$\quad C_{accuracy} > A_{accuracy}$!

Of course this is a very simple example, and your situation will have many more complexities, but establishing the linear independence of the validation accuracy and validation loss and understanding these curves is a critical aspect of determining which epoch model to select.

TLDR: With this in mind, you should use the model instance at epoch B. This is the point at which overfitting starts, not at epoch 1250. (although I suggest saving many intermediate models and empirically determining their viability on another dataset).

After epoch B, your validation accuracy is not changing while your training loss decreases. Therefore, your weights are fitting more to the training data without improving your results, ergo overfitting. At epoch 1250, your model is still improving validation accuracy quite dramatically, so stopping training then would yield a very poor model.

As we can see from your validation accuracy curve, this model will probably perform similarly to models beyond epoch B, but there are a couple reasons you might prefer model B.

If you ever wanted to resume training (perhaps you add more data to your training set but do not want to restart completely), then having weights that are not overly fitted (or even vanished!) to the initial dataset would be ideal.

A very similar reason is that, even though this particular dataset shows virtually no difference between the epoch B model and models beyond epoch B, other datasets might. It would then be easier to adjust model B for these new datasets rather than an overfitted model.


Classification/Regression Problem where Response Variable is Ordinal

You can treat this problem as multiclass classification and use classification algorithms to solve this problem. You can use any of the classification metrics like accuracy, precision, recall etc.

Regression should be used only for the continuous data and continuous data is the one which can have infinite number of potential values within a given range. For example, for your problem the range is $1-10$, if it is continuous data, then it will have values like $1.04783,6.92838,8.2381,3.999,5.0$ etc. In this case you can opt for Regression algorithm.


If you have an imbalanced dataset, you may want to take steps to resample. When evaluating your results of your algorithm, it may be a good idea to choose a metric that is better suited for class imbalance problems.

Accuracy is not a good metric for evaluating the performance of class imbalance problems because accuracy is rewarded for predicting the most commonly occurring label. Accuracy is the measurement of the the number of labels predicted correctly over the number of all the observations. For example, if you had 99% of the data with label 'a' and 1% of the data with label 'b', and you then labeled the entire dataset with label 'a', your accuracy would be 99%. However, your ability to label 'b' correctly was 0%. Log loss is also not particularly good for class imbalance problems. You may want to consider weighted log loss instead.

Here are some other metrics that you can investigate.

Precision and Recall

Precision is the ratio of correctly predicted positive labels to the total predicted positive observations. Recall is the ratio of correctly predicted positive observations to the all observations in the given class. You can read more about these metrics here. For a Binary Classification problem, you can view the predicted class vs the actual class in a confusion matrix.

F1 Score

F1 Score is the weighted average of precision and recall. F1 score takes false negatives and false positives into account. So, it is more informative than accuracy when evaluating class imbalance problems.

F1 Score exists as a Python package in sklearn. It also exists as a function in R.

Area Under Precision Recall Curve

You can also investigate Area under precision recall curve (PR), which the measurement under the area of precision recall curve plot. It can be used to evaluate large class imbalance problems. PR Curve exists as a python package in sklearn and an R package.


Can I use the training rows multiple times while training with different labels attached to it?

The problem you are describing is a perfect use case for Fuzzy Logic.

Contrary to boolean logic where things are either black or white, fuzzy logic allows things to be grey.

Take the example of weather. Given a temperature, some people might say it is cold whereas other would say it is warm. There will be situations where all say it is cold, but there exists a mid-term where the classification isn't $\{0,1\}$ but rather $[0,1]$.

Therefore, it won't be a problem given that your chosen algorithm allows for probabilistic output.

So given the temperature example. You have the classes [cold, warm, hot].
If your algorithm can only output binary classifications (e.g. [0, 1, 0]), then better only input data points with only 1 label. However, if your algorithm can output probabilistic classifications (e.g. [0.2, 0.7, 0.1]) then you can input the same datapoint with multiple labels.

In the end, you can pick the class with the largest probability, but you need to understand that this might not give the output you are expecting. If you get a classification of [0.49, 0.51, 0.0], you might say is class 2 where in truth the model is not actually telling you that.


How can you build a model that reads out receipts and invoices?

You should start by trying OCR.  I am not sure why you are rejecting it without trying it. Start with Tesseract (free but not very good) and then try a commercial OCR as well (Abbyy is well regarded, but you could also try Adobe Acrobat).  Also, research document structure extraction.

That idea that you are going to re-invent a machine learning model that does OCR better than existing OCR solutions seems ... not very realistic.  Current OCR tools have years or decades of engineering put into them.  There is no way you are going to be able to afford to put that much effort into your own custom tool.

Image quality on those example images is not wonderful, and you might have to accept that accuracy will be less than 100%.


Marketing Mix Model (MMM) using XGBoost?

Hi and welcome to the forum. As far as I know, MMM are causal models with the purpose of making statistical inference. This is the standard world of econometrics, where people stick to OLS or logistic regression since results are easy to understand based on estimated coefficients.

Boosting (XGBoost is only one implementation of specific type(s) of boosting), is a technique which is mainly applied in predictive modelling since you have no "easy to understand" coefficients. So if your MMM application is not primarily about making a prediction, maybe boosting is not a good idea. However, you can still calculate pseudo marginal effects based on predictions, so there might be a way to make boosting applicable for inference (similar like inference is done in logistic regression).

It is not clear what you mean when you say: "due to that interactions effects are dealt with in a good way". Since boosting (usually) is based on very small decision trees, there are no interaction terms. What boosting does is, it focusses on "hard to predict" cases by weighting. So with boosting you usually get much better results (lower SSR, higher ROC etc) compared to OLS, logit etc.

I don't know if and how boosting has been applied to MMM, but generally speaking, there are (in the moment) few applications of boosting in economics. So applying boosting can be innovative in this field, provided that you get it right. This requires a thoughtful definition of model requirements and objectives of the modelling approach.


Filling missing values with pyspark using a probability distribution

You could try modeling it as a discrete distribution and then try obtaining the random samples. Try making a function p(x) and deriving the CDF from that. In the example you gave the CDF graph would look like this

Once you obtained your CDF you can try using Inverse Transform Sampling. This method allows you to obtain random samples for the distribution specified.

This link would be helpful for your understanding of that technique. These random samples can fill those missing values as per your requirement of probabilities.

Note:

There are other techniques as well, you could search and explore along the lines of random sample generation from discrete distributions. It might be the case that your actual data might fit for example something like Poisson's distribution etc. For all such cases, you can find various examples on how to generate random samples for those particular distributions


What clustering algorithm is appropriate for clustering paths?

Like Ricardo mentioned in his comment on your question, the main step here is finding a distance metric between paths. Then you can experiment with different clustering algorithms and see what works.

What comes to mind is dynamic time warping (DTW). DTW gives you a way to find a measure of "distance" (it is actually not strictly a distance metric, but it is close) between two time series. One very useful thing is that it can be used to compare two time series that are of different lengths.

There are many good blog posts on DTW, so I won't try to give yet another explanation of it. There are also many python implementations of it. And a lot of work has gone into making the algorithm very fast. DTW is a strange algorithm--in some ways very simplistic, but typically works well. Once you modify the algorithm to deal with paths, you can construct the distance matrix and use that for clustering. One common clustering algorithm that is used in conjunction with DTW is spectral clustering, since the distance matrix can be used directly (instead of the matrix of data points, which we don't have here).


1 - Nearest Neighbor , dealing with same distance

The best approach would really depend on your application and what is important too you.

However, things you can try include:

Increase K until there is no tie anymore - if you increase to 2, you will likely have another tie, since they are at the same distance already. So, 3 or higher should do the trick.
Include another feature to your classifier - adding another dimension to your space could solve the problem if the values for both these data points are different.
Choose another distance metric - you could have a preferred way of measuring distance, but for ties, chose another metric that will break it.
Establish a rule for breaking ties, e.g.

Pick the class with the most observed data points
Randomly assign a class

More information on your application would help to choose a best approach.


Feature Engineering Lists\Vectors as values in dataframe

I think a good starting point is what you have mentioned. For every element in a list, create a feature for that element. If that element is present in the list for a data point, then the element is denoted as a 1 in your feature vector. If it is not present in the list, then it is denoted by 0. This is very much a bag-of-words type way to create features. You could limit the number of features by only taking the top k occurring elements, where you determine k. Another variant is using counts of frequencies, if an element appears multiple times in a list, but it doesn't sound like that is the case.


Replacing words by numbers in multiple columns of a data frame in R

I guess this is off-topic but here's a reproducible code:

#Preparing mocking data
>set.seed(123) #for reproducibility
>Party <- sample(c('D', 'N','R'), 10, replace = TRUE) #sample random values
>d <- data.frame(Party) #put values into a dataframe

# Here's how you substitute the values in just one line. This operation is vectorised.
>d$Response <- ifelse(d$Party=='R', -1, ifelse(d$Party=='D', 1, 0))

Basically in new Response column, you check if R and assign -1, if not you check if D and assign 1; otherwise you assign 0.

> d
   Party Response
1      D        1
2      R       -1
3      N        0
4      R       -1
5      R       -1
6      D        1
7      N        0
8      R       -1
9      N        0
10     N        0


Where to find state-of-the-art time series data deep learning approaches?

There is an archive of papers related to time series analysis at Arxiv Sanity Preserver which you may find useful. Among the many papers listed there are some which deal with deep learning approaches to time series.


New to ML and NLP. Is topic/subject extraction a good place to start?

You could learn about word embeddings. These will provide you a natural path to topic models, and many other NLP tasks. Look up word2vec.

I think it is better not to learn two new subjects (NLP and ML) at once. I would start with ML on its own; take this class perhaps. A good place to start is linear regression, or binary classification.


Detecting features in XY Plots using CNNs

I am a bit surprised that fully connected neural networks are unable to learn this. It might be due to one or more of the many possible problems like few data or overparameterisation.

Keeping this aside, your problem seems to be solvable by tree classifiers like CART, Random Forest, Gradient Boosting Machine or XGBoost.

The reason I think tree methods can work is that the problem can be solved using a decision tree. To detect the kinks one needs to find whether the datapoint is below the red line within some specific range of X and Y. Once the kink is detected it is easier to predict TRUE or FALSE as you mentioned.


What is exactly psi and segZ function in "segment" function in R

You must attach the data frame curved_data before running the commands. Then, run your command like this:

seg_model <- segmented(reg, seg.Z = ~X, psi = c(-35, 4, 30),
                       control = seg.control(display = TRUE))


How to represent linear regression in a decision tree form

The basic idea would be to divide up your feature space in small multi-dimensional intervals, and then assign to each point in a given interval the average value that your linear regression model has in that interval. This is something you can do with a tree.

This is similar in spirit to approximating (in 1D space) the function $y = x$ with a "piecewise constant", "staircase-like" function like http://mathworld.wolfram.com/NearestIntegerFunction.html: you could divide your 1D space in equal intervals (e.g. of length 1), and you'd assign to each interval the average value that the function $y = x$ has in it.

Note that such "piecewise constant" function can be defined as a tree:   suppose you wanted to know the "tree" approximation of $y = x$ for $x^* = \pi = 3.14159..$, then you could do:

Is $x^* > 0$? Yes
- Is $x^* < 0.5$? No (if it was, I would have approximated $x^*$ with 0)
- - Is $x^* < 1.5$? No (if it was, I would have approximated with 1)
- - - Is $x^* < 2.5$? No (if it was...)
- - - - Is $x^* < 3.5$? Yes -> I approximate $x^*$ with $3$ (3 is the average of $y = x$ between 2.5 and 3.5).

Note that as the size of the intervals shrinks (in "tree language", as you grow your tree more and more), the better the approximation would become. Also, rather than decision one would speak of regression tree in this context.

To generalize this idea, just imagine that you could carry out a similar procedure for any linear function in 1D $y = ax + b$ using a "staircase-like" function, and in N dimensions for any linear function in N-D $y = a_0 + a_1 x_1 + \cdots + a_n x_x$.
Actually, you don't have to restrain to linear functions, as as you said trees are more flexible still (you just need to appropriately assign the values to each interval of your feature space)!


Need help understanding the structure of this convoluted neural network

The so-called motor command $v_t$ (I don't know what it means but it looks to be some scalar feature although it would work the same if it's a vector) is fed into a layer that builds 64 representations of this value, one for each feature map in the convolutional layer that we are going to add it too. The conv layers have a spatial resolution however, and this representation is only one number for the corresponding feature map. What they do is tile this number so that this whole motor command representation has the same dimensions as the max pooling layer after the convolutional layer (pool2). Now that the dimensions match we can use an element-wise addition operation to inject this information into the convolutional network.


Why feature crosses may work better than having them as individual features?

It is not necessarily cross features. Actually, it is a subset of a general concept, namely, kernel trick. Using kernel trick you will be able to find decision boundaries which are more complex than usual hyperplanes which can be found using the linear combination of current feature space. Suppose you have a single feature $x$. By employing that you will be able to make a line as a separator of two classes. If you add $x^2$ alongside $x$ you will be able to have a parabola decision boundary rather than a simple line, which enables you to have a complex decision boundary in cases which is needed. Moreover, because you have both $x$ and $x^2$ you can set the coefficient of each to zero which means you have a kind of flexibility to decide to have a simple line or a parabola.


Is removing poorly predicted data points a valid approach?

The process you described is not valid, it is almost certainly overfitting. However the overall idea may very well be valid. What you should do instead is take a portion of your dataset as the testing which is not looked at during algorithm development, and split the rest between training and validation sets however you please.

Looking only at the training and validation sets, develop your entire end to end algorithm, including the classifier!

Once you are happy with the algorithm, try running it on the test set to get a more accurate measure of performance.


Correlation between specific columns of a data set

I recommend you to use the following example and try to manipulate the arguments and adjust them for your work:

from matplotlib import cm
cmap = cm.get_cmap('gnuplot')
scatter = pd.scatter_matrix(YOUR_TRAINING_DATA, c = YOUR_LABELS_OF_TRAINING, marker = 'o', s = 40, hist_kwds = {'bins':15}, figsize = (12, 12), cmap = cmap)


Michael Stonebraker started a company that claims to do just that, schema matching using machine learning:

https://www.tamr.com/

Their site no longer has many details on their approach but this article talks about some of the techniques they used, like :

  Perform fuzzy string comparisons over attribute names using trigram cosine    similarity.
  Treats a column of data as a document and tokenize its    values with a standard full text parser. Then, measure TF-IDF cosine
  similarity between columns. This method is suitable for text fields.
  Use minimum description length (MDL) to compare values of two    attributes.
  Compute the ratio of the size of the intersection of two    columns' data to the size of their union. This method is well suited
  for categorical fields with small number of values.
  Compute Welch's t-test for a pair of columns that contain numeric values and get the probability the columns were drawn from the same
  distribution.


I believe you can use a classification algorithm where you manually overrepresent the "anomalies" class. By how much, depends on the cost induced by the anomalies.

Just to illustrate what I mean:

Anomalies cover a continuum between two extremes: 1) those which can be safely ignored, because they induce no costs, and 2) those whose costs is unbearable. As an example close to the extreme (1), imagine a flower shop getting an order for Iris virginica. The customer will hardly object if, among 6 million flowers shipped, 50 actually belong to the species Iris versicolor.

On the other hand, if you are shipping airplane parts, and a faulty part would cause a disaster with 300+ dead, then you are closer to the extreme (2). You will likely want to avoid shipping even one faulty part.

In practice, it will have to be a trade-off: You can be on the safe side if you ship no parts at all, but then you go bankrupt. On the other hand, if you do no quality assurance at all, you ship enough faulty parts to cause a disaster in a foreseeable future, have to pay damages, and go bankrupt again.


Image classification with different number of training image for each class

Sometimes it could be the case that one class is simply more common than others. That really bears thinking about, usually. I would guess that is not really the case for what you describe (i.e. it's rather arbitrary what kind of things people would want to classify images of in production later), so let's ignore that case (although it may affect performance on a test set depending on what's the most common in the test set). In general though, you would try out some strategies (e.g. no oversampling whether some extent of oversampling with data augmentation) and test their performance on a realistic test set.

Definitely don't throw away any images. Normally we struggle to create realistic images in data augmentation - here you already have them! Thus, a much more appealing approach is to use a data generator that sample evenly (or in whatever proportions make the most sense) from each class to create batches of training data (with some data augmentation such as slight rotations, changes in colors/saturation etc.).


Always separate training a model from using the model to make predictions.  During training, bootstrap samples are drawn and trees built using those resamples.  Voting only occurs when predicting, and all of the trees are used, whether your datapoint is from the original training set or not.
There is a bit of a caveat here though, because we may be interested in the "out-of-bag" scores of the random forest, in which case we want to let the trees that didn't see a given training datapoint vote on the outcome for that datapoint.  In that case, we need to keep track of which bootstrapped samples contain which datapoints, and then predict for each using the appropriate subset of trees.


I actually tested this recently on a random forest fitting using two approaches:

Using Jupyter notebooks to fit my model via python with data that had been tidied in the same notebook also using python.

vs:

Using reticulate in a rstudio notebook to fit the same model in python but using a converted dataset I had tidied in r in the same notebook.

I made the following observations:

Native python is faster, in my benchmark it fitted the model ~1 min faster (9 vs 10 mins) on my 16GB Ram laptop.
Additionally you lose some time converting objects between the two languages, especially if we talk about bigger datasets.

Conclusion

In most cases doing r stuff first, saving objects to CSV and then opening jupyter or python command lines to finish a job is not necessary thanks to reticulate.

If you do a repeated or time-critical job try to use python natively but otherwise reticulate is fine especially, if like me you save a lot of time doing prep work in r.


This looks like it could be framed either as Ordinal Regression or Ranking using Discounted cumulative gain

Ordinal regression is useful when you are trying to classify classes that have some inherent order like "Medium", "Low" and "Very Low". Used in psychology and marketing extensively it's a fit modeling procedure for this particular problem. The implementation is quite complicated but in a nutshell can be understood as overlapping distributions that can be understood as probabilities of them belonging to said class. R has an Ordinal Regression package.

Another one is Ranking, most machine learning packages include that in the form of Discounted Cumulative Gain which is a measure of rank quality and it is often used to measure effectiveness of web search engine algorithms or related applications. In xgboost it's found as rank:ndcg.

A naive approach but that can be accurate too is a simple multi-class classification. Information about the order will not be included in the algorithm but surprisingly its results are similar in these kinds of problems because the algorithms underneath like neural nets are now quite powerful specially when you have a big dataset.


Simulating datasets of species abundances with varying degrees of turnover in R

Without any example of your data or distributions it is a hard question to answer. Without further info I think there are two approaches:

resample your data using stratified sampling. The R sample package can be of good help here.
generate a multivariate distribution, as far as I know there are no packages that will directly do this for you. Depending on the distributions of your data, creating the right multivariate distribution which includes the right correlations may be very hard. In some cases you may even need to use copulas (this is especially necessary if you are interested in tail events, correlation will not be sufficient to create dependence of these events). Note that this topic can be a separate PhD thesis on itself.

I hope this helps.


Converting RDD to spark data frames in python and then accessing a particular values of columns

To read a csv file to spark dataframe you should use spark-csv.
https://github.com/databricks/spark-csv

df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')

How to use spark csv
If you are using pyspark directly from the terminal. Instead of calling

$SPARKHOME/bin/pyspark

You have to use

$SPARKHOME/bin/pyspark --packages com.databricks:spark-csv_2.11:1.4.0

and then use the code above.

If you are using ipython + findspark, you'll have to modify your PYSPARK_SUBMIT_ARGS (before starting ipython)

export PYSPARK_SUBMIT_ARGS=--master local[4] --packages "com.databricks:spark-csv_2.11:1.4.0" pyspark-shell


Random Forest Classifier cannot recognise parameter grid

Your grid search dictionary contains the argument names with the pipeline step name in front of it, i.e. 'randomforestclassifier__max_depth'. Instead, the RandomForestClassifier has argument names without the pipeline step name, i.e. max_depth. You therefore need to remove the first part of the string which denotes the name of the step in your original pipeline. You can do this using a dictionary comprehension:
# original
{'randomforestclassifier__max_depth': 4, 'randomforestclassifier__max_features': 'sqrt', 'randomforestclassifier__n_estimators': 500, 'randomforestclassifier__random_state': 0}

# splitting the key on '__' and take only the last part
{k.split("__")[-1]: v for k, v in grid_imba_rf.best_params_.items()}
# {'max_depth': 4, 'max_features': 'sqrt', 'n_estimators': 500, 'random_state': 0}

This changes one line in the original script to:
rf_final = RandomForestClassifier(
    **{k.split("__")[-1]: v for k, v in grid_imba_rf.best_params_.items()}
).fit(x_train,y_train)
rf_final.feature_importances_


Help interpreting GAN output, and how to fix it?

In theory, we should first train the discriminator to optimal. However, if it becomes too good, then it will reject fake data every time, thus making the generator unable to learn. Unlike other ML task, lower GAN losses does not mean the training is converging. Newer GANs have better techniques to make training GAN easier such as Wasserstein loss, spectral normalization and progressive growing.


Are dimensionality reduction techniques useful in deep learning

Deep learning does not use dimensionality reduction because deep learning itself is a useful dimensionality reduction technique. Deep learning learns a compressed, nonlinear representation of the data through the hidden layers. Since Deep Learning can learn nonlinear mappings, it is a more flexible dimensionality reduction technique than PCA which restricted to  linear mappings.


Making predictions with missing numeric data

2 Questions.

Yes there are certain algos that deal with it internally. For example, lightGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Check section 3.2 here Or Catboost for example also

Try to save/impute information as much as possible, if there is just too much missing values (30%+) than just drop the column. You will do more harm injecting the noise while imputing. (NOTE 30% is approximation based on experience, some datasets require less or more)


In sklearn's classification report, is f1 the best accuracy measure?

You got recall incorrectly. It does not mean "correct out of all the samples".
$\hspace{50mm}$

Looking at the image,

$\hspace{50mm}$ Recall = $\frac{TP}{TP+FN}$

It is explained here.

From the above documentation, you can also see that you can modify your F-beta score to suite which to weigh more, precision or recall. Therefore, f-score would give you an overall of how good your classifier is, but if you need to know at which samples your classifier fails, you need precision and recall.


How can i use Hellinger Distance on array of different length?

What are you trying to do?

Don't blindly for together functions without thinking about the underlying math!

Hellinger distance is usually applied to histograms, and your before don't look like histograms. So something is wrong I'm your approach... Go back to the drawing board, not to the code.

P.S. k-means will also need vectors of the same length, and doesn't minimize Hellinger, I'd assume...


Classify documents using a set of known vocabularies

First of all you need to have available a train set, which means that you should annotate manually which document is related to soccer and which not. Then you need to process the available corpus (remove numbers, stop-words etc., stemming) and build a vocabulary. After that you should choose the appropriate feature representation. Each term is a feature and you have to decide how you are going to reprsent each feature, which means what kind of weight you will assingn. One way is the tf-idf representation. Then you will be able to train a classifier.

*The only way to avoid labeling manually the texts is to find some already labeled in the same language.


SGD data should be randomly selected or sequentially feed?

If you feed the data randomly, there are fewer chances of overfitting than if you feed the data sequentially.

If the data is fed sequentially, then the network will not learn to generalize as the weights will be updated for that batch according to that same image again and again.

It is always recommended to feed the randomly.


It's a usual trick in deep learning. Matrix multiplications can be done in parallel:
$$
(A.B)_{ij} = \sum_{k} A_{ik} B_{kj}
$$
since you have to do this sum for each component, then you can divide your computing time by the number of processes you use - i.e. compute these components several at a time rather than one by one. Assuming you use a naive matrix computation then you have a $O(\dfrac{n^3}{K})$ with K number of processes. That's why GPUs are so much better for deep learning (usually have hundreds if not thousands of cores) which allows you to divide by a factor of several 100s at worst the time to compute the matrix multiplication.
In this particular case, you can use block matrix multiplications (we expend the rules of matrix multiplications to matrices):
$$
\mathbf{}_{∗()}[_,_{−1}]= \begin{bmatrix}
  \mathbf{}_{∗} & \mathbf{0} \\
  \mathbf{0} & \mathbf{}_{∗y}
\end{bmatrix}[_,_{−1}]
$$
if you apply the traditional matrix multiplications (keeping in mind matrix sizes), you get:
$$
\mathbf{}_{∗()}[_,_{−1}]=\mathbf{}_{∗}_ + \mathbf{0} * _{−1} + \mathbf{0} * _ + \mathbf{}_{∗y} * _{−1} =\mathbf{}_{∗}_ + \mathbf{}_{∗y} * _{−1}
$$


What's the difference between feature importance from Random Forest and Pearson correlation coefficient

Pearson correlations capture linear relationships between the input and target variables. Therefore this only makes sense for continuous inputs and a continuous target variable, and not continuous inputs with a binary/categorical output. Correlations essentially measure the positive/negative 'change' in one feature as you increase/decrease the other.

So it doesn't make much sense to compare the relationship between your input features and the categorical outputs this way. You may as well calculate the mean input for each feature and each label, and calculate the differences between those. I found this answer on Cross-Validated which explains this much better than I can.

Feature importance in tree based models is more likely to actually identify which features are most influential when differentiating your classes, provided that the model performs well. How this feature importance is calculated depends on the implementation, this article gives a good overview of how different tree based models calculate importance for features.


Relationship between batch size and the number of neurons in the input layer

In general I think you've understood both concepts. I'll try to address both of them in more detail.
Input layer size
In Neural Networks (NN), the size of the input layer is always equal to the number of variables (or features as we usually call them in Machine Learning) in your data.
Batch size
This refers to how many samples of data the network will see before updating its weights.
Normally, when performing gradient descent (i.e. the algorithm that trains the NN), the network would be input all of the samples in order to calculate the loss and update the weights. However, especially in Deep Neural Networks (DNNs) this is infeasible, as the size of the dataset is too big and even if the network had enough memory the computation of the gradients would be inefficient. What we do instead is pass a few samples (called a batch) at a time and train the model on them. Then another batch is passed, and so on. The number of samples we input to the model at each iteration is called the batch size.
Relationship between the two
In theory, if you have many features (in the thousands) you can't use a large enough batch size and the more features you have the less of a batch size you can use. However, in practice I have found the two to be practically unrelated! In Deep Neural Networks, the batch size is primarily governed by the size of the model. An easy (but not always accurate) way of estimating this size is through the number of trainable parameters in a model. More parameters, mean more memory for the model (and longer gradient computations), leading to a smaller batch size. In DNNs, that have multiple layers, only a fraction of the parameters are those of the input layer. Thus, the number of features plays a minor part in the selection of the batch size.
Example
In your example you say you have 50 rows and 4 columns, out of which 3 are the input variables.
The number of neurons in the input layer will be 3, to match the number of input variables (or features). This we cannot change, even if we wanted to!
The batch size is something we can change. Let's say we use a batch size of 5.
1st iteration:
In the first iteration, the NN will be given the first 5 rows of data. For each, the first three columns will be passed to the first layer of the network, which will calculate an output. After all 5 outputs are generated, they will be compared with the first 5 rows of the 4th column, to calculate a loss. Using this loss, the network will compute the gradients and perform the updates. Again, this loss corresponds to the first 5 samples, not just the 5th.
2nd iteration:
The rows 6-10 will be passed to the network, which will do the same procedure as in the first step.
10th iteration:
The rows 45-50 will be passed to the network. This marks the point where the NN has seen every sample of data once. In the context of Machine Learning, we call this an epoch. At this point, the data are shuffled and the 1st iteration of the 2nd epoch begins.
Note:
In most frameworks the samples in each batch are processed in parallel by the network and not sequentially. This makes no difference to the result, I just wanted to point this out.


Multi-class text classification with LSTM in Keras

After I read the source code, I find out that keras.datasets.imdb.load_data doesn't actually load the plain text data and convert them into vector, it just loads the vector which has been converted before.

As for your problem, I assume you want to convert your job_description into vector. Maybe you can try sklearn.feature_extraction.text.CountVectorizer.

I'm no expert on NLP, but I've encountered problems (e.g. Two Sigma Connect: Rental Listing Inquiries) which need such technique.

In the meanwhile, there are other word2vec/embedding techniques you may try. Here is a tutorial from keras which shows a detailed example on word embedding.

References

https://github.com/fchollet/keras/blob/master/keras/datasets/imdb.py

http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html

https://www.kaggle.com/sudalairajkumar/two-sigma-connect-rental-listing-inquiries/xgb-starter-in-python


What is the best machine learning algorithm for large, noisy datasets with interaction between variables?

Probabilistic Random Forest tends to work better then other algorithms on noisy datasets. But the data you are using also plays a major role on whether a algorithm will work or not.
Check this paper Probabilistic Random Forest for more details. Happy Learning


Is the _error_ in the context of ML always just the difference of predictions and targets?

The error can have different forms depending on the application. For example for a simple regression we often use the sum of squared deviations between the actual output $y_n$ for the input $x_n$ and the predicted output $\hat{y}(x_n)$ for the input $x_n$. The total loss $J_\text{Gauss}$ is then given as the sum over all squared errors (also known as Gaussian Loss)for each observation.

$$J_\text{Gauss}= \sum_{n=1}^N\left[y_n-\hat{y}(x_n)\right]^2$$

If we use absolute values instead of squares we obtain the Laplacian loss function $J_\text{Laplace}$, which is given by

$$J_\text{Laplace}=\sum_{n=1}^N\left|y_n-\hat{y}(x_n)\right|$$

If we rather try to compare two probability distributions $p(x)$ and $q(x)$ we use a unsymmetric distance meassure called Kullback-Leibler divergence

$$
{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{-\infty }^{\infty }p(x)\ln {\frac {p(x)}{q(x)}}\,dx}.
$$

For binary classification we can use the hinge-loss

$$J_\text{hinge}=\sum_{n=1}^N\max \{0, 1- t_n \hat{y}(x_n)\},$$

in which $t_n=+1$ if observation $x_n$ is from the postive class and $t_n=-1$ from the negative class.

For support vector regression the $\varepsilon$-insensitive loss $J_\varepsilon$ is used. It is defined by the following equation.

$$J_\varepsilon=\max\{0,|y_n-\hat{y}(x_n)|-\varepsilon\}$$

This loss acts like a threshold. It will only count something as an error if the error is larger then $\varepsilon$.

As you can see there are some meassures (see this Wikipedia article for loss functions used for classification) of error for comparing the predicted output and the observed output.


How to visualize data of a multidimensional dataset (TIMIT)

Like I said in the comment, you'll need to perform dimension reduction, otherwise you'll not be able to visualize the $\mathbb{R}^n$ vector space and this is why :

Visualization of high-dimensional data sets is one of the traditional applications of dimensionality reduction methods such as PCA (Principal components analysis).

In high-dimensional data, such as experimental data where each dimension corresponds to a different measured variable, dependencies between different dimensions often restrict the data points to a manifold whose dimensionality is much lower than the dimensionality of the data space.

Many methods are designed for manifold learning, that is, to find and unfold the lower-dimensional manifold. There has been a research boom in manifold learning since 2000, and there now exist many methods that are known to unfold at least certain kinds of manifolds successfully.

One of the most used methods for dimension reduction is called PCA or Principal component analysis. PCA is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. You can read more on this topics here.

So once you reduce your high dimensional space into a ${\mathbb{R}^3}$ or ${\mathbb{R}^2}$ space you will able to project it using your adequate visualization method.
References :

Information Retrieval Perspective to Nonlinear Dimensionality
Reduction for Data Visualization - Jarkko Venna

EDIT: To avoid confusion for some concerning PCA and Dimension Reduction, I add the following details :

PCA will allow you compute the principal components of your vector model, so the information are not lost but "synthesized".

Unfortunately there is no other imaginable way to display 39 dimensions on a 2/3 dimension screen. If you wish to analyze correlations between your 39 features, maybe you should consider another visualization technique.

I would recommend a scatter plot matrix in this case.


Why can't we feed datetime to Linear Regression and how does toordinal() different from any other integer datatype?

Linear Regression is associating any numerical (or binary, which is a particular numerical) value to a coefficient. Multiplying those values by those coefficients gives you an output, and setting the threshold, you know if the model predicts 1 or 0. (This is a brief summary, you'll find plenty of people explaining in details how it works).
If your variable is a date, you have a format, like "Year/Month/Day", and the regression doesn't know how to interpret it, since it need numerical data.
So as to use the date in a regression, you can basically take the year, as a variable.
If you want to be more precise, you'll have to create your own variable, with the meaning you want (Ex : a binary variable taking 1 if the day is a Saturday or Sunday, 0 either).
Side Note : Watch out when you use dates variables, if you want to use your model to predict new, future data. For ex, if you have a variable Year, and you test your model in a Cross-Validation, it will have an impact since the trend can change with years. Moreover, if you want to predict new values input, Year will always be '2020', so this won't be that useful. Utilizing it to evaluate your model might biaise your results.


f1.merge(f2, left_on='num', right_on='num', how='outer')

see https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.merge.html


Getting feature vectors from CatBoost pool

The reason you are getting the error is that your pool contains categorical features. If all of your features were numerical, it would work fine.

SMOTE is irrelevant here.


How can I justify whether there is seasonality present/absent in the
  data after differencing?

Plot the amplitude of the Fourier transform of the signal.

If there is seasonality, you will see a peak at the appropriate frequency on the Fourier plot. This should be close to the plot's origin, because seasonality means slow changes and thus low frequencies.


how to augument speech sentiment dataset?

Your problem is to identify whether a person is sad, happy, angry or neutral from his voice which is emotion classification problem. For speech, we use frames of short duration like 10-20 ms, and extract features from the same like MFCC or other frequency domain features. We extract short duration frames because speech is non-stationary over time, and frequency domain features make the features shift-invariant.

I suggest you read some latest research papers on emotion classification to get latest research in emotion classification.

Emotion in speech is captured by variation in pitch and amplitude across time. So capturing short time energy and instantaneous pitch frequency in speech across time  are basic features for emotion classification.

Modifying speech by adding variation in volume and white noise will not help at all.

You need to build a emotion classification system with 4 classes: sad, happy, angry or neutral, using the below steps:

Extract frames of 10-30 ms duration
Convert these frames to frequency domain features like STFT or extract instantaneous pitch
Use these STFT features along with pitch to train a LSTM


Convert categorical data in numeric preserve euclidean distance

If i understand your question correct, you are misusing the word categorical. Categorical is always a 0 or 1 in their respective indices.

for example:

Data - [M, F, M, M]

Categorical Data: [[1, 0], [0, 1], [1, 0], [1, 0]]

If three types of classes are there, then it would be a 3 arrayed input for each datapoint.

If you feel that having the numbers 10 and 20 for Male and Female is meaningful to you, then you can go ahead and use it. There's nothing wrong in that. But when you want to finally train on the Data, say LSTM, it prefers taking in the categorical data.

But if you are talking about the input attributes, then you need not worry about the 0-1 problem, Just use as they are.

Vote up, if you find this helpful ;)


Which algorithm to use to identify clusters with a similar value?

What you are describing is called ordinal regression. The target variable (income) is divided into discrete groups where the relative ordering between different groups is preserved (low, middle, high).

Binning a continuous variable as the advantage of better handling noisy data.


Hidden Markov Model: Forward Algorithm implementation in Python

Maybe this python library could help you: hmmlearn

When I tried to build an hmm I used it and it worked well.


What affect will replacing words with bigrams have on TfIDF?

TFIDF decreases as term frequency will  be decreased linearly and idf increases log linearly.

Document similarity will decrease as value of tfidf vectors should decrease as reputation of bigrams are more less than each single word.


Why not using linear regression for finetuning the last layer of a neural network?

The common approach to fine-tuning an existing pre-trained neural network is the following:

Given an existing pre-trained neural network model (e.g. imagenet), remove the last layer (which does classification in the pre-training task) and freeze all weights in the remaining layers of the model (usually with setting the trainable parameter to false).
Add a new final dense layer that is to be trained on the new task.
Train the model on the new task's dataset until convergence.
[optional] After fine-tuning is converged, possibly unfreeze the all the layers and train further until convergence with a lower learning rate.

A reason to use gradient descent over a different ML algorithm as you suggest is to enable further training after initial fine-tuning (step #4 above). However, it's not necessary to do this. The approach you suggest (to use take the output of the pre-trained model as input to another ML model) may provide satisfactory performance and be more computationally efficient.
Tradeoffs between these approaches is also discussed in the Keras Transfer Learning guide in the section on the "Typical Transfer Learning Workflow".


Are there libraries or techniques for 'noisifying' text data?

I you want some kind of data-sets like Google spell checking data I suggest you look into the The WikEd Error Corpus dataset. The corpus consists of more than 12 million sentences with a total of 14 million edits of various types, this edits include: spelling error corrections, grammatical error corrections, stylistic changes. All these from the Wikipedia correction history. The owners (authors) of the data-set describe the data mining process in this paper. Also check this question in quora it contains links to various data-sets with spelling errors. Finally this page can also be useful.


Training a model where each response in the observation data has a different known varience

I may be understanding the question now.  Still using the coin example, as I said above, the number of trials for a given coin only affects the confidence for the estimated probability/bias for that one coin.  So it seems like you are asking how to incorporate that "confidence" into the response variable, if at all.  In other words, you are asking if your model should reflect the uncertainty concerning the true value of $p$ for each coin, given the number of coin flips you performed.

I don't think assigning different weights to observations is appropriate in this situation because, again, the number of flips for one coin does not have anything to do with other coins.

I am not sure if this will satisfy your needs, but there is something called interval regression that is used to model a response/dependent variable that is defined as an interval between lower and upper bounds.  It is a type of regression for censored data (a problem where the true value of response is not known) and is typically used for modeling such variables as income ranges or survival times.  In your case, you could compute a confidence interval for the true value of $p$ for each coin, using the $p$ calculated from your trials and the number of trials specific to each coin.  Then you would use this regression with two response variables: lower limit and upper limit of the confidence interval.

Based on my quick search, I am not finding a lot of Python support for this type of model, except:

Maximum Margin Interval Trees - Decision trees for interval regression

Drouin, A., Hocking, T.D. & Laviolette, F. (2017). Maximum Margin Interval Trees. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

https://arxiv.org/abs/1710.04234

https://aldro61.github.io/mmit/


How do I assign a title to each of the histograms I have posted at the same time?

Using the axes variables you have, which is a numpy.ndarray of size 7

axes[0].set_title('Title for Fig. 1') #change title for first histogram
axes[1].set_title('Title for Fig. 2') #change title for second histogram
axes[2].set_title('Title for Fig. 3') #change title for third histogram


How to perform feature selection with Categorical Variables and Continuous Target, provided that data is not normally distributed?

If I understand your question correctly, You are asking how to reduce categorical features from a dataset. If yes then few of the approach I can think of -

1> Iterative Process - Build a model with all numerical features and one categorical feature then evaluate your improvement of the model by whatever metrics you are using and then add other categorical features and so on. So if you have N cat features you will be building N+1 models.

2> chi square test of predictor and target variables.

3> (what I use) Build a model with all the available features and measure it's performance and then use the feature importance functionality of that model to determine which feature is important. In case of linear regression, Higher the value of coefficient better the feature. Alternatively, you can use L1 regularization to check for non zero features. Do check for multi-collinearity before considering feature importance in linear regression.


With more than two variables, you have a dimension problem. Here, with 3 variables and one output you would need a 4 dimensionnal graph, which is not possible unless you use some trick.

1. Reduce the dimension of your problem

Generally speaking, if you need to observe a problem for which the dimension is too big, you may want to reduce its dimension. Observe relations with only one or two variable. Of course this means that you will have some difficulties to to observe more complex relationships.

For your exemple, that would mean plotting independently for (X1,X2), (X2,X3) and (X1,X3):

ax.scatter(x1, x2, y_data[0], c=y_data[0], cmap='viridis');

To be honest, this is not ideal as some point may recover others. This can be adressed by adding some transparancy to the points (parameter alpha), but it doesn't improve the visualisation that much. I would recommend to start with 1D plot (y against another variable), to really understand what is happening:

plt.scatter(x1, y_data[0], c=y_data[0], cmap='viridis');

2. Use color and make the graph interactive

One way to add a 4th dimension to graph is to make the use of color. It has some limitations (you need a good color scale : one that would still render if printed in B&W, one that is color-blind friendly). Indeed, It won't apply to more than 3 variables.

For your exemple, that would mean something like :

ax.scatter3D(x1, x2, x3, c=y_data[0], cmap='viridis');

This faces the readability problem as above (but I find it better as the colour bring some information instead of repeating what is on the vertical axis).

An option is to make the graph interactive, with something like plotly. (More info here: https://plot.ly/python/3d-scatter-plots/)

3. use contour curves

Another approach for adding a dimension to a graph would be to plot contour curves, which represent an ensemble of X values that give the same y). Note that you won't get any "single line representing y_data". Generally speaking I am quite sure this would not render well in 3D (plotting an ensemble of 3D curves), except maybe for your linear regression problem (you would get an ensemble of 3D plane). Again, the main option is to plot a reduced version of your problem, i.e. 2D plots with 2D contour curves.

One main requirement of this approach is that you need to provide the relationships between X and y, which is unknow. So you have to build a model and adapt it to what you want to plot.

For linear regression you would get something like :

Get the estimated model :

w_est = [0.29,0.51,0.09]
b_est = -0.19

def output_X1_X2(X1, X2):
    return X1*w_est[0] + X2*w_est[1] + 0 * w_est[2] + b_est

Set the values for plotting :

x1_plot = np.linspace(-3, 3, 50)
x2_plot = np.linspace(-3, 3, 50)

X1_plot, X2_plot = np.meshgrid(x1_plot, x2_plot)
Y = output_X1_X2(X1_plot, X2_plot)

Plot the output and associated contour :

contours = plt.contour(X1_plot, X2_plot, Y, 20, colors='black')
plt.clabel(contours, inline=True, fontsize=8)
plt.imshow(Y, extent=[0, 3, 0, 3], origin='lower',
           cmap='viridis')
plt.colorbar();

You get a graph with different value of y for X1 and X2. The main drawbacks are : you don't see the interaction with X3, you have to set a given X3 (0 here). Meaning that you have to plot similar graphs with (X2,X3) and (X1,X3), plus you have to make the set aside variable move to value other than 0. Even if this could be automated, it can rapidly be a pain with lots of variables.


How to pre-process the name String of a customer?

You may want to tokenize the strings, e.g., “Mustermann GmbH” tokenizes into "Mustermann" and "GmbH". Allow for spaces and commas certainly, perhaps also hyphens and other punctuation.

You may want to look into Natural Language Processing (NLP) if you're classifying text, but whatever method you choose should have better luck sniffing out business vs. non-business using tokens of the strings.


This is explained in the original article where deterministic policy gradient theorem was first proposed, in section 3.3:

The deterministic policy gradient theorem does not at first glance look like the stochastic version (Equation 2). However, we now show that, for a wide class of stochastic policies, including many bump functions, the deterministic policy gradient is indeed a special (limiting) case of the stochastic policy gradient. We parametrise stochastic policies $\Pi_{\mu_\Theta,\sigma}$ by a deterministic policy $\mu_\Theta : S \rightarrow A$ and a variance parameter $\sigma$, such that for $\sigma = 0$ the stochastic policy is equivalent to the deterministic policy, $\Pi_{\mu_\Theta,0} \equiv \mu_\Theta$.
Then we show that as $\sigma \rightarrow 0$ the stochastic policy gradient converges to the deterministic gradient (see Appendix
C for proof and technical conditions).

Therefore, the Deterministic Policy Gradient method is equivalent to a normal (stochastic) Policy Gradient Method in the limit.


How to apply class weight to a multi-output model?

I wansn't able to use the class_weight parameter yet, but in the mean time i've found another way to apply class weighting to each output layer.

Current solution

In this keras issue they have supplied an easy method to apply class weights via a custom loss that implements the required class weighing.

def weighted_categorical_crossentropy(y_true, y_pred, weights):
    nb_cl = len(weights)
    final_mask = K.zeros_like(y_pred[:, 0])
    y_pred_max = K.max(y_pred, axis=1)
    y_pred_max = K.reshape(y_pred_max, (K.shape(y_pred)[0], 1))
    y_pred_max_mat = K.cast(K.equal(y_pred, y_pred_max), K.floatx())
    for c_p, c_t in product(range(nb_cl), range(nb_cl)):
        final_mask += (weights[c_t, c_p] * y_pred_max_mat[:, c_p] * y_true[:, c_t])
    return K.categorical_crossentropy(y_pred, y_true) * final_mask

where weights is a CxC matrix (where C is the number of classes) that defines the class weights.
More precisely, weights[i, j] defines the weight for an example of class i which was falsely classified as class j.

So how do we use it?

Keras allows to assign a loss function for each output.
so we could assign each output a loss fucntion with the correct weights matrix.

For example, to satisfy the request i made in the question we could suggest the following code.

# Define the weight matrices
w1 = np.ones((2, 2))
w1[1, 0] = 10
w1[1, 1] = 10

w2 = np.ones((3, 3))
w2[0, 0] = 5
w2[0, 1] = 5
w2[0, 2] = 5
w2[2, 0] = 10
w2[2, 1] = 10
w2[2, 2] = 10

# Define the weighted loss functions
from functools import partial
loss1 = partial(weighted_categorical_crossentropy, weights=w1)
loss2 = partial(weighted_categorical_crossentropy, weights=w2)

# Finally, apply the loss functions to the outputs
model.compile(loss={'output1': loss1, 'output2': loss2}, optimizer='adam')

And that accomplishes the request :)

Edit

There is a small edition that needs to be made.
The loss functions must have a name, so we can supply this with the following:

loss1.__name__ = 'loss1'
loss2.__name__ = 'loss2'


Guidelines to debug REINFORCE-type algorithms?

The only resource I could find so far :

https://github.com/williamFalcon/DeepRLHacks

For my specific case, I made a few errors :

Frozen some part of the network that shouldn't be frozen
Wrong learning rate

Even if I could overfit a small dataset, it didn't mean anything : while training on the whole dataset, the average reward was not going up.

You should look for a reward going up.

I'm not accepting this answer as I believe it is not complete : it lacks general and systematic guidelines to debug a Reinforcement Learning algorithm.


How to pass 2 features to LSTM , one of them is one-hot-encoded with Keras?

(1) Yes, it is a common practice to encode the categorical feature by one-hot encoding, for example, encode [1,44] as [1,0,0,0,44], encode [2,44] as [0,1,0,0,44], etc.

(2) Same as (1), just concatenate the one-hot encoded categorical feature and the numerical feature(s).


Are month and weekday rulers of surveys? How can I test the validity?

It looks like a good case for ANOVA, which can be seen as a generalization of Student t-test for more than two groups.

For visualizing the differences I'd suggest using boxplots (or violin plots) for every group: it's more informative than only plotting the mean which is sensitive to outliers.


LSTM classification with different sizes

masking and padding are indeed popular preprocessing choices when working with LSTM. The easiest way is to pad all the short sequences to the same length as the longest sequence.
Here is an example of masking and padding in Keras
https://www.tensorflow.org/guide/keras/masking_and_padding
Is that what you're looking for?
raw_inputs = [
    [711, 632, 71],
    [73, 8, 3215, 55, 927],
    [83, 91, 1, 645, 1253, 927],
]

# By default, this will pad using 0s; it is configurable via the
# "value" parameter.
# Note that you could "pre" padding (at the beginning) or
# "post" padding (at the end).
# We recommend using "post" padding when working with RNN layers
# (in order to be able to use the
# CuDNN implementation of the layers).
padded_inputs = tf.keras.preprocessing.sequence.pad_sequences(
    raw_inputs, padding="post"
)
print(padded_inputs)

Output:
[[ 711  632   71    0    0    0]
 [  73    8 3215   55  927    0]
 [  83   91    1  645 1253  927]]

Besides, you can also add start and end tokens to your sequences, like this example (https://www.tensorflow.org/tutorials/text/nmt_with_attention)
def preprocess_sentence(w):
  w = unicode_to_ascii(w.lower().strip())

  # creating a space between a word and the punctuation following it
  # eg: "he is a boy." => "he is a boy ."
  # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation
  w = re.sub(r"([?.!,¿])", r" \1 ", w)
  w = re.sub(r'[" "]+', " ", w)

  # replacing everything with space except (a-z, A-Z, ".", "?", "!", ",")
  w = re.sub(r"[^a-zA-Z?.!,¿]+", " ", w)

  w = w.strip()

  # adding a start and an end token to the sentence
  # so that the model know when to start and stop predicting.
  w = '<start> ' + w + ' <end>'
  return w

```


Training LSTM with different sequence lengths in Keras functional api

Model specification is per sample, number of samples should not be included in the dimensions, i.e. (a, b, c) or (sample_size, timesteps, dimension) should be changed to (timesteps, dimension). Also, for variable-length sequences, timesteps should be None. That is, (timesteps, dimension) should be (None, 295). The same goes for (x, y, z).

Please check out this post for feeding variable-length multi-dimensional sequences to Keras LSTM.


In SGD you just feed an example to your model, compute the gradient of the loss function of that example and update the weights according to the gradient of the loss of that example.

In mini-batch gradient descent you feed a batch to your model, compute the gradient of the loss of that batch and update the weights according to the gradient of the loss of that batch.

In fact, SGD is mini-batch gradient descent with batch size equal to 1.


Is Cross Validation needed for regression if you already know the predictors in your model?

You might.
(But probably not.)
The point of cross validation is to help you conduct model selection. You've already selected your model. One place where you might want to use cross validation is if you want to use some kind of regularization, but that might interfere with your ability to infer how each $X$ influences $Y$. (The regularized estimates are biased.)


The thresholds don't matter; what matters are the (FPR, TRP) values at those thresholds, as they are points on the curve. Sort them by FPR ascending. For this to work out, you'll want to include the points (0,0) and (1,1) in your list, corresponding to thresholds 1 and 0.
You can use a trapezoidal approximation, as each successive pair of points defines a trapezoid of area under the curve. You'll just add up the areas.
Let's take two successive points (FPR1, TPR1) and (FPR2, TPR2). The area is (FPR2 - FPR1) * (TPR1 + TPR2) / 2. Just sum that over all successive pairs of points.
Of course, most libraries can compute this for you from this input, like scikit-learn.


How do you visualize neural network architectures?

I recently created a tool for drawing NN architectures and exporting SVG, called NN-SVG


Correlation coefficient and non linear association

Yes, Dr. Sullivan is right. For example, take a perfect quadratic relationship between X and Y.

Here is some Python code to show nine sample points and calculate their Pearson correlation coefficient. You can skip the code and just look at the results below if you trust me.

import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')
plt.rcParams["figure.figsize"] = [5, 8]

x = np.arange(-4, 5)
y = x**2  # This is x squared.

plt.scatter(x, y)

plt.axis('equal')
plt.title('$X = Y^2$', fontsize=20)
plt.xlabel('X', fontsize=14)
plt.ylabel('Y', fontsize=14, rotation=0)
plt.xticks([-4, -2, 0, 2, 4])
plt.yticks([0, 1, 4, 9, 16])

# Calculate the Pearson correlation coefficient:
print('r =', np.corrcoef(x, y)[0, 1])

r = 0.0

Surely there is a relation between X and Y in this example, and yet the Pearson correlation coefficient is zero.


How to decide neural network architecture?

Sadly there is no generic way to determine a priori the best number of neurons and number of layers for a neural network, given just a problem description. There isn't even much guidance to be had determining good values to try as a starting point.

The most common approach seems to be to start with a rough guess based on prior experience about networks used on similar problems. This could be your own experience, or second/third-hand experience you have picked up from a training course, blog or research paper. Then try some variations, and check the performance carefully before picking a best one.

The size and depth of neural networks interact with other hyper-paramaters too, so that changing one thing elsewhere can affect where the best values are. So it is not possible to isolate a "best" size and depth for a network then continue to tune other parameters in isolation. For instance, if you have a very deep network, it may work efficiently with the ReLU activation function, but not so well with sigmoid - if you found the best size/shape of network and then tried an experiment with varying activation functions you may come to the wrong conclusion about what works best.

You may sometimes read about "rules of thumb" that researchers use when starting a neural network design from scratch. These things might work for your problems or not, but they at least have the advantage of making a start on the problem. The variations I have seen are:

Create a network with hidden layers similar size order to the input, and all the same size, on the grounds that there is no particular reason to vary the size (unless you are creating an autoencoder perhaps).
Start simple and build up complexity to see what improves a simple network.
Try varying depths of network if you expect the output to be explained well by the input data, but with a complex relationship (as opposed to just inherently noisy).
Try adding some dropout, it's the closest thing neural networks have to magic fairy dust that makes everything better (caveat: adding dropout may improve generalisation, but may also increase required layer sizes and training times).

If you read these or anything like them in any text, then take them with a pinch of salt. However, at worst they help you get past the blank page effect, and write some kind of network, and get you to start the testing and refinement process.

As an aside, try not to get too lost in tuning a neural network when some other approach might be better and save you lots of time. Do consider and use other machine learning and data science approaches. Explore the data, maybe make some plots. Try some simple linear approaches first to get benchmarks to beat, linear regression, logistic regression or softmax regression depending on your problem. Consider using a different ML algorithm to NNs - decision tree based approaches such as XGBoost can be faster and more effective than deep learning on many problems.


This is not a limitation of the Sobel operator, if the boundaries are clear, even a single pixel wide rectangle should be detectable.

I ran your code, but just used an empty image as the "im" image and it worked fine for me.

Here is my code (similar to yours, just without 'Test.png' and without saving the images):

from PIL import Image
import numpy as np
from scipy import ndimage

im = Image.fromarray(np.zeros((81,91)).astype(np.uint8))
pix_val = np.array(list(im.getdata())).reshape(81,91)
for i in range(12,14):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(63,65):
    for j in range(12,78):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(12,14):
        pix_val[i,j]=255
for i in range(14,63):
    for j in range(76,78):
        pix_val[i,j]=255

Oim1 = Image.fromarray(pix_val.astype(np.uint8))
pix_val2 = np.delete(pix_val,range(16,75), 1)
Oim1.show()

Oim2 = Image.fromarray(pix_val2.astype(np.uint8))
Oim2.show()

dy1 = ndimage.sobel(Oim1, 1)
Oim3 = Image.fromarray(dy1.astype(np.uint8))
Oim3.show()

dy2 = ndimage.sobel(Oim2, 1)
Oim4 = Image.fromarray(dy2.astype(np.uint8))
Oim4.show()

This is what I get:


Can be possible to solve Rubik's cube using DQN?

I think yes you can since rubric group can be represented as grid word environment
see this link from GitHub they attempted solving using dqn


What is different between R2 and mean of R2 in multiclassification probelm? Which one is correct?

First of all, if you are in a classification problem as you said, R2 score is not good, it should be used for regression problem. For classification problem you have to use something like accuracy, precision, recall and F1 score.
First question
Anyway, I will answer your first question about R2score:
R2 mean: you are calculating the R2 for each split you have in your CV(train the model with training data, and evaluating it in test data). And then, you calculate the mean of each value of the split. In your case, you will have 9 values of R2score since you have 9 CV. And the result is the meaning of these values.
r2_score: you are evaluating the y_true and y_pred. You already have the y_true. To obtain the y_pred you used cross_val_predict. What you do, is calculate the predicted value using only the CV where you have the y_true in you test split.
I would recommend you use R2 mean, it's more robust score in my opinion
I recommend read this post to understand it in more detail.
Second questions
First of all, take care of what I mention: r2 is not a good score for classification problem. But let's suppose that you use another score and you have similar score outputs.

Is my model overfit\underfit?

Yes, it is. Basically, if your training accuracy is much bigger than your test data, you are overfitting. Your scores must be quite similar. What you have now, is a model that is "memorizing" and not learning. Be careful.

Shall I tune my hyperparams?

Yes, you have to try tunning your hyperparameters in order to achieve similar score between training data and test data. You should check how the model works changing hyperparameters and decided which helps you to avoid overfitting. You can do it manually or using GridSearchCV for example.


Cleaning a certain feature to predict salary using Machine Learning

There are two ways to handle this:

Aggregating values and feature engineering

Instead of simply inputting the individual city, you could try to aggregate this value into a new variable "JobCitySize" to cluster cities by inhabitant size or a variable "Region" to cluster by geographic region. This will increase cases and also help solve the question of how to deal with unseen data.

Defining NA or unseen values

With any model and categorical encoding you have to think ahead what you want to do with "unseen data" in your case cities that aren't present in your training data set. Do you want to set them as NA, give a standard value ("Other"), etc.? Whatever process you use for these values you could also use for any value in your data set that falls below a threshold of "usefulness".


How to train model to predict events 30 minutes prior, from multi-dimensionnal timeseries

This is a fun problem. This is a time series and from this time series you want to identify the trigger of a certain event. So it is a binary classification problem. Based on the information from the specified window will a spike occur? Yes or No.
The first step is to set up your database. What you will have is a set of instances (which can have some overlap but to avoid bias it is best for them to be independently drawn) and then for each instance a human needs to label if there was a spike or if there was not a spike.
Then you need to identify the time window you want to use for your time series analysis. You have done this and decided 30 minutes is a good start.
Now, you have 6 waveforms in a 30 minute window from which you can extract data to get information about your classification. You can use the raw data samples as your features, but this is WAY TOO many features and will lead to poor results. Thus you need some feature extraction, dimensionality reduction, techniques.
There are a million ways you can extract data from these waveforms. First, ask yourself, as a human what are the telltale signs that these other waveforms should have which would mean a spike would arise. For example, in seismic data, if you see agitation in a waveform from a neighboring town then you should expect to see agitation in your town soon.
In general, I like to extract all the basic statistics from my waveforms. Get the mean, standard deviation, fluctuation index, etc. Get whatever you think might help. Check how these statistics correlate with your labels. The more correlation the better they might be. Then there are some very good techniques for extracting time and frequency information from your time-series. Look into envelope mode decomposition and empirical mode decomposition. I have used empirical mode decomposition successfully on some time series data and obtained far better results than I expected.
Now even though you have your reduced feature space you can do better! You can apply some dimensionality reduction techniques such as PCA or LDA to get a lower dimensional space which may better represent your data. This might help, no guarantees.
Now you have a small dataset with instances that are a Frankenstein concoction which represents your 6 waveforms across the 30 minute window. Now you are all set to select your classifier. You will want a binary classification algorithm, luckily that is the most common. There are many to choose from. How to choose?

How many instances do you have?
$\# instances > 100* \#features$?
Then you are all set to use a deep learning technique such as neural networks, 1D convolutional neural networks, stacked autoencodders, etc...
Less than that!!!!
The you should stick with shallow methods. Check out kernel support vector machines, random forests, k-nearest neighbors etc..

Common misconception: A shallow method CAN and WILL perform better than a deep learning technique if you have properly selected your features. feature extraction is the most important aspect of a machine learning architecture.

I want to use anomaly detection!
This would also work and there are some good techniques that would do this. However, the nature of anomaly detection is to learn the distribution of the nominal case. So you would feed your algorithm all the instances in your dataset that did not result in a spike. Then from this your algorithm would be able to identify when a novel instance is significantly different from this nominal distribution and it will flag it as a n anomaly. This would mean that a spike will occur in your context.
Check out:

Learning Minimum Volume Sets
Anomaly Detection with Score functions based on Nearest Neighbor Graphs
New statistic in P-value estimation for anomaly detection

You can also use more rudimentary anomaly detection techniques such as a generalized likelihood ratio test. But, this is kind of old-school.


Personally, I would side with deploy it as is. I.e. do not retrain on all the data.

Once you train on new data, then you have a new model. You have no idea how this model is going to react to unseen test data coming in from the big wide world. You can’t validate or test it, because you’ve used all that data up in training.

Who knows what might happen!

As you receive more data and get feedback on the model you can create new train/test sets, fine tune the model, then redeploy it as required.


Time required to train a model based on features shape and CPU capacity?

There is no answer to that question because besides model parameters, hardware specs and dataset size, there is also the question of the difficulty of the problem, which is not quantifiable.

An easily-solvable problem can require fewer epochs for the model to sufficiently converge. Take for example two image classification tasks:

The first is ILVRC which contains around 1m images.
Another dataset that contains 1m red and green images (just pure red and pure green).

Both datasets have the same image dimensions, the same number of images and will be used to train the same model on the same computer. Even though these tasks seem identical, the latter is much easier to solve and will require only a few iterations (i.e. a few seconds). The former is harder and could require 2-3 weeks.

Due to this huge difference caused, solely, on a non quantifiable characteristic of the dataset, there is currently no way to tell how long a model will require to train.

Furthermore, due to the stocahasticity of the training process, two identical models might take different times to converge, even on the same dataset (due to different initial conditions).

Note: what you can estimate is the time it takes for each epoch if you know the model (number of parameters, choice of certain hyperparameters), the dataset (features, samples) and the hardware.


Depending on the item domain You can expect different results. I would try to cluster data using cosinus similarity over item latent features. If number of items is large try DIMSUM (it's already implemented in Spark).

For the visualization You can try one of the available implementation of t-SNE. Quick links: wiki and YT - Google Talk about t-SNE Visualization. But here I cannot give You any more advice, because I have never tried to visualize it. I looking forward You will share Yours results.


Closed Domain Question Answering which doesn't answer Questions

Question Answering model of simpletransformers takes data with is_impossible option in the training phase. Also during prediciton, it does not generate answers for all questions since the model learns which questions should be answered.
You can find details below:
https://pypi.org/project/simpletransformers/


Which algorithm should be used for an accurate job recommendation system

You are much, much too early in your process to even begin thinking about your models. At this stage, you should be thinking about what the data looks like and how much of it do you need. For starters, what are you measuring? What is the answer that you seek? Is it job satisfaction? Likelihood to land an interview? Likelihood to get a job offer?

Once you have that, how are you quantifying that? Is it a categorical variable? A continuous variable? Then you would have to decide on the likely list of factors that go into that answer and determine how you're going to get that data. Do you need to run a custom survey? Are there other datasets that you can leverage? You need to think about all of these answers before you're ready for any sort of modeling discussion.


I suggest this guide for a good introduction to the resources available to you to learn to do data science. You are starting at a good time, there are loads of excellent (and mostly free) resources to teach you this interesting field.


if for same input values we have different class labels, Should we have to remove those training examples before building any model on it?

No, you should not remove them.

The problem in your case are not these individual samples. The real problem is that the attributes (variables) that you chose, i.e. A1-A2-A3, are not informative enough to accurately discriminate the samples. This is visible from the fact that the same input space (i.e. X-Y-Z) leads to different classes (A-B).

If you visualize this in a 3D plot with different colors for each class, you will see that the same datapoint will have two colors (!). This is because two datapoints fall exactly on top of each other and it is impossible to find a linear/nonlinear separation for them, no matter how hard you try.

Therefore, if you wish to increase the classification accuracy, you should pick other attributes/variables and not drop these individual samples; you have to change the way you look at the problem :)


It seems that the SeqSelfAttention layer is expecting all the time-steps.
i.e. return_sequences=True
 Same is shown in the home page example. Link
import tensorflow as tf, numpy as np
from tensorflow import keras
from tensorflow.keras.layers import Dense, Dropout,Bidirectional,Masking,LSTM
from keras_self_attention import SeqSelfAttention

X_train = np.random.rand(700, 50,34)
y_train = np.random.choice([0, 1], 700)
X_test = np.random.rand(100, 50, 34)
y_test = np.random.choice([0, 1], 100)

model = tf.keras.models.Sequential()
model.add(keras.layers.Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(keras.layers.Bidirectional(LSTM(20, dropout=0.25, recurrent_dropout=0.1, return_sequences=True)))
model.add(SeqSelfAttention(attention_activation='sigmoid'))
model.add(keras.layers.Flatten())
model.add(Dense(1, activation='sigmoid'))

model.summary()

model.compile(loss='binary_crossentropy')
model.fit(X_train,y_train,epochs=2)


Random Forest prediction fails due to unseen Features

Problem is the way you're onehot encoding.

Best practice for any type of encoding :

You should train an estimator for  Onehot encoding on the training data only, and when encoding test data, you should use the same estimator used on training data.

Eg : sklearn.preprocessing.OneHotEncoder does this, and it has a parameter called : handle_unknown.

  handle_unknown{‘error’, ‘ignore’}, default=’error’
  Whether to raise an error or ignore if an unknown categorical feature is present during transform (default is to raise). When this parameter is set to ‘ignore’ and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. In the inverse transform, an unknown category will be denoted as None.

Optimal option is : You could use this parameter and set it to ignore, in order to ignore the unknown feature value and avoid an error, until you retrain your model eventually and add the new feature values to your model.

from sklearn.preprocessing import OneHotEncoder
ohe=OneHotEncoder(handle_unknown='ignore')
train=ohe.fit_transform(train)
test=ohe.transform(test)

Or you could , as you said , manually correct differences in the feature space, but would be time-consuming at each update of your model, without excluding the possibility of your code raising an error if you're sloppy in your manual correction.


Why training error is larger than validation error after PCA?

The basic principle that helps us implement PCA is it's ability to explain the variation, so it depends how much PC you are taking and how much variation they are explaining. They are definitely not explaining 100% of variation so there are good chance that Validation Error is higher.

Before applying PCA you need to understand why we apply it? We apply it to reduce computational effort and practical implementation of problem. The idea of PCA is not to give better result than actual variables in problem.

Better results are generally available with complete list of variables.


Problem with Logistic regression using categorial variables

The issue is that logistic regression cannot make use of features which are strings. To use these features in your model you have to convert them to values first by encoding them. There are several ways of encoding the string values into numerical values, examples of this are using a LabelEncoder, OneHotEncoder, and OrdinalEncoder.


Is it possible to plot decision boundaries for only a subset of features?

For decision trees, this can be both possible and useful. For the random forest (RF), each decision tree has their own boundaries for these features. And, since RF trains each tree with a subset of original features, some trees don't have any these features and so boundaries. This might also occur due to pruning. So, you have a set of boundaries, which can be plotted at the same time to get a feeling of how the trees in the forest learnt your dataset. However, these boundaries also depend on other features. If some more explanatory feature is selected in a higher node in some trees, the low level thresholds might mislead you.


Chossing between gradient boosting algorithms

I would say Catboost and lightgbm perform similarly and its purely a matter of choice.
Some of my colleagues prefered Catboost when dataset has lots of categorical columns, but I rarely saw any advantage over lightgbm.
there is a great article comparing CatBoost vs. Light GBM vs. XGBoost
https://towardsdatascience.com/catboost-vs-light-gbm-vs-xgboost-5f93620723db
And another article comparing even more boosting algos
https://medium.com/@divyagera2402/boosting-algorithms-adaboost-gradient-boosting-xgb-light-gbm-and-catboost-e7d2dbc4e4ca


Libraries for (label propagation algorithms/frequent subgraph mining) for graphs in R

This is an old post, but there is a subgraph package and accompanying book/documentation for doing this in R:
https://www.csc.ncsu.edu/faculty/samatova/practical-graph-mining-with-R/PracticalGraphMiningWithR.html

Although I personally don't get the connection between subgraph mining and label propagation in this case. SVD++ might be closer to what you're looking for (supported by GraphX of Spark, which I think also supports label propagation).
http://spark.apache.org/graphx/


Ideas to check if a given sentence A is the response to a query B

So, I gathered that your question is about how to evaluate whether a sentence A is a response to a given question Q or not.

Drawing from knowledge of question-answer systems, you can evaluate whether a sentence responds to question Q by comparing the underlying lambda calculus (essentially translates language meaning into a logical, more computation-looking form - https://www.cl.cam.ac.uk/teaching/1314/NLP/slides6.pdf) which make both the question Q and a response A.

To get the lambda calculus for a question Q or response A, we can use an encoder-decoder model to generate these.
In an encoder-decoder network architecture, we simply encode input (e.g. question Q using an RNN/LSTM) information into a "hidden representation", which we then subsequently decode into our desired output (e.g. Lambda Calculus, using an RNN/LSTM). This process is actually called semantic parsing and here are a couple of papers which do this and label the architectures they used to generate these, including:

https://www.aclweb.org/anthology/P16-1004/
https://www.aclweb.org/anthology/P18-1068/

Maybe this can be an implicit way of comparing the questions with the responses, by then looking at similarities in their lambda calculus.


Any Neural Network implementations that allow for a cost function of more than just the network output?

You should be able to do that in Keras, actually you should be able to do that in almost any flexible package. For example regularization does exactly that, it adds a penalty term to your loss function that punishes high weights.


What kind of data (in context of trends in data) is Logistic Regression appropriate for?

As you would be aware that, logistic regression is a standard linear model, following these steps would be helpful :

Plot the input data, just observe how would probably the decision boundary look like.
If it's completely non-linear then other models will give you a good result than logistic regression.
If there is a high chance of linearity possible then you can go with logistic regression


How is the number of grid cells in YOLO determined?

As far as I can tell, there is no specific rule. It will depend in part on how crowded your scene will become with items that you want to detect and locate separately. Creating a high granularity grid increases computational cost for training, and there is no reason to do so if it would only cover additional cases that are much rarer than the detection accuracy that the base algorithm achieves.

The choice can be driven by the ground truth data. The ground truth for YOLO needs to be expressed in the form of grid locations with classes and bounding rectangle sizes. If you don't find any training examples where you want to label two items with their centre inside the same grid square, then this is a good indication that your grid size is fine-grained enough.

Even if there are one or two examples with a clash like this, you may be able justify labelling just one item in the ground truth and be OK with a resulting model that is not able to cope with close overlap between two separate objects. And even with smaller grid squares, YOLO may not be able to learn to separate the objects, if such an overlap only occurs rarely.

I would expect a simple rule of diminishing returns applies. As datasets grow larger, and object detection can be trained on more powerful computers, we may see state of the art models still using YOLO but with more grid points.


How can I have a sequential model inside Keras' functional model?

I managed to trial and error through this.

x = Sequential()
# add some layers...
inputs = [Input(...), Input(...), ...]
outputs = ... #some layers stacked together via the functional model
result = Model(inputs=inputs, x(outputs))

If you want the sequential in the middle of the outputs up there for some reason:

output1 = Layer(input)
output2 = x(layer1)
outputs = Layer(output2)


Can we use pca for supervise classification?

Let $X\in\mathbb{R}^{n\times m}$ be your data and $Z=XT\in\mathbb{R}^{n\times k}$ be the PCA transformed data.

  Can we use "pca feature selection" for supervised classification? What will happen to labels when we use dimension reduction?

Yes. It is commonly used for feature selection and linear dimensionality reduction. Nothing happens to the labels. At the start you have training pairs $(x_i,y_i)$, and after PCA you have $(z_i,y_i)$ instead.

  If I understand it right when we use pca for feature extraction after do it we have old features but in new dimension then can we say these are the new features or mixed features?

You have new features which are simply linear combinations of the old features. Another way to see it is that you chose a new set of (orthogonal) axes in the data space (potentially throwing away a few of them), and then projected your data onto this new subspace. So $Z$ is (literally) a mixture of the features from $X$.

  Is "feature extraction" different with "feature selection" in pca? I mean they are two things to do with different ways.

It can be both or either. To me, feature extraction implies obtaining latent descriptors from raw data. So if PCA is run on raw data, then it is doing feature extraction. Otherwise, it is not. As for feature selection, I would say whenever you remove any principal components (i.e., performing linear dimensionality reduction), then you are doing feature selection because you are selecting some features but not others.

  How can select effective features with pca? There is a lots of references on the internet but I couldn't find any MATLAB sample code or video (on youtube) about pca feature selection, I mean not in theory, I'm looking for sample code in any language(Paython, Matlab,etc).

People tend to either (a) choose a fixed variance to explain based on the normalized singular values (e.g., 90% of the original variance is explained) or (b) treat the number of selected features ($k$) as a hyper-parameter, and cross-validate over a validation set with it. Some discussion on the explained variance and its connection to the reconstruction error are here or here.

As for practical code samples, the scikit-learn library has some simple examples:

from sklearn import decomposition, datasets

k = 3
iris = datasets.load_iris()
X = iris.data
Y = iris.target
pca = decomposition.PCA(n_components=k)
pca.fit(X)
Z = pca.transform(X)


Deduplicating e-commerce products based on title/description/images

As a starting point chapter 3 of Mining Massive Datasets (MMDS) offers a very good introduction on document similarity. You can also look here. For near-duplicate image detection look at this question on StackOverflow.

For the text data, first you have to convert your strings to a vector of features, this features could be tf-idf vectors of all the tokens (words) that appear in the database, could also be a vector of n-grams. For a discussion of semantic and syntactic features you can look here.

Given the vectors for each product on your database you would have to compute the similarity among all pair of products. Then the most common similarities measures are cosine similarity and Jaccard similarity.

The brute force approach is O(n^2), where is the number of products in the database so this could be unfeasible. A common technique to deal with this issue is to use fingerprinting (hashing) together with Locality Sensitive Hashing (LSH), both techniques are discussed in chapter 3 of MMDS.


Evaluating a IR system (Precision and Recall)

My doubt is related to the following question: Do we use those two measures only if we are testing a IR system or not?

Technically the answer is no because precision and recall are used to evaluate not only IR systems but also many other tasks. However your question seems to be specific to IR so I'll assume that it's actually about the distinction between testing and evaluation:

Testing a ML system consists in predicting the target variable for a set of instances given as input (in the case of supervised learning a "model" obtained from a previous stage of training is required as input as well). At this stage we don't know whether the predictions are correct or not.
Evaluation is the process of assessing the quality of the predictions: it's done after obtaining the predictions from the testing stage, and it requires some form of "gold standard", i.e. data which says what is the correct answer for every instance.

In IR, the testing stage happens every time the system is run to find relevant documents based on a query.

Naturally one wants at first to make sure that the system works properly and returns actually relevant documents, so the system needs to be evaluated, for instance with precision and recall using a dataset containing some queries and their relevant documents (gold standard).
Once the quality is evaluated, the goal is to use the IR system (testing) without evaluating the results every time. Of course there's no evaluation so the performance measures (precision and recall) are not used.


what are the main differences between parametric and non-parametric machine learning algorithms?

Non-parametric machine learning algorithms try to make assumptions about the data given the patterns observed from similar instances. By not making assumptions, they are free to learn any functional form from the training data and hence are flexible.
Unlike parametric approach, where the number of parameters are fixed, in non-parametric approaches the number of parameters grow with training data.
If your data set is too small or otherwise is a set that is not representative of the entire population, then your result will be biased in more ways than possible with parametric methods. To get better results in Non-parametric machine learning algorithms we need large amount of data where relationship between features are not known.
A Gaussian mixture model (GMM) is basically a k-means clustering with probabilistic cluster assignment so it depends on number of components/clusters as parameter whereas in KDE, the free parameters are the kernel which specifies the shape of the distribution placed at each point and the kernel bandwidth which controls the size of the kernel at each point.
Refer this blog to visualize KDE better.


Repeating samples will probably only slow down the learning process. As long as the mini batches are small this will probably not have a big effect.
However, if batches and the number of repetitions become large gradient based learning can run into difficulties.

If you don't shuffle the batches, the repeated samples will prevent the model from taking in new information and may nudge it to "overfit" on the current samples. Since at every batch you only get one new sample (while all the others are repeated), the learning will either slow down or become stale and stuck at an unfavorable location.

Since your approach only has a two-dimensional input, this is probably not a major concern since the loss function will probably not be as complex as in other neural network applications.

Overall, in your case, you probably will not see a big impact on performance except for a longer learning time.

You do not need to be concerned with combining predictions for repeated samples, since you can make predictions using the final parameters after the learning is completed.