Based on my own experience and in reading what others have written, SQL is one of those skills employers look for, perhaps even assume that you have along with some of the basic skills of communication and teamwork.  The main reason is that lots of data are stored in a relational database with SQL being the primary way to extract that data to get it into your models.


Training and testing AdaBoost for low probability classification

Is it possible that adding generated data to your data set will decrease the fraud/non fraud ration and make your dataset more representative / usable ?

At GenieLog, we are producing test data for designing and testing fraud detection tools. We our generator GEDIS Studio we can define regular profiles and fraudster profiles, instantiating each category to a customizable ratio (for ex. 2 % of customer will have fraudulent usage of generated events.)

We did it successfully for telecom CDR (http://www.gedis-studio.com/online-call-detail-records-cdr-generator.html) and Credit Card usages. There's a freely available access to the online generator on http://www.data-generator.com

I'm pretty sure that even if the tool is not matching your needs at least the approach can be valuable. Otherwise I would be interested to read any objection :)

Regards


I have this data set for crimes of a 12 month time period, over 250k rows. I want to predict future crimes by date and location

Also I would say think about the implications of your model (even if it is just some toy data).

If your model has overtly data from one location and not the other you might end up introducing some bias in your data - some crimes are not always known to the authorities. Crime is a very complex topics are your features relevent to the problem? Maybe some other features should be in the dataset and they are not?

Models applied in "high stake scenarios" often discussed on topics such as the fairness of AI should be very carefully validated. Once again that is just some toy data, but I thought it was noteworthy.


Ensemble modelling using model's probabilities

Let's say you fit n models on given dataset, to put them together, you have in fact few options:

Take the average (I'd play with weighted or geometric)
Voting (soft or hard). Basically you predict a class instead of probability and take the one which is most often predicted (hard) or take argmax or probabilities (soft). You havent mentioned what language you are using, but scikit has nice documentation for that.
Fit a metaclassifier on these predictions. You have n models, hence n predictions for each record, so you fit another model on these predictions. This is called stacking as mentioned in the comment.
You can go further for blending, boosting or bagging, there is really nice article about these methods here.


I believe that the simplest is to use Seaborn. Check out the example below:

import numpy as np
import seaborn as sns

sns.set()
np.random.seed(0)

uniform_data = np.random.rand(10, 12)
ax = sns.heatmap(uniform_data)

And the output looks as follows:

This is taken from the seaborn documentation


Representing 2 dimensional data for facial recognition

You are describing is a Cartesian coordinate system where each point is an ordered pair of signed numbers.

In order to compare across people, you need to define the same origin (0,0) and define all points relative to that. Typically, the origin of a face is the tip of the nose.

The input to a model would be a collection of these Cartesian coordinate pairs for each training instance.

Google's Firebase ML-Kit has a very specific API for Face Detection  with specific facial landmarks and each point is a VisionPoint. Given the predefined, rigid, and hierarchical structure, it would be more useful to store the data as JSON to preserve the relationship between label and coordinate pair.


What is a logworth statistic, and how useful is it?

logworth is a p-value transformation based on Pearson Chi-Squared test.

Pearson Chi-Squared test evaluates the probability of having a split caused by chance. The higher the Pearson Chi-Squared value, the higher the chance that the split is caused by dependence. A p-value is given according to the Chi-Squared score and the degree of freedom.

Because some data can have high dependence relationship, the p-value can be extremely small. Taking a logworth of p-value (-log(p-value)) allows us to mitigate this extremely low value.


What's a good machine learning model for an univariate data set?

Your question is not very specific. I am assuming that you are looking for the full solution (Language, package, function, template).  So with that in mind I'll point you to the following question:
Polynomial regression using scikit-learn

Half way down the answers is a solution by Salvador Dali for:

Language: Python
Package:scikit-learn
Function: linear_model.LinearRegression()


In Incremental Learning will the model be updated automatically?

Incremental learning is analogous to online learning. It is based on the assumption that your model can receive a continuous stream of data from which it can keep learning indefinitely. Training is therefore based on Mini-Batch Gradient Descent optimization: you feed batches of data into the Network as soon an new data comes in.

Hope this helps, otherwise let me know.


Why my perceptron doesn't train well and produces bad results on test data?

Two points about the whole thing

You did not test yet. The point behind the training process is to make machine able to learn from the data conditioned on the ability of generalizing this to predicting samples which it has not seen before. Otherwise, the good training is actually overfitting. So here you trained on X and you need to create new samples and check the result to really call it testing (this is an introductory explanation).
Machine Learning is about features a lot! Playing with features and cleaning, modifying and filtering them is a key point. In your example, the last dimension of your 3d data is always 1. Does it distinguish anything? (in my course you get a complete explanation of this in the Lecture 2). So that feature (dimenstion, element of the vector) can/should be removed. To better understanding, imagine your 3d spread of the data. The z axis is always 1 which means the topology of points is what you see in x-y plain. So use only that one.


I got it thanks to the scikit team, I put the answer here for the people to come. The split used in scikit uses weights in calculating the Gini coefficient, just add the following lines before returning:
    ....
    gk *= len(kids)/len(df)
    ga *= len(adults)/len(df)


Is there a way to recognize which of these scatter plots is "better"?

You could:

Label manually few of them (say 100-150), then train a simple model to classify data. A small Random Forest could do the job well.
Train a super basic model on the each dataset used to produce each scatterplot. Something like a Linear Classifier. If the Classifier doesn't make mistakes, you have "clearly separated data", if it makes mistakes then it's likely to be the opposite.


How to calculate precision at K and NDCG for ranking algorithms

Precision at k (precision@k) requires labeled data. Each item needs to be labeled as relevant or not relevant for the given user.

$$ precision@k = \frac{Number\ of\ relevant\ @k}{Number\ of\ recommended\ items\ @k} $$

Labeling data is independent of train/test split. Train/test split is useful to avoid overfitting and for estimating a model's ablity to generalize.


How to get correlation between two categorical variable and a categorical variable and continuous variable?

Two Categorical Variables

Checking if two categorical variables are independent can be done with Chi-Squared test of independence.

This is a typical Chi-Square test: if we assume that two variables are independent, then the values of the contingency table for these variables should be distributed uniformly. And then we check how far away from uniform the actual values are.

There also exists a Crammer's V that is a measure of correlation that follows from this test

Example

Suppose we have two variables

gender: male and female
city: Blois and Tours

We observed the following data:

Are gender and city independent? Let's perform a Chi-Squred test. Null hypothesis: they are independent, Alternative hypothesis is that they are correlated in some way.

Under the Null hypothesis, we assume uniform distribution. So our expected values are the following

So we run the chi-squared test and the resulting p-value here can be seen as a measure of correlation between these two variables.

To compute Crammer's V we first find the normalizing factor chi-squared-max which is typically the size of the sample, divide the chi-square by it and take a square root

R

tbl = matrix(data=c(55, 45, 20, 30), nrow=2, ncol=2, byrow=T)
dimnames(tbl) = list(City=c('B', 'T'), Gender=c('M', 'F'))

chi2 = chisq.test(tbl, correct=F)
c(chi2$statistic, chi2$p.value)

Here the p value is 0.08 - quite small, but still not enough to reject the hypothesis of independence. So we can say that the "correlation" here is 0.08

We also compute V:

sqrt(chi2$statistic / sum(tbl))

And get 0.14 (the smaller v, the lower the correlation)

Consider another dataset

    Gender
City  M  F
   B 51 49
   T 24 26

For this, it would give the following

tbl = matrix(data=c(51, 49, 24, 26), nrow=2, ncol=2, byrow=T)
dimnames(tbl) = list(City=c('B', 'T'), Gender=c('M', 'F'))

chi2 = chisq.test(tbl, correct=F)
c(chi2$statistic, chi2$p.value)

sqrt(chi2$statistic / sum(tbl))

The p-value is 0.72 which is far closer to 1, and v is 0.03 - very close to 0

Categorical vs Numerical Variables

For this type we typically perform One-way ANOVA test: we calculate in-group variance and intra-group variance and then compare them.

Example

We want to study the relationship between absorbed fat from donuts vs the type of fat used to produce donuts (example is taken from here)

Is there any dependence between the variables?
For that we conduct ANOVA test and see that the p-value is just 0.007 - there's no correlation between these variables.

R

t1 = c(164, 172, 168, 177, 156, 195)
t2 = c(178, 191, 197, 182, 185, 177)
t3 = c(175, 193, 178, 171, 163, 176)
t4 = c(155, 166, 149, 164, 170, 168)

val = c(t1, t2, t3, t4)
fac = gl(n=4, k=6, labels=c('type1', 'type2', 'type3', 'type4'))

aov1 = aov(val ~ fac)
summary(aov1)

Output is

            Df Sum Sq Mean Sq F value  Pr(>F)
fac          3   1636   545.5   5.406 0.00688 **
Residuals   20   2018   100.9
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

So we can take the p-value as the measure of correlation here as well.

References

https://en.wikipedia.org/wiki/Chi-square_test
http://mlwiki.org/index.php/Chi-square_Test_of_Independence
http://courses.statistics.com/software/R/R1way.htm
http://mlwiki.org/index.php/One-Way_ANOVA_F-Test
http://mlwiki.org/index.php/Cramer%27s_Coefficient


Log loss vs accuracy for deciding between different learning rates?

According to me, it is not correct to co-relate loss with accuracy.

  Loss is used to optimize the hypothesis such that we can get best
  weights whereas accuracy is used to identify how well model is doing in term
  of correctly predicting the values.

Model internally takes the reference of predict_proba() and returns 1 if probability is > .5 otherwise 0. For example if returned predict_proba() is (.49, .51), model will return 1 as an classification output.

Now consider a use case where some trained model is used for test-data prediction. Assumed such model has 100% accuracy but predict_proba() value close to (.49,.51) or (.51,.49) i.e. having very low confidence level.

In such case logloss is quite high, even though it's having 100% accuracy.

If our criteria for model selection is "accuracy value" then we are selecting the bad model.


I'm wondering if this is really a machine learning problem.  If you can get to text out of the PDF then you should be able to do a lot with a rules-based text analysis algorithm.


Preparation for Career in Data Analysis Without College

At least based on what I and other data analysts/scientists do in my company, your technical topics list seems sufficient. But I would also add:

Visualization (ggplot2 in R, matplotlib in Python, d3.js for really cool stuff)
Design of experiments

Communication skills are also quite important.

For more inspiration, here's a good "curriculum" represented as a metro map: http://nirvacana.com/thoughts/becoming-a-data-scientist/

Let me also recommend building up a portfolio of data science projects. This could consist of your analyses of data sets you find online (e.g. on UCI), Kaggle competitions, or class projects (e.g. via Udacity or Coursera). That way, you can give direct proof of your technical skills, your communication in the form of reports or graphics, and your ability to extract insight.


How can I implement tangent distance for k-nearest neighbor in python/scikit-learn?

ComOnGetMe's KMeans tangent distance metric looks good, if it didn't work for you initially you should fork it and work on the code. I would contact him directly, he probably would have insight for you. Scikit-learn doesn't have a distance metric for tangent distances but the documentation states you can call a user-defined distance (all be it with overhead).

Contributing code might be a good direction to go if you would want to contact scikit-learn's developers and there are a couple of researchers with published papers implementing tangent distance as a metric, so you could have luck there too.


2 stage ensemble -- CV MSE valid in 1st stage but not in 2nd

To clarify: you build one random forest on training data and get some results which seems to have no overfit, since CV and test results are similar. The second RF is built on the predictions of the first RF and some other additional features. The error on CV training data is very low and on test data are very high.

Now we will analyze the performance of the second RF without considering how it was built. Usually if you fit data very well on training set and very poor on test set it means you simply overfit data. This happens when your model learns too much from the data. The point is that in your training data you have two kinds of information. The first one is the real structure of the data, and the second one is the noise around that data, assuming that you have enough data and your model is able to handle the useful structure. The ideal case happens when you learn only the useful structure and when leave out the noise.When that happens you will have close errors in training and test data, which means that what your model learnt is at least almost only useful data. When you learn too much, which means you learn useful and noise together, your training error decreases because the results have the noise structure which is in training data, and on test data the errors grows since the noise from test data does not look like the noise from training data.

Now the first learner looks like it does not over fit. This of course does not always mean that you can't do better. It only means that what your learn is a structure which is also found in test data and thus has predictive power. So is understandable that you aim to improve the performance of the first predictor. One way would be to try to improve that predictor, and you can search for that by studying how that model learns. For example you can study if that model improves performance with more data by training repeatedly on samples of increased size and see if stabilize itself or not. There are many approaches. Another approach would be ensembles and a prominent one is stacking.

But you did not understood the idea of stacking. The whole point of stacking can be expressed in the following statement: explore the model space by building various different model families and in the end get the best from all of them. So stacking starts with the assumption that since we do not know which would be an ideal model for our data, we should try many imperfect ones. Some of them will behave well in some regions of space, some others in other regions of space, but none of them will be able to handle all the space at high performance. The key point is to have models from different families. Sometimes the same algorithm can produce very different models, and sometimes not. For example two SVMs with very different kernels can produce totally different surfaces. Sometimes even for the same kernel an SVM can produce very different results for very different complexity parameter. Often for RF this is not the case, but it is not impossible to get different surfaces with different parameters on RF. So the conclusion is that stacking aims to explore different ways of learning.

What you have done is to apply stacking on a single base learner by adding new features. First of all is that in general it does not work. The predictions from the first learner has a string structure, it comes from averaging on regions. So, this predictions compared with additional variables from the second model are more powerful. This means that every wrong thing learned in the first RF becomes very powerful on the second RF, because it uses the same averaging over regions procedure for learning. So the errors from the first learner amplifies in the second learner. I do not know about the additional features, but in this conditions it looks like it is very easy to overfit.

Turning back to what's in a data set: this data set has a structured content which you have to learn and an unstructured content (noise) around that. Your purpose of learning is to learn only the structure. The first RF is able to distinguish the structure from noise simply because the noise is not altered. The second RF is not able anymore to make that distinction, since the predictions of the first RF are already smooth in regions, which means it has a powerful structure, at least from the point of view of the second RF, which learns in the same way.

A last idea is that it looks like what you have done is somehow similar with a boosting procedure. The first learner gives you a structure which is used as a baseline by the second learner. But there is a big difference than in boosting. While in boosting you have a base line which you aim to improve, you improve that baseline by looking again at the original features, which have the noise unstructured. In your case at the second iteration you do not have again that chance and everything is structured. Thus errors amplifies.

My 2 cents: try multiple learners, eventually using additional features in those learners, try different models for base learners, start with very simple (low complexity or small degrees of freedom for) the stacking learner, like a penalized logistic regression and more further to more complex ones in a controlled fashion.


There are multiple ways you can achieve this. I guess the easiest would be to create a function that saves a bar chart for your KPI to a file:

save_KPI_plot <- function(fn, kpi_data) {
    png(paste0(fn, ".png"))
    # Plot code
    dev.off()
}

You can call the function either as follows,

save_KPI_plot("kpi_1", kpi_1_data)

or store your file names in one list and your data in another and loop over both:

kpi_fns <- list("filename_1", "filename_2")
kpi_data <- list(kpi_1_data, kpi_2_data)
for (i in seq_along(kpi_fns)) {
    save_KPI_plot(kpi_fns[i], kpi_data[i])
}

If you prefer to have a different image format you can change png() to bmp() or jpeg().


Resampling : My dataset is categorical or numerical?

For purely categorical data like yours Chewla et al. proposed SMOTE-N in the original paper. SMOTE-N is implemented in imbalanced learn and the user guide describes the differences compared to vanilla SMOTE and SMOTE-NC as follows:

If data are made of only categorical data, one can use the SMOTEN variant [CBHK02]. The algorithm changes in two ways:

the nearest neighbors search does not rely on the Euclidean distance. Indeed, the value difference metric (VDM) also implemented in the class ValueDifferenceMetric is used.
a new sample is generated where each feature value corresponds to the most common category seen in the neighbors samples belonging to the same class.

Also note that your dataset is not extremely imbalanced so depending on the differences in misclassification cost of your classes you may or may not benefit from sampling techniques.
As a side note: ROC Curves can have some caveats when used on imbalanced datasets see, for example, "The Relationship Between Precision-Recall and ROC Curves" and "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets".


How to determine threshold in Sigmoid function

As per Andrew Ng's Course, if you use the sigmoid activation, the outputs represent the probability of either outcome 0 or outcome 1.

So the decision boundary is 0.5

if prediction > 0.5 , the prediction is 1

if prediction <= 0.5 , the prediction in 0

Here's a screenshot from Andrew Ng's slides:

I have gone through your code and results. There seems to be something wrong with the implementation as none of your predictions give a value greater than 0.5 .

I couldn't pin down the problem. Some debugging will be needed on your side.

What's paradoxical is that your loss is reducing. I suspect this might be because your data set is unbalanced i.e., you have 221 0s and about 30 1s. This could be the reason for other problems as well.

Consider the wikipedia example, where the values are correctly matching.

If you manage to find the error or the solution conclusively, please post it here so that we can all learn.

Hope this helps!


Create price matrix from tidy data without for loop

You can do the following:

df1 = df.set_index(['name', 'date'])
prices = df1['close'].unstack()
marketcaps = df1['marketcaps'].unstack()


Example of a problem with structured output labels

Structured Learning is basically learning prediction functions that is used to map input data to complex output space.
The regular SVMs are used for univariate classification where input is mapped to atomic labels or regression where input is mapped to vectors or scalar numbers.
Structured Output learning extends this very process to more complex output spaces like:
- Multi Class Classification : where output is a "set" of class labels.
- Object localization Problem : where output space defines bounding boxes.
- Sequence alignment problems in protein homology detection,
- Parts of Speech tagging in NLP , etc.

Hope this clears it up for you.


Is there an asymmetric version of nominal correlation?

I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.

I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python

EDIT: I also have a blogpost about it.


I don't know the framework but it seems that's what happens:

Since this is multi-label classification, it's possible that an instance is assigned no label at all by the model. Apparently at least one of the samples contains only instances which are predicted with no label. This means that there are no instances predicted positive for any label in this sample. This causes an NaN value for the precision of this particular sample, which in turn makes the average precision over all samples undefined (hence the error).

I assume that this is not related to the fact that all the classes have a positive precision and recall, since these values would be calculated over the whole dataset, not sample by sample.


NLP Text Summarization - which metrics to use in evaluation?

So, your question is talking about whether human reference summaries are required to evaluate summarisation models.

The short answer is yes at the moment. The most important things about an output summary that we need to assess are the following:

The fluency of the output text itself (related to the language model aspect of a summarisation model)
The coherence of the summary and how it reflects the longer input text.

The problem with have an automatic evaluation system for a text summarisation model is that, although we can assess fluency from a language model, we can't really assess whether the model has pulled "the most salient" pieces of information from the original, longer text (and this can subjective from person to person). Hence why we need multiple human reference summaries to compute ROUGE and BLEU. However, as you are aware, these metrics have their limitations.

ROUGE is essentially a further development of BLEU, which have been commonly used a dubious proxy for output text fluency in research to compare summarisation and translation models. These metrics are dubious because they simply look at how much they overlap with reference texts from humans (https://rxnlp.com/how-rouge-works-for-evaluation-of-summarization-tasks/#.Xt1ewy-ZOi4).


Why my svm.SVC.fit( ) (linear kernal) run so long time?

From scikit-learn documentation:

  The implementation is based on libsvm. The fit time scales at least quadratically with the number of samples and may be impractical beyond tens of thousands of samples. For large datasets consider using sklearn.linear_model.LinearSVC or sklearn.linear_model.SGDClassifier instead, possibly after a sklearn.kernel_approximation.Nystroem transformer.

Yo can change

clf = svm.SVC(kernel='linear').fit(X_train, y_train.ravel())

by

from sklearn.svm import LinearSVC
clf = LinearSVC(random_state=0, tol=1e-5)
clf.fit(X_train, y_train.ravel())


A/B Testing (Binomial Distribution vs Random Distribution)

The question you are asking is somewhat confusing and I believe there's a few assumptions you are using which are incorrect.  Please bear with me and correct me if I'm misinterpreting.
You suggest there are different types of click thru rates, constant and random (normally distributed); This is not true.  Click thru rates are assumed to be randomly distributed in a sample.  I do not know what a constant click-thru rate is, but there is no such assumption in the T-test calculation.  Order of clicks do not matter, only the probability, which is equivalent to the actual click thru rate (which is unknown).  When you sample, through measurement, you will then calculate the estimated click thru rate.  We know that this process is probabilistic, and we can end up with estimates that are higher, or lower than the actual CTR, but it should be close.  It is the sample mean which is normally distributed as described by the Central Limit Theorem.  The standard deviation of the estimated CTR, also known as the standard error, is set by the estimated CTR.
The "distribution" of clicks in your sample is assumed to be random, but the order should not matter in the calculation, and should not affect the validity of the test results.
Running an AA test is a good way to confirm whether all the math, statistics, and logging are working out.  If you are finding statistically significant results more frequently than your significance threshold (alpha), something is wrong.  But I cannot see how the "distribution of clicks assumed to be constant" can possibly be the root cause.
Good luck and may your tests be properly powered.


Hash function for text documents that maps similar documents to the same value

Hashing the unique copies of anything, including documents, is most commonly called fingerprinting.

Picking the fingerprinting hash function depends on your use case. For your use case, pick a rolling, non-cryptographic hash function. One of the simplest examples is Rabin–Karp algorithm. Once applied, similar documents will have similar hash values.

Another issue is comparing hash values to identify near-duplicates. Exact nearest-neighbors algorithms work best but are not scalable. Approximate nearest-neighbors algorithms are scalable but can have errors. Locality-sensitive hashing (LSH) is an example of an approximate nearest-neighbors search algorithm. You'll have to decide the trade-off between scale and potential errors.


Why don't we want Autoencoders to perfectly represent their training data?

The only way an autoencoder can to perfectly represent the training data is by having a hidden layer that is the same size as the input and output layer. Thus, there would be no compression of the training data. The data would be its own model (i.e., f and g are identity functions).

The goal of an autoencoder is to learn a compressed, lossy model of the data.


Avoid iterations while calculating average model accuracy

Yes, you can do all this using the Caret (http://caret.r-forge.r-project.org/training.html) package in R. For example,

fitControl <- trainControl(## 10-fold CV
                           method = "repeatedcv",
                           number = 10,
                           ## repeated ten times
                           repeats = 10)

gbmFit1 <- train(Class ~ ., data = training,
                 method = "gbm",
                 trControl = fitControl,
                ## This last option is actually one
                ## for gbm() that passes through
                verbose = FALSE)
gbmFit1

which will give the output

Stochastic Gradient Boosting

157 samples
 60 predictors
  2 classes: 'M', 'R'

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times)

Summary of sample sizes: 142, 142, 140, 142, 142, 141, ...

Resampling results across tuning parameters:

  interaction.depth  n.trees  Accuracy  Kappa  Accuracy SD  Kappa SD
  1                  50       0.8       0.5    0.1          0.2
  1                  100      0.8       0.6    0.1          0.2
  1                  200      0.8       0.6    0.09         0.2
  2                  50       0.8       0.6    0.1          0.2
  2                  100      0.8       0.6    0.09         0.2
  2                  200      0.8       0.6    0.1          0.2
  3                  50       0.8       0.6    0.09         0.2
  3                  100      0.8       0.6    0.09         0.2
  3                  200      0.8       0.6    0.08         0.2

Tuning parameter 'shrinkage' was held constant at a value of 0.1
Accuracy was used to select the optimal model using  the largest value.
The final values used for the model were n.trees = 150, interaction.depth = 3
and shrinkage = 0.1.

Caret offers many other options as well so should be able to suit your needs.


How do we get the coefficients and intercept in Logistic Regression?

Maybe this part of the code is trying to teach you activities post ML training i.e. prediction
The coefficient and intercept are the parameters of the Model. These are determined by using Training data (Features and Labels) and training process
You follow these steps(Very high level) -
 Get data - X , Y
 Define model i.e. Logistics Regression
 Train Model using the data - Here you get the Coef/Intercept
 Predict using the Model
In your work, you are at the last step.
 You have a model ready with the parameters and you also got the data i.e. Features(X)
 It is as good as calling a function i.e. get_answer(x) and get_answer is your model with it's parameters.
Then you say
if answer > THRESOLD(normally 0.5 but we get the best by trials) it's Class-A else it's Class-B
You may learn Logistics regression from scratch form Blog-machinelearningmastery


feature engineering in test and train sets (on combined data or separately on train and test)

The idea is to build features based on your training set. In a real-world application, all you will have is a training set to build and deploy a model, and that model will predict the test examples that the application needs. In this setting, it is not realistic to assume that you will have the test data to generate features, as you will already have your model delivered. So, the mechanism to generate the features can only be fed with training information.

About this:

  What are the implications when the distribution of a categorical variable are different in train and test and what happens when they are similar?

The implications of having different distributions of the data in the training and test sets are very bad. Your learning algorithm will only learn about things that are on the training set, and for that reason, if your test set has a different distribution you should consider changing your training set. However, if they follow the same distribution, then your algorithms will generalize properly if you do things right.


What are some of the available methods for handling multi-label classification for longer sequences of text

Traditional methods don't have such a limit: Naive Bayes, SVM, decision trees...

Also see https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification


Converting pandas series object to int in pandas

Your code is correct. The problem is that you have NaN values in your id column and python interprets NaN as float. In your code any decimal values it encounters will be converted to integer but represented as $x.0$ due to presence on NaN values. See here for more.

As for a solution to your problem you can either drop the NaN values or use IntegerArray from pandas. It is currently experimental but suits yor problem.


Optimal Dimension of Graph(Vertex) Embedding

I think that finding the absolute dimension of expressivity is a difficult problem. Here are some important facts to consider when you come with a word embedding size.

Make sure you leave enough dimensions for expressivity. You want to
make sure that there is at least enough dimensions to express the
complexity of the structure you are looking to encode
That the number of dimensions is not too large so that you suffer some difficulties in computation.

To conclude, finding the infimum can be done post-hoc, in the sense that you can evaluate the performance on the function and see which value of $d$ does not significantly decrease the performance and choose it this way. However, at a post-hoc level, the only way I can see of doing this is by evaluating the embedding space beforehand, I have not read too much literature about this.

https://arxiv.org/pdf/1711.00331.pdf - Semantic Structure and Interpretability of Word Embeddings

There is also a lot of work done to evaluate the embedding schemes that we have chosen. In the recent NAACL conference there has been work that has been done in this domain particularly.


Difference between Bernoulli and Multinomial Naive Bayes

Bernoulli models the presence/absence of a feature. Multinomial models the number of counts of a feature. Here's a concise explanation.

Wikipedia warns that

  Note that a naive Bayes classifier with a Bernoulli event model is not
  the same as a multinomial NB classifier with frequency counts
  truncated to one.

To understand why, we should note that, as this page nicely explains,

  Whereas the binomial distribution generalises the Bernoulli
  distribution across the number of trials, the multinoulli distribution
  generalises it across the number of outcomes, that is, rolling a dice
  instead of tossing a coin.

What does this imply for us? Multinomial NB cares about counts for multiple features that do occur, whereas Bernoulli NB cares about counts for a single feature that do occur and counts for the same feature that do not occur.

This means that, for example, Multinomial NB will classify a document based on the counts it finds of multiple keywords; whereas Bernoulli NB can only focus on a single keyword, but will also count how many times that keyword does not occur in the document.

So they do model slightly different things. If you have discrete multiple features to worry about, you have to use Multinomial NB. But if you only have a single feature to worry about, then you can make a modelling choice based on the above.


Is there an established methodology for human expert evaluation of machine learning predictions?

Measuring the inter-annotator agreement is one of the ways to measure the reliability of the annotations.

(I assume that there are also other methods but I'm not that knowledgeable about the topic)


I want to add demographic data to a data set. Any suggestions on where to find zip code level data?

The US Census American Community Survey can provide a lot of demographics by different geographies, including zip. Check out:

https://factfinder.census.gov/


How Mllib in Spark select variables in logistic regression

First, the spark programming guide for LogisticRegressionWithSGD recommends using L-BFGS instead, so perhaps focus on the one.  As for variable selection, the model description on the MLLib page for regressions has a nice explanation of how models are constructed and selected, but it does not address variable selection.  This leads me to believe that it considers all variables, and simply chooses the model with the best fit.


if we don't know $F(x)$ for unseen data, how does a decision tree minimize this error?

Every supervised ML method relies on the assumption that the test data (any unseen data) follows the same distribution as the training data (note that this is not specific to Decision Trees). In fact both the training data and the test data are assumed to be sampled from the true population data. As a consequence $F(x)$ is assumed to be the same for the training data and the test (unseen) data.
If one uses a trained model on some unseen data which is not distributed like the training data, the results are simply unpredictable and the performance is likely to drop.

Why do we estimate the error for unseen data using the error observed in the training data?

You seem to suggest to use the "unseen data" in the training process. You would indeed get better results on the "unseen data" if you optimized on it, but then you would lose the point of having a portion of data set apart. "Unseed data" is necessary to estimate how good your model will perform on data never seen before. If you don't keep some data set apart you may have a better model but you have no way of estimating how good it will be when put into production.


How to identify and extract patterns from emails

It depends exactly on which kind of patterns you are talking about. Are they deterministic? That is, they are all the same, so you want to get everything after Dear, or before Att / Best Regards, you can explore regular expression patterns. In python, you can use re library:

https://docs.python.org/3/library/re.html

There are books about regular expressions, so this is a big topic and a strong start if you want to check patterns.

Other kind of tasks you could do:

Word Embeddings on raw e-mails to check word similarity.
Clustering e-mails using Bag-of-Words or Word Embeddings.
Label af set o e-mails and train a supervised algorithm to predict an outcome of interest.

Anyway, text mining is really a big discipline, so if you want a more precise answer, I would recommend to narrow down your question.

Hope it helps.


Is train/test-Split in unsupervised learning of neural network necessary?

Yes it is still necessary, you are fitting your model on that data and learning it to find a good representation for that sample. Validating whether or not this was actually an anomaly is a lot more difficult then.


Present results of the best or the last iteration on dev set?

Is dev set a validation set here? Your best model occurred where the error on the validation set was lowest. That's the model you choose. However the validation error will be a slightly optimistic estimate of the true generalization error. Ideally you have held out a test set as well, and evaluate and report error on that.


What is an example of normalization (making the row unit norm)?

A very simple example is the use of relative frequency instead of absolute frequency, for instance in text classification: if the features represent the raw absolute frequency of every word, then large documents have higher values than small documents. In this case the learning algorithm won't be able to distinguish between a word used 10 times in a small 50 words document (so this word is important for the document) vs. the same word used 10 times but in a large 1000 words document (not very important for the document).
Using relative frequency instead (i.e. dividing by the size of the document) solves this problem. It's an example of row-wise normalization, there are many others.


How can I prevent this model to learn more(less) :)))

Well, by making your model better, I suppose you mean enforcing some sort of regularization on it, so that it doesn't overfit. Early stopping, however can be viewed as a regularization method by itself.

I would suggest sampling the performance of your model more frequently (than the 200 examples you are doing currently) and if you see the validation score is not improving after a few iterations stop the training.

Early stopping can be implemented through the monitor parameter in the GradientBoostingClassifier's .fit() method.


Linear Regression of Sine wave using Gradient descent Not working

The problem seems to be that in the case of frequency parameter, the error function is not convex, and has multiple local minima of value greater than 0. The gradient descent is converging to one of these minima. The error will reduce to 0 only if the starting estimate lies in the right convex basin.

For the amplitude and center value parameters, the error functions are convex, and for phase parameter, there are multiple local minima all of value 0.


Machine Learning & Partial Differential Equations

Neil is correct. There are partial derivatives evwrywhere in gradient computation for machine learning models training.

For instance you can look at the gradient descent method used in the backpropagation method for a neural network. The course from AndrewNg on coursera describes it very well.


How can I get a measure of the semantic similarity of words?

Word2vec does not capture similarity based on antonyms and synonyms. Word2vec would give a higher similarity if the two words have the similar context. Eg
The weather in California was _____ .
The blank could be filled by both hot and cold hence the similarity would be higher. This concept is called Paradigmatic relations.
If you are interested to capture relations such as hypernyms, hyponyms, synonyms, antonym you would have to use any wordnet based similarity measure. There are many similarity measures based on wordnet. You may check this link


"Hadoop" formats for user database: online advertising

Storing user profiles

If you just want to store all user profiles... just save them into normal RDBMS. Assuming one user profiles takes 10Kb of storage, you need only ~9.5Gb for every million of users, which is pretty little and gives you all advantages of mature relational databases.

It makes sense to use HBase only when you have really many users (say, > 1B) or when data is very sparse (most columns are empty). But don't expect it to be as convenient as good old SQL databases.

In advertising, and especially in real-time bidding, very fast retrieval of user profiles is needed. Aerospike becomes more and more popular for this task.

Analysing data slices

Common use of business logs is to analyse specific slices of data, e.g. number of users from France that visited sites from "game" category on November 1-14, 2014. Standard way to manage such data efficiently is to organize them into data cubes. You won't get individual records (e.g. users), but you'll get aggregated statistics really fast.

Such cubes may have many different dimensions, but in 99% of cases they have date field that they are partitioned by. It makes great sense, because almost every query includes time period to get data from.

As for software, Vertica is great for such aggregations. Cheaper* solution from Hadoop world is Impala, which is also great.

(* - if you count only license price)

Machine learning

It really depends on concrete tasks and ML toolkit in use. For real-time bidding you would want blazing fast access to user profile vectors and would probably prefer Aerospike. For online learning Spark Streaming may be used as a data source, and no storage used at all. For offline machine learning there's excellent MLlib from the same Spark project, which works with a variety of sources.


The problem was : X_train and y_train were not numpy array. I transformed them to numpy arrays and it solved the problem.


Identifying “clusters” or “groups” in a matrix

What do you think is distance measure in your case?

I assume there are three dimensions here:

RowN (row number)
ColN (column number)
Value (value: A, B or C)

That means data you get from 4x5 matrix looks like:

Sample1 -> (1, 1, A)
Sample2 -> (1, 2, B)
...
Sample5 -> (1, 5, A)
Sample6 -> (2, 1, A)
...
Sample15 -> (3, 5, C)
...
Sample20 -> (4, 5, A)

Is value scaled? In other words, is A < B < C?

If yes, then

you can replace {A, B, C} with {0, 1, 2} (or may be {10, 11, 12}, if you want this difference be less important than RowN and ColN attributes)
normalize your data
use, for example, K-Means clustering algorithm (http://stat.ethz.ch/R-manual/R-patched/library/stats/html/kmeans.html) from stats R package

In that case the distance between two will be:

Sqrt( (RowN1-RowN2)^2 + (ColN1-ColN2)^2 + (Value1-Value2)^2 )

If value is not scaled (regular categorical variable), use some modifications of K-Means that work with categorical data.

So in case of 100x100 matrix you have 10000 observations and three variables, which is pretty trivial sample size.


Question 1: Is my understanding and construction of the confusion
matrix correct?

Yes, you are correct in your definitions and the way you construct the confusion matrix. The links you have provided also agree with each other. They just switch rows and columns, since there is no hard rule regarding the presentation, as long as the correct relations are maintained.
Link 1 shows this matrix:
          | Pos Class | Neg Class
Pos Pred  |    TP     |    FP
Neg Pred  |    FN     |    TN

Link 2 shows the same matrix, but transposed:
          | Pos Pred  | Neg Pred
Pos Class |    TP     |    FN
Neg Class |    FP     |    TN

Question 2: What is the intuitive difference between Precision and
recall?

Precision is the rate at which you are correct when you predict a positive class. It takes into account all of your positive predictions and figures out which proportion of those is actually correct. When your precision is high, this means that once you make a positive prediction, you are likely to be correct about it. This says nothing about how correct your negative predictions are -- you might make 1 positive and 99 negative predictions on 100 actual positives and still get 100% precision, since your only positive prediction just happened to be correct.
Recall is the rate at which you are able to predict the positive class correctly. It takes into account all of the actual positive classes and figures out which proportion of those you have predicted correctly. When your recall is high, this means that very few actual positives slip by your model without being detected as such. This says nothing about how good you are at being actually correct with your positive predictions -- a model that always predicts a positive class easily achieves 100% recall.
One usually strives to optimize both precision and recall by finding the most acceptable balance between the two. You might want to read this article about the Precision-Recall curve to get a fuller understanding of the relationship between these metrics.

What happens if precision < recall?

As you have highlighted in your post, the two formulas differ only in the denominator. It follows that when precision is smaller than recall, then the number of false positives in your predictions is larger than the number of false negatives.


Passing TFIDF Feature Vector to a SGDClassifier from sklearn

You can find a nice tutorial how to achieve that on this blog
This is acctually a part II. In part I the author discusses how and what is "Term frequency".
Link to part I.


New to data Science. Which techniques best to use Large data set in insurance company?

I think you need to do couple of tests to see what all variables are important with respect to your Target Variable(client can be insured: Yes/No) - this kind of test is called Predictor Importance test.

As you you have mentioned this is sector is new for you, I would suggest you to take all the variable you think are useful. Convert categorical variables to factors using as.factor() numeric variable to numeric as as.numeric(). The reason for explicit transformation is sometimes algorithms cannot understand like:

test1 <- c(1,2,2,4,4,1,2)
summary(test1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  1.000   1.500   2.000   2.286   3.000   4.000
test1 <- as.factor(test1)
summary(test1)
  1 2 4
  2 3 2

Now once the data is ready, you can give them as an input to Boruta, by using this you get the predictor importance graph.
For better understanding you can go through this Link
or else if you want to learn why, how and when you can go through this Link for different tests for different variables.

Boruta does all the above by itself and the outcome is set of important features, With Respect to that you can feed those respective features to your model for getting better results/accuracy.

As your problem is Binary Classification, you can use the following Algorithms:

Decision Trees
Random Forest
Bayesian Network
Support Vector Machines(SVM)
Neural Network

For all the above algorithms I've attached a link for your reference in which you can find at-least one example for each.

Hope my answer is helpful, mark as answer if you got what you needed.


Here is a couple of tools that may be worth a look:

Bart, an open source tool that have been used for several languages, including German. Available from the website
Sucre is a tool developed at the University of Stuttgart. I don't know if it's available easily. You can see this paper about it.


The correct notation would be math.ceil(log(T)). Also what you are asking is/can formally be solved by Binary Search Trees. You can easily traverse it by if-else conditions and arrive at the correct result in math.ceil(log(T)) steps.

It looks something like:

Just place the colour of the balls in the leaf nodes (last nodes). Your first question would be does the ball fall in the first 4 leaf nodes or the second group of 4 nodes.Then you can progress to level 2 and so on.

BST reduces the entropy by maximum amount. You can easily construct other unbalanced trees but then you will require more steps to arrive at a determinate result (in some cases) and less steps in some other cases. BST gives an absolute number of steps you must travel in order to reach the result.

More theory here.


You would need to balance the data according to the joint distribution of all 3 features.

One simple option is to treat each feature as categorical (binning any continuous features), then map each combination of features to a single new feature:

doc_len_tokens  | status   | gender | level
----------------|----------|--------|------
1               | active   | male   | 1
1               | active   | female | 2
1               | inactive | male   | 3
1               | inactive | female | 4
2               | active   | male   | 5
2               | active   | female | 6
2               | inactive | male   | 7
2               | inactive | female | 8
... etc ...

You would then stratify on level when sampling.

Here is an example in SQL:

SELECT
    data.unique_id,
    data.doc_num_tokens,
    data.status,
    data.gender
FROM (
    SELECT
        data.unique_id,
        data.doc_num_tokens,
        data.status,
        data.gender,
        row_number() OVER (
            PARTITION BY
                data.doc_num_tokens,
                data.status,
                data.gender
            ORDER BY
                newid()
        ) AS row_num
    FROM
        my_data AS data
) AS data
WHERE
    row_num <= ${number of rows to sample}


For multi-class classification in SGDClassifier how do I tell if it is using one-vs-rest or one-vs-one by default?

It's there in the User guide [Link]

"....SGDClassifier supports multi-class classification by combining multiple binary classifiers in a “one versus all” (OVA) scheme. For each of the  classes, a binary classifier is learned that discriminates between that and all other  classes. At testing time, we compute the confidence score (i.e. the signed distances to the hyperplane) for each classifier and choose the class with the highest confidence...."


Agglomerative Clustering without knowing number of clusters

A minimum cluster size will not generally be satisfiable in hierarchical clustering. Instead, you have to expect many clusters with just a single point.

ELKI has some fairly interesting techniques to cut a dendrogram. Check the clustering.hierarchical.extraction (or so) package. If I remember correctly, some allow you to set a minimum size (but there will be a "noise" cluster with all the leftovers).


Why my loss is negative while training SAE?

Use a linear output and mean squared error loss, assuming you are predicting normalised pixel intensity values.

Cross-entropy over sigmoid output layer activations can do odd things when the values are not strictly in $\{0,1\}$, depending on implementation.


"Random Forest" variant of other classifiers

The notion of ensembles of models leading to better outcomes is widespread in the Data Science and Machine Learning communities. For example, the widely read text 'The Elements of Statistical Learning' by Hastie, Tibshirani and Friedman, devotes space to the general notion of model averaging (section 8.7-8.8) with further discussion of boosting in relation to trees (e.g. chapter 10). Hence, it is common to see examples of ensembled models in the wild, using any conceivable algorithm as the base, even to the point of effectively seeing ensembles of ensembles.

Specifically with respect to Support Vector Machines, there are at least a few attempts at improving SVM performance by using ensembles. A recent example is EnsemblesSVM (Claesen,De Smet, Suykens, De Moor - see homes.esat.kuleuven.be/~claesenm/ensemblesvm/). The authors promise that this algorithm will lead to better training times by ensembling lightly trained SVMs, and distribute a free implementation via the website above.


You have to tell

pd.read_csv

the separator that you want to use by doing:

pd.read_csv('movies.dat', sep = '::')

This should work.


a$group=="a" gives

# [1]  TRUE FALSE  TRUE FALSE  TRUE FALSE  TRUE

and a$marks[order(a$marks,decreasing=T)] gives

# [1] 8 7 6 5 4 3 2

Combining those to using a logical and (&) does not make much sense here. TRUE & 8 is TRUE - the subsetting may accidentally work. But you want order to look at your subset only, not at all rows in a.

Try for instance

newa <- a[a$group=="a", ]
newa[order(newa$marks, decreasing = TRUE), ]
#   group marks  upd
# 7     a     8 down
# 5     a     6 down
# 3     a     4 down
# 1     a     2   up

or, more convenient and readable:

library(dplyr)
a %>% filter(group=="a") %>% arrange(desc(marks))


Visualize n-dimensional bayesian optimization results

One way to do it is to look at the "trajectory" of the best point as a function of the number of functions evaluations. When the objective function is 2D, you can actually look at contours of the objective functions and plot the best point trajectory at the same time, just like I did here:
http://infinity77.net/go_2021/thebenchmarks.html#solver-solvers-example
For higher dimensionalities, you can try and track a one-dimensional evolution of the objective function and its parameters. Example below with N=3:
import numpy
import matplotlib.pyplot as plt

from scipy.optimize import dual_annealing

def rosen(x, function_history, params_history):
    """
    Rosenbrock objective function.

    This class defines the Rosenbrock global optimization problem. This is a
    minimization problem defined as follows:

    .. math::

       f_{\\text{Rosenbrock}}(x) = \\sum_{i=1}^{n-1} [100(x_i^2 - x_{i+1})^2 + (x_i - 1)^2]

    Here, :math:`n` represents the number of dimensions and
    :math:`x_i \\in [-5, 10]` for :math:`i = 1, ..., n`.

    *Global optimum*: :math:`f(x) = 0` for :math:`x_i = 1` for :math:`i = 1, ..., n`

    Jamil, M. & Yang, X.-S. A Literature Survey of Benchmark Functions
    For Global Optimization Problems Int. Journal of Mathematical Modelling
    and Numerical Optimisation, 2013, 4, 150-194.
    """

    f = numpy.sum(100.0 * (x[1:] - x[:-1] ** 2.0) ** 2.0 + (1 - x[:-1]) ** 2.0)

    params_history.append(x)
    function_history.append(f)

    return f

def solve():

    N = 3
    numpy.random.seed(1)
    bounds = list(zip([-5.] * N, [10.0] * N))
    minimizer_kwargs = {'method': 'L-BFGS-B', 'options': {'maxiter': 100, 'disp': False, 'ftol': 1e-6},
                        'bounds': bounds}

    x0 = numpy.random.uniform(-5.0, 10.0, N)
    function_history, params_history = [], []
    res = dual_annealing(rosen, bounds, x0=x0, seed=1, local_search_options=minimizer_kwargs,
                         args=(function_history, params_history))

    fig = plt.figure()
    ax = fig.add_subplot(2, 2, 1)

    start_f = function_history[0]
    nfun = [1]
    f_progression = [start_f]
    params_progression = [params_history[0]]

    for k, f in enumerate(function_history):
        if f < start_f:
            nfun.append(k+1)
            f_progression.append(f)
            params_progression.append(params_history[k])
            start_f = f

    ax.plot(nfun, f_progression, lw=2)
    ax.set_ylabel('$F(X_1, X_2, X_3)$', fontsize=16)
    ax.set_yscale('log')
    params_progression = numpy.array(params_progression)

    for i in range(3):
        ax = fig.add_subplot(2, 2, i+2)
        ax.plot(nfun, params_progression[:, i], lw=2)
        ax.set_ylabel('$X_%d$'%(i+1), fontsize=16)

    for ax in fig.axes:
        ax.grid()
        ax.set_xlabel('Functions Evaluations', fontsize=16)

    plt.show()

if __name__ == '__main__':
    solve()

You get a picture like the one below:

Not perfect, but one way to do it :-) .
Good luck.


Creating an Artifical Neural Network that produces a set of possible outputs

You don't need to scale the output data. For classification with a ANN the best activation is the softmax function:

f(x) = e^x_i / sum e^x_j

Which normalizes an input vector by applying the exponential function element wise, then dividing by the sum. This produces a discrete probability distribution.

Then your output layer should have 5 output neurons, apply softmax, and you will get a discrete probability distribution over the set [0, 1, 2, 3, 4]. Then you don't need to normalize the data, just do a one-hot encoding. Note that this might differ depending on the framework used.


what actually word embedding dimensions values represent?

These columns are actually arbitrary, they do not represent anything for humans. However, it does not mean they are useless, quite the opposite - computers can extract features from this highly dimensional space more easily, such as in neural networks.

To extract information which is useful for humans, I recommend to have a look at examples in original paper by Tomas Mikolov:

We can have a look at common capital city.  Athens is to Greece (in sense that they have similar relationship) as is Oslo to Norway.
We can use afterwards vectors obtained using GloVe and construct

$$v(Athens) - v(Greece) + v(Norway)$$

and this vector in 'globally' (meaning that trained on universal dataset, such as wikipedia) trained model would be closest to $v(Oslo)$.


Regression and Classification, which is better in financial market price prediction?

From my personal experience, I think what matters the most in terms of return is how good your risk management is. You can use both regression or classification, but all approaches have some errors associated with it, so in the long run, it comes down to how you can manage incorrect predictions.
So you can start with any model, say you go for classification and you achieve 70% accuracy. then you use your model to predict on some historical data (that for sure you have not used during trading) and you build some trading strategy using your forecasts.
Then, you can use something like Pythons Backtrader and check how are you doing in terms of return on historical data. Sometimes its not enough to correctly predict price movement in 9 out of 10 cases, because your one loss can be bigger than all the profitable bets and you also need to take into account fees that will burn your profit. Thats a lot of issues you need to think about before putting this model into production.
And even using this approach you should be aware that historical success of your model+strategy does not guarantee that it will be the same in the future.
To sum it up: I think it comes down to the strategy and not the model itself.


Your reviews column is a column of lists, and not text. Tfidf Vectorizer works on text. I see that your reviews column is just a list of relevant polarity defining adjectives. A simple workaround is:

df['Reviews']=[" ".join(review) for review in df['Reviews'].values]

And then run the vectorizer again. That will fix the problem.


How do I determine if a neural network can be computed by a network that contains only an input layer and an output layer?

You are asked whether we can mimic the given neural network with a new network that contains only a input layer and an output layer, which in turn means whether $a_5$ can be represented as a linear combination of $x_1, x_2, x_3, x_4$. Since every action the neural net does is linear it can be represented as a smaller one without a hidden layer.

Let's calculate the weights

$$a_5 = (a_1*\theta_9 + a_2*\theta_10 + a_3*\theta_11 + a_4*\theta_12)*C $$
$$a_5 = ((x_1*\theta_1 + x_2*\theta_2)*\theta_9*C + (x_1*\theta_3 + x_2*\theta_4)*\theta_{10}*C + (x_3*\theta_5 + x_4*\theta_6)*\theta_{11}*C+ (x_3*\theta_7 + x_4*\theta_8)*C)*\theta_{12}*C $$
$$a_5 = C^2 * (x_1*(\theta_1*\theta_9+\theta_3*\theta_{10}) + x_2(\theta_2*\theta_{9}+\theta_4*\theta_{10}) + x_3(\theta_5*\theta_{11}+ \theta_7*\theta_{12}) + x_4(\theta_6*\theta_{11}+\theta_8*\theta_{12})) $$

So our weights are:
$$<\theta_1*\theta_9+\theta_3*\theta_{10},\theta_5*\theta_{11}+ \theta_7*\theta_{12} , \theta_2*\theta_{9}+\theta_4*\theta_{10}, \theta_6*\theta_{11}+\theta_8*\theta_{12}> $$
and our activation function is multiplication by $C^2$


How to obtain the optimum model using Keras? Deep Learning

A lot of this comes down to just trying different configurations, it can almost be seen as more as an art than a science. However, there are certain rules of thumb that can be applied based on existing research and models. Examples of this are that the Adam optimizer works well enough for most cases (point 6) and convolutional layers generally work better than others (e.g. dense layers) when working with images (point 2). It is therefore always good to see if you can find existing papers that work on problems that are similar to yours and see which approaches are used and what (doesn't) work(s) well.


Is my model overfitting? The validation loss keeps on fluctuating

Overfitting is a situation when model starts to perform more better on training set than on validation (example of such behaviour: loss curves are moving to different ways). According to your plot the model hasn't overfitted. Validation loss seems to fluctuating more than train, because you have more points in training dataset and errors on test have higher influence while loss is calculated.


Can you clarify what you're trying to predict with these responses?

My initial reaction is that with open-ended surveys you'll have a tough time implementing classification algorithms. Open-ended-ness means you don't have a finite feature space and thus you can't do the usual transformation of responses into a feature matrix.

However, there may be other ways to make this work. If you have 5 questions, for instance, you may be able to use sentiment analysis or other methods to come up with metadata about the open ended response that can help you design a classification scheme.


You reinitialized res to 0, which is a scalar, hence the error.


In general, use a storage method that allows you to quickly query it. If your collection is huge, you might need something Lucene-based, like ElasticSearch. If you are a SQL crack and your favorite DB supports it, a full-text index might do the trick. For small sizes like the 5000 documents, even Linux' LocateDb+grep or OSX' spotlight could be enough.

The important point is to be able to quickly verify assumptions about the content of your data - how many documents contain X and Y, does any document contain W but not V, etc.

This will be useful at both the whole set level as well as for analyzing your topic clusters. Finally, a few GNU tools or SQL mastery can also help you profile your document sets more efficiently (n-gram counts/ranks, collocations, concordances, etc)

Edit: that means, for the above reasons and given your collection size, good old plain text (in a file system or a database) might be more efficient than any "fancy" format.


using LSTM encoder decoder framework to generate sentences

I don't know about any research on this topic, but I have thought about similar ideas before. Let's assume the decoder is some RNN that outputs probabilites conditioned on the encoder vector and on what it has generated so far, $P(Word|EncodedVector, DecodedSoFar)$. What you can do is a type of beam search (there is research on this), which is basically greedily take the most likely option so far, but keep all the paths in memory while they still have the potential for a likely enough sentence. The probability of a sequence is just the product of all the predictions that we have sampled. Let's say our model has the following properties (we ignore the encoder):

$P(A) = 0.3$,  $P(B) = 0.5$, $P(End) = 0.2$

Now, condtioned on one sample:

$P(A|A) = 0.6$, $P(B|A)=0.1$, $P(End|A)=0.3$

$P(A|B)=0.7$, $P(B|B)=0.2$, $P(End|B)=0.1$

Now let's say when we have two samples the probability of ending is always 1.

This means we get the following probabilities:

$P(End) = 0.2$

$P(A) = 0.3 * 0.3 = 0.09$

$P(B) = 0.5 * 0.1 = 0.05$

$P(AA) = 0.3 * 0.6 * 1. = 0.18$

$P(AB) = 0.3 * 0.1 * 1. = 0.03$

$P(BA) = 0.5 * 0.7 * 1. = 0.35$

$P(BB) = 0.5 * 0.2 * 1. = 0.1$

But we don't want to iterate over all possibilities, this grows way too quickly. What we can do is build a tree which we can bound by our threshold. Let's say we only want samples with at least probability $0.15$. Our root node is empty, with probability 1. Our terminal nodes are the ones that have $End$ as the last part of the sequence. We start growing our tree with the most likely scenario, which is $B$, which means we have $P(empty)=1$, $P(B)=0.5$. For our next option, we can either look at another start than B or by expanding upon B. $P(A)=0.3$ but $P(BA)=0.35$ so we first expand $B$. We now have $P(empty)=1$, $P(B)=0.5$ and $P(BA)=0.35$. The next step is the highest probability, staying at 0.35 by ending $P(AB-end)=0.35$. The next step is expanding our root by going to $P(A)=0.3$. The next step is expanding root with $P(End)=0.2$. Our next step would be $P(AA)=0.18$ but this is below our threshold. We know there are no sequences that are more likely because of the way we have built our tree, which means we have all sequences above the given threshold. You can use something similar to generate the top k most likely samples.

EDIT: Be aware that this does bias towards shorter sentences if your model allows for them, because every probability decreases the total probability. I have not really thought about how much of this is a problem, it also depends a lot on your generating RNN. You could think of some 'penalties' for short ones but you want the metric to be monotonically decreasing because else you cannot bound your tree properly.


making conclusions after sentiment analysis

Depends what your Goal is.
But generally you see that positive and negative Sentiment probabilities are disjunct. Meaning your model focuses for one class only, negative one. ANd thats it, you conclude your data sample belongs to class "Negative". What that means, depends on the Definition of this class.


Visualizing large number of points as a 3D density map

In Wolfram Language you may use ListDensityPlot3D.

First we create some data to plot using a 3D density function at three time points (0, .5, and 1) using Table.

p[t_, x_, y_, z_] := Cos[t z] + x y
densityClouds = Table[{x, y, z, p[t, x, y, z]},
   {t, 0, 1, .5}, {x, -1, 1, .05}, {y, -1, 1, .05}, {z, π/2, 3 π/2, π/40}];

densityClouds contains the 3 clouds but it is not the correct format for ListDensityPlot3D.

Dimensions@densityClouds

{3, 41, 41, 41, 4}

We can Flatten the two upper layers of each cloud (the x and y coordinates) to the third layer (the z coordinate) to get the correct format.

densityClouds = Flatten[#, 2] & /@ densityClouds;
Dimensions@densityClouds

{3, 68921, 4}

That we have data we can just plot the one of the clouds. The Last cloud is t = 1.

ListDensityPlot3D[Last@densityClouds,
 PlotLegends -> Automatic,
 AxesLabel -> {"x", "y", "z"}]

You may find other list 3D plots in the Data Visualization guide useful (like ListContourPlot3D) and their 3D equivalents in the Function Visualization guide (like DensityPlot3D).

Hope this helps.


Subtracting grand mean from train and test images

There is a kind of bias that you are introducing, yes. You are basically extracting some statistics (i.e. the mean) from your hold-out set and using that to train, which makes your final claims of accuracy a little weaker (some people might say they are useless).

The general approach is to compute the mean of your training data, then you may subtract that from all of the data, including hold-out data.

You can do the mean subtraction, in general, using something like the ImageDataGenerator. The mean to be subtracted can be computed using all or some of the training data. That class also offers other augmentation functionalities, such as normalising the dataset too, adding rotations etc.

you mentioned you read features from a CSV file, so if you are not talking about images, as long as you can use e.g. NumPy, you can perform is manually on all data at the beginning.


Interpreting Decision Tree in context of feature importances

It is not necessary that the more important a feature is then the higher its node is at the decision tree.

This is simply because different criteria (e.g. Gini Impurity, Entropy-Information Gain, MSE etc) may be used at each of two these cases (splitting vs importance).

For example, at SkLearn you may choose to do the splitting of the nodes at the decision tree  according to the Entropy-Information Gain criterion (see criterion & 'entropy' at SkLearn) while the importance of the features is given by Gini Importance which is the mean decrease of the Gini Impurity for a given variable across all the trees of the random forest (see feature_importances_ at SkLearn and here).

If I am right, at SkLearn the same applies even if you choose to do the splitting of the nodes at the decision tree according to the Gini Impurity criterion while the importance of the features is given by Gini Importance because Gini Impurity and Gini Importance are not identical (see also this and this on Stackoverflow about Gini Importance).


Input and output feature shapes in CNN for speech recognition

DNN/CNN prediction(training) is done for 1 frame at a time. The output can be any of the 183 outputs states. Length of the audio files is not a problem since the input to the DNN/CNN is of same dimension only the number of inputs change with audio length.

e.g 1.wav has 500 features and each feature is 39 dimensional and 2.wav has 300 features,
the trained model will take a 39 dimensional input and output will be 183 dimensional. So depending on length we'll get different number of outputs.

Since all the frames in an utterances are tested against all the 183 possibilities so the output always remains 183 dimensional. There is no need to specify number of phonemes an utterance can have as everything is being done at frame level.

Frame concatenation (9-15 frames) is done to leverage contextual properties of speech data. Phone changes are context dependent. For 15 frame context, we change the input of DNN to [7*39 (left_context) 39 7*39(right_context)], a 585 dimensional vector. So now DNN will take 585 dimensional data as input and will output a 183 dimensional vector.

CNN input

  There exist several different alternatives to organizing these
  MFSC features into maps for the CNN. First, as shown in
  Fig. 1(b), they can be arranged as three 2-D feature maps,
  each of which represents MFSC f
  eatures (static, delta and
  delta-delta) distributed along both frequency (using the fre-
  quency band index) and time (using the frame number within
  each context window). In this case, a two-dimensional con-
  volution is performed (explained below) to normalize both
  frequency and temporal variations simultaneously. Alterna-
  tively, we may only consider normalizing frequency variations.
  In this case, the same MFSC features are organized as a number
  of one-dimensional (1-D) feature maps (along the frequency
  band index), as shown in Fig. 1(c). For example, if the context
  window contains 15 frames and 40
  fi
  lter banks are used for each
  frame, we will construct 45 (i.e., 15 times 3) 1-D feature maps,
  with each map having 40 dimensions, as shown in Fig. 1(c).
  As a result, a one-dimensional convolution will be applied
  along the frequency axis. In this paper, we will only focus on
  this latter arrangement found in Fig. 1(c), a one-dimensional
  convolution along frequency

So the input to the CNN will be an image patch of size 45 * 40 regardless of the length of the audio file, just the number of such inputs will depend on the length of audio file.

Why force alignment?

Now we are doing everything at frame level so for each frame we need the state labels. Usually this timing information is not available.

Transcribed data usually looks like this,
1.wav -> I am a cat

Now I don't know how many frames belong to I or to a.

HMMs are trained on this data and force alignment is done to generate state level labels for each frame.

Better modeling of input-output relation gives improvement over other methods. Filter bank features also contribute to improvements. A DNN trained with filter bank features gives better performance as compared a DNN trained with MFCC features.


How to use Autoencoders for outlier detection on images

If you use supervised learning you need to weighted you labels (here).

Otherwise I strongly recommend to exclude every leaking image for a training dataset for an CAE. If you have the anomaly in your trainign dataset, your CAE will learn it as normal. This is actually an active research field, and not trivial for real datsets, but if you have a static image of the pipe, it should be easy. I would suggest that you use traditional CV for it and not an DL model, because you can interpret and fine tune the result.


Does a post-doc in Mathematics help getting a Data Science job?

There was perhaps a time where a math Phd would be an immediate ticket to a data science position but I think that phase has come and gone. The reason? It has a simple answer: Tensorflow.

Tensorflow is essentially all of Google's Phds getting together, writing out the formulas for a large number of data science models and, in the process, that takes a lot of the math out of data science. And this makes sense since most math in data science is "write one code" anyways.

In a similar vein, DataRobot is effectively the best Kaggle competitors in the world getting together and also taking a lot of the math out of data science.

So, I hate to be the bearer of bad news, but the math Phd just isn't attractive to most data science teams.


Unbalanced training data for different classes

The problem you face is commonly called the class imbalance and has been the subject of quite a bit of research. Here's a literature review, if you're interested: He, H., & Garcia, E. A. (2008). Learning from imbalanced data

In particular, you might encounter two forms of imbalance:

Absolute imbalance/rarity occurs when, while you have plenty of data from some classes, you have only a few examples of some other classes (or subconcept of a class). In this case, the issue is that there may not be enough data for the learning algorithm to learn the minority class. In the example you give, with 100 examples of the minority class, depending on the nature of your data, you might have this problem. If you expect to have more data in the future, however, absolute rarity should eventually no longer be an issue.

Relative imbalance, on the other hand, does not go away with more data. You have a relative imbalance when the prior probability of certain classes is much larger than that of some other class or classes. For example, you will always have 10 times as many examples of class 1 than examples of class 2, because class 1 occurs 10 times as often.

Most learning algorithms for classification optimize for accuracy, or something similar like RMSE. This means that, when solving the real classification problem is hard enough, and the data is strongly imbalanced towards one class, the model may resort to predicting the majority class whenever there's a doubt. Recall for the majority class may be great, but not so much for the minority class.

This becomes an issue in many domains where detection of the minority class is particularly important. For example, in medical diagnoses, we may be willing to sacrifice overall accuracy (because of more false positives) in order to have a better true positive rate.

In short, it depends on your domain. Are you OK with optimizing for overall accuracy, or is it more important to have comparable performances across the classes? If you choose the latter, then there are a few things you may try:

Use cost-sensitive learning: Certain learning algorithms and implementations allow you to assign a cost to each class, essentially describing how bad it is if an example of that class is misclassified. If I recall correctly, this is usually considered the best approach when you have a good idea of these different costs.
Rebalance the classes: You can oversample the minority class (which carries a risk of overfitting), undersample the majority class (which is dangerous if you don't have a lot of data), use a mixture of both or perhaps something a bit more advanced like synthetic sampling (attempt to generate new examples of the minority class using something like SMOTE)

Overall, you should also be careful to pick the right evaluation metric. Evaluating your model using accuracy may lead you to believe that your model is performing very well when in fact it is classifying everything into the majority class. There are many metrics you may use, each of which have their pros and cons. The area under the ROC curve (AUC) is a common metric which gives you a general idea of your model's performance averaged over different class misclassification costs. If you can plot the ROC curve for multiple models and notice that one curve dominates the others across the entire width of the plot, then that is the clearest sign that you have a winner. There's a whole chapter on the subject, if you're interested, in the book Imbalanced Learning: Foundations, Algorithms, and Applications.


Difference between stacked lstm vs attention mechanism in LSTM

Stacked LSTM is a special version of hierachical recurrent neural networks, where hard-wired memory and gating units help long-term preservation of state information.
Hierarchy and recurrence have been explored in many works.
One early example is the Neural Abstraction Pyramid, which introduced recurrent computation to hierarchical convolutional neural networks (a.k.a. deep learning).
It incorporates partial interpretations from a larger-and-larger context via horizontal and vertical feedback loops in order to iteratively refine an interpretation.
It was trained using Backpropagation Through Time to solve several computer vision tasks, such as image denoising, superresolution, and object detection.
The recurrent computation is also well suited for maintaining hierarchical state information when processing image sequences.
http://www.ais.uni-bonn.de/books/LNCS2766.pdf


AxisError: axis 1 is out of bounds for array of dimension 1

y_class = np.argmax(y_pred, axis = 1)

here axis=1 represents check max value index column wise
you are facing the error because you are passing vector (size,)
y_class = np.argmax(y_pred, axis = 0)

use above code to remove the error


Extract data from json format and paste to column using python

Given that the data snippet you provided is not a valid json structure I assume that the data is a list with dictionaries in it. Using this assumption, you can extract the values to a nested list using a list comprehension:
data = [
    {
    "id":"list",
    "item":[
        {
            "value":"Hergestellt in Italien aus 100% reinem Platin-Flüssigsilikon"
        },
        {
            "value":"Geruchs- und geschmacksneutral"
        },
        {
            "value":"Kältebeständig bis -60°C"
        },
        {
            "value":"Inklusive Rezeptbuch und 50 Eisstielen"
        },
        {
            "value":"spülmaschinengeeignet"
        }
    ],
    "decorators":[]
    }
]

print([[i["value"] for i in x["item"]] for x in data])

# [['Hergestellt in Italien aus 100% reinem Platin-Flüssigsilikon',
#  'Geruchs- und geschmacksneutral',
#  'Kältebeständig bis -60°C',
#  'Inklusive Rezeptbuch und 50 Eisstielen',
#  'spülmaschinengeeignet']]


During a regression task, I am getting low R^2 values, but elementwise difference between test set and prediction values is huge

This line looks wrong to me:

diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i])

Shouldn't the abs be around the entire calculation?

diff.append(100*np.abs((y_pred[i]-Y_test.values[i])/Y_test.values[i]))

That aside, the RMSE calculation looks accurate and is in the scale of the error, and the $R^2$ is great, so all things being equal, I would lean towards looking for something you did wrong in assessing the errors.  That's why I was focused on your calculation.

One other thought, have you checked for outliers?  This could affect some measures and not others as drastically.


Your question is relatively devoid of details, so all I can really suggest is to use a standard convolutional network classifier.

The problem you seem to be having is that most such models output a one-hot vector, i.e., categorical distribution, corresponding to a single label: $p=(p_1,\ldots,p_n)$, where $\sum_i p_i = 1$ and $p_i$ is the probability of assigning label $i$ (and only label $i$) to the input. Usually this is computed via $p = \text{SoftMax}(f_\theta(x))$ for some input image $x$ and neural network $f_\theta$.

To change this to handle multiple labels, let $q=(q_1,\ldots,q_m)$, where $q_i\in[0,1]$ is the probability of assigning label $i$ to the input. You can think of each $q_\ell$ as parameterizing a Bernoulli distribution over a single label. You can compute $q$ via, for instance, $q = \sigma_E(f_\theta(x))$, where $\sigma_E$ is the elementwise sigmoid function. You can then assign labels to $x$ by, for example, thresholding each $q_j$, i.e., assign label $k$ to $x$ if $x_k \geq T$, where $T$ is a hyper-parameter that can be, say, 0.5.

In other words, take the classifiers on Github and replace the softmax with an elementwise sigmoid, change $n$ to $m$, and use your training data; that's it.

Hopefully that's what you meant.


Does it make sense to train multiple classifiers for multiple partitions of data?

Yes, there is no need for you to manually split the dataset into two parts and train separate classifiers for them.

That's something that decision trees (eg. Random Forest), will do, as you say, under the hood.

The manipulation that you could do on your dataset is probably the dimensionality reduction - but, this is also done, informally, by decision trees. See "Random Forests / Ensemble Trees" section here.


Clustering on imbalanced data that has high correlation

When you have high correlation problems you should go for dimensionality reduction. Multicollinear features can be "summarized" and controlled for by techniques such as PCA or Autoencoders for dimensionality reduction. I don't recomment PCA, since it can only extract factors that linearly associated with your data. In my opinion:

Autoencoders  >  PCA

There are many other dimensionality reduction techniques, but these are the most common.

Once you have reduced the dimensionality of your data you can run your cluster analysis on the reduced dataset, and observe how different observations (and their categories) will be distributed with respect to each other.

Hope this helps, otherwise let me know.


RMSE(Source)

  The RMSE is the square root of the variance of the residuals. It indicates the absolute fit of the model to the data–how close the observed data points are to the model’s predicted values. RMSE is an absolute measure of fit. As the square root of a variance, RMSE can be interpreted as the standard deviation of the unexplained variance, and has the useful property of being in the same units as the response variable. Lower values of RMSE indicate better fit.
  RMSE is a good measure of how accurately the model predicts the response, and is the most important criterion for fit if the main purpose of the model is prediction.

The best measure of model fit depends on the researcher’s objectives, and more than one are often useful.

  The RMSE for your training and your test sets should be very similar if you have built a good model. If the RMSE for the test set is much higher than that of the training set, it is likely that you are badly overffiting the data.


About different structures of neural network

I will answer your questions one by one:

By hidden layer we mean the layer that is inbetween the input and output. If number of layers = 1 with 10 hidden neurons (as shown in second figure) then is it essentially a neural network which is termed as an MLP. Is my understanding correct?

The fundamental building block of a Neural Network is the perceptron. It's modeled on biological neurons: a unit that receives multiple inputs, weights them, and outputs a signal transformed. In its simplest form (i.e. a perceptron with sigmoid activation) it's practically identical to a logistic regression.
When you compose many perceptrons on more than one layer you will form an MLP (Multi Layer Perceptron). The term "MLP" is used as a synonim of Neural Networks - in their basic feed forward architectures.
Neural Networks with no hidden layer cannot even be considered as Neural Networks.
The ones with just one hidden layer are called shallow Neural Networks, no one really uses them, since they are not as powerful as Deep Neural Networks.
When you have more than one hidden layer, then you talk about Deep Neural Networks. They are the real big thing right now, the most powerful ML models available right now.

Do we call the machine learning model as linear or nonlinear ? Or is this term associated to the mapping function ?

ML models can be either linear or non-linear, the choice is yours. Different models will map input and output in different ways (linear or other non-linear ways).
I think it's important to stress that Machine Learning is not a set of models, but an approach to data. You could use a linear regression (i.e. a linear model) with a ML approach, or an SVM or a Deep Neural Network (i.e. non-linear models) to solve the exact same problem.

Which layer's mapping function determines this?

If you refer to Deep Neural Networks, I believe all layers altogether are responsible of the non-linear mapping between inputs and outputs. Non-linearities can be learned only thanks ot a combination of depth and non-linear activation functions. The deeper a Network is, the more complex will be the non-linear patterns that it will able to learn. It's like if you need "all of them at once" to do that, in my opinion.


Best algorithm to create segment in case of categorical variables

Do not approach this by using whatever algorithm you can get to run.

There is no use in just doing something that you don't understand.

Instead, you will first need to identify what is a good way to group data that solves your problem. Then you can begin looking for an algorithm that helps solving your problem (and not some useless other problem).


Effective Methods for Background Removal on Images

This problem used to be solved by analytic methods as you can find here, but these are extremely outdated and boring. However, for simpler cases they work very well and are much faster to implement.

Using deep learning is much more powerful for this particular case and you dont actually need that much data, although you will need to take some time to prepare your data.

The concept

We will train the data on a set of smaller image patches extracted from the original images where the target labels will be foreground and background.

Preparing the data

First, you will need to take your images and draw a mask over them in order to identify your labels. You can do this using paint I suppose. You will take your original images and color the foreground in white and the background in black. This will be labels 1 and 0 respectively.

In Python you will load the original images and their respective label images. You will then split the image and labels into patches of size $k \times k$. You can pick whatever patch size you think is best suited for your kind of data. This is a hyper-parameter you will need to tune using cross-validation. Each patch will have its associated label which is the label of the center of the patch.

Build the model

Then you will build a standard convolutional neural network model where the inputs are going to be the images patches and the output will be the label.

Segmenting new images

To segment new images, split that image into patches and predict the label. All the patches which result in $label=1$ is the foreground.

Alternative method

Alternatively, you can predict the values for the entire patch at once, that means the outputs of your model network will be the same size as your input. You will thus have a label for each pixel in the patch.


There are several steps:

Setup a system that collects tweets from certain accounts.
Preprocess the tweets to create tokens that similar to the tokens in the pretrained model (e.g., lowercase and/or stem).
Create two sets of watch list tokens and tweet tokens. Take the intersection of the two sets.

There are two options for tokens that in tweets but not in pretrained model:

Ignore them.
Explicitly model them as "UNK" / unknown


A quick fix has been published at https://stackoverflow.com/a/50583925/3866828

It works adding

export QT_STYLE_OVERRIDE=gtk2

at the end of the ~/.bashrc file. Then it's a matter of executing
source ~/.bashrc to see the content of the menus the next time that Orange 3 is loaded.


In principle, it is possible to reuse the special tokens as you describe.
However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality.
Another option would be to reuse just the embeddings instead of the whole model.


AR is the abbreviation for Average Recall.
max=1 only looks for 1 detection per image.
Thus, ARmax=1 is the average ability of the model to find a given category in an image. For example if there is a dog in the image, the model should predict a dog in the image. That performance is averaged across all categories.


When combined correlation of features decreases

The only metric you can use to assess the added value of each feature, is the accuracy of your predictive classifier. You can simply include all three feature variables and build your classifier from those. Then you can remove each of the $3$ features and build the three possible classifiers from each of the remaining $2$ features (for example, 'appearances' and 'goals', 'appearances' and 'goals_per_game', and so forth). This analysis gives you the added value of each of the three features to your predictive accuracy. Note the apparent dependency between 'goals' and 'goals_per_game'.
This strategy to feature selection is called sequential backward search. If you have an excess of $10$ features, more advanced algorithms like floating search and MCMC are likely to yield better performing subsets of features. In your case with solely $3$ feature variables sequential backward search should work fine.


May the training sets of two predictive modeling cases (with an overlap in features) be combined?

Well, there is a  obvious trade-off of columns vs rows. Why don't you first check the importance of columns C and D in the individual models? The less relevant they are, the better idea it is to throw them away and comine the tables.
For example, you can try a random forest with sklearn and then check feature_importances_.


Is it possible feed BERT to seq2seq encoder/decoder NMT (for low resource language)?

Sure, why not? An encoder/decoder is basically agnostic to the format of the token vectors, whether they be derived via Word2Vec, BERT, GPT2, etc.

The more challenging aspect of this, should you not have figured it out already, might be finding a pretrained embedding model for your low resource language. Given a small dataset, training your own from scratch seems infeasible. You could potentially find a BERT pretrained over a similar language (e.g. within the same family or grammatical structure) and fine-tune the last layer according to your dataset. Conventional BERT is trained over the English language, with some BERTs spinning off in other languages.


Amount of data needed and hypothesis for SVD

SVD is operation of decomposing a matrix $M$ into the matrix product $M=U\Lambda V$, where $U$ and $V$ are unitary matrices ($U^TU=UU^T=I$, etc.), and $\Lambda$ is diagonal.

So the formal answer is:

No, there are no assumptions about distribution of $M$. It does not to be random at all, it is just a matrix of any nature.
Condition $n>m$, where $(n, m)$ is shape of $M$, is not necessary. Indeed, if you have $n>m$, you can transpose the equation  $M=U\Lambda V$ to get  $M^T=V^T\Lambda U^T$, and it will be also a valid SVD decomposition.

Now the less formal part.

In data mining SVD has two main applications: for computations (like matrix inversion, least squares, etc.), and for dimensionality reduction (e.g. to compress the user-item matrix in recommender systems). For computations, only algebraic properties of SVD (shown above) matter.

For dimensionality reduction, truncated SVD is used: the small elements of $\Lambda$ are discarded, and only $k$ largest are kept. This operation is equivalent to finding a $k$-dimensional hyperplane such that projection of the data ($M$) on this hyperplane is closest (in Euclidian distance) to the original data.

In the last case, the analyst would like the decomposition to generalize well to unseen data, and this problem can be formulated well in probabilistic language. If we state that $M$ consist of $n$ IID $m$-dimensional random variables, then it appears that SVD works best if they are joint normal. That's because:

a. multivariate normal is indeed distributed nearly a hyperplane, and

b. for multivariate normal, Euclidean distance is tightly connected with probability density. Therefore truncated SVD (or PCA, which is mathematically identical) can be viewed as maximum likelihood estiomation of multivariate normal distribution with $k$ independent components. For more details, see the article by Bishop.


Decision trees involve a lot of hyperparameters -

min / max leave
size
depth of tree
criteria for splitting (gini/entropy) etc

Now different packages may have different default settings. Even within R or python if you use multiple packages and compare results, chances are they will be different.

There is nothing which suggests R is "better"

If you want to get the same results, you need to make sure the implicit defaults are similar. For instance, try running the following:

fit <- rpart(y_train ~ ., data = x_train,method="class",
             parms = list(split = "gini"),
             control = rpart.control(minsplit = 2, minbucket = 1, xval=0, maxdepth = 30))

(predicted5= predict(fit,x_test))
    setosa versicolor virginica
149      0  0.3333333 0.6666667

Here, the parameters minsplit = 2, minbucket = 1, xval=0 and maxdepth = 30 are chosen so as to be identical to the sklearn-options, see here. maxdepth = 30 is the largest value rpart will let you have;

sklearn on the other hand has no bound here. If you want probabilities to be identical, you probably want to play around with the cp parameter as well.

Similarly, with

model = tree.DecisionTreeClassifier(criterion='gini',
                                    min_samples_split=20,
                                    min_samples_leaf=round(20.0/3.0), max_depth=30)
model.fit(iris.data, iris.target)

I get

print model.predict([iris.data[49]])
print model.predict([iris.data[99]])
print model.predict([iris.data[100]])
print model.predict([iris.data[149]])
print model.predict([[6.3,2.8,6,1.3]])

[0]
[1]
[2]
[2]
[1]

which looks similar to your initial R output.

All in all, I believe the defaults in R are better suited for the dataset that you are working on, hence the "better" results. But rest assured, they are similar given the parameters are explicit and equal.

Hope this helps!