Correlation between categorical variables based on the target distribution

Your answer is $\chi^2$ (chi-squared) test of independence.

First we have to calculate the expected value of the two nominal variables.  We can calculate the expected value of the two nominal variables by using this formula:

$E_{i,j} = \frac{\sum_{k=1}^cO_{ik}*\sum_{k=1}^rO_{kj}}{N}$

Where,

$E_{ij}$ = Expected value of the cell $i$, $j$

$\sum_{j=1}^cO_{ij}$ = Sum of the ith column

$\sum_{k=1}^rO_{kj}$ = Sum of the kth row

$N$ = total number

After calculating the expected value, we will apply the following formula to calculate the value of the Chi-Square test of Independence:

$\chi^2 = \sum\sum\frac{(O_{ij}-E_{ij})^2}{E_{ij}} $: Chi-Square test of Independence

$O_{ij}$ = Observed value of two nominal variables

$E_{ij}$ = Expected value of two nominal variables

Degree of freedom is calculated by using the following formula:
$df = (r-1)(c-1)$

Where,
DF = Degrees of freedom
r = number of rows
c = number of columns

Hypothesis:

Null hypothesis: Assumes that there is no association between the two variables.

Alternative hypothesis: Assumes that there is an association between the two variables.

Hypothesis testing: Hypothesis testing for the chi-square test of independence as it is for other tests like ANOVA, where a test statistic is computed and compared to a critical value.  The critical value for the chi-square statistic is determined by the level of significance (typically .05) and the degrees of freedom.  The degrees of freedom for the chi-square are calculated using the following formula: df = (r-1)(c-1) where r is the number of rows and c is the number of columns. If the observed chi-square test statistic is greater than the critical value, the null hypothesis can be rejected.


What circumstances causes two different classifiers to classify data exactly like one another

Your results are reasonable. Your data brings several ideas to mind:

1) It is quite reasonable that as you change the available features, this will change the relative performance of machine learning methods. This happens quite a lot. Which machine learning method performs best often depends on the features, so as you change the features the best method changes.

2) It is reasonable that in some cases, disparate models will reach the exact same results. This is most likely in the case where the number of data points is low enough or the data is separable enough that both models reach the exact same conclusions for all test points.


What should I check if model accuracy is no better than baseline level(random guess)

This is a common problem with rare events modelling, and your options are relatively limited (as far as I am aware, at least). It may well be the case that the features you're using are not very informative with respect to predicting these outcomes.

The major issue is that your predictors, in the context of this model, are not very informative. The model tries to balance false positives and false negatives, but with so few true positives any mistakenly-predicted positive outcome will have a large effect on your classification accuracy.

It seems likely in this case that your predictors do not offer enough information to predict outcomes well. You may have reached the ceiling of what this model can do. This could be an artifact of the rarity of the "hired" outcome in your data set, or it could simply be that the relationship between these predictors and the outcome is weak.

There are a few options, involving use of different techniques (like a Firth regression, designed for rare events modelling). But using different predictors may be the best option, if it is possible to do so. Not every event can be modelled well with some arbitrary set of features, and it may be that you've found one of those.


Is it possible to forecast the evolution of cars?

This can be treated as Multivariate Time Series Forecasting, for this you can look into Vector AutoRegression(VAR). Because there might be some association between attributes which we need to take care of hence we can't treat them as separate time series entities.
If you have found some hierarchical relationship between variable you can also look into hierarchical time series forecasting.


Split a list of values into columns of a dataframe?

It looks like you're trying to "featurize" the genre column.

df = pandas.Series([('Adventure', 'Drama', 'Fantasy'), ('Comedy', 'Family'), ('Drama', 'Comedy', 'Romance'), (['Drama']),
                    (['Documentary']), ('Adventure', 'Biography', 'Drama', 'Thriller')]).apply(frozenset).to_frame(name='genre')
for genre in frozenset.union(*df.genre):
    df[genre] = df.apply(lambda _: int(genre in _.genre), axis=1)

The output:

| row | genre                                   | Romance | Documentary | Thriller | Biography | Family | Drama | Comedy | Adventure | Fantasy |
|-----|-----------------------------------------|---------|-------------|----------|-----------|--------|-------|--------|-----------|---------|
| 0   | (Drama, Adventure, Fantasy)             | 0       | 0           | 0        | 0         | 0      | 1     | 0      | 1         | 1       |
| 1   | (Comedy, Family)                        | 0       | 0           | 0        | 0         | 1      | 0     | 1      | 0         | 0       |
| 2   | (Drama, Comedy, Romance)                | 1       | 0           | 0        | 0         | 0      | 1     | 1      | 0         | 0       |
| 3   | (Drama)                                 | 0       | 0           | 0        | 0         | 0      | 1     | 0      | 0         | 0       |
| 4   | (Documentary)                           | 0       | 1           | 0        | 0         | 0      | 0     | 0      | 0         | 0       |
| 5   | (Drama, Biography, Adventure, Thriller) | 0       | 0           | 1        | 1         | 0      | 1     | 0      | 1         | 0       |


How could I improve my FB Prophet forecast?

Data here is bit noisy & has a lot of fluctuations. As few comment suggest apply some transformation on it. I would say get your data in some smaller range & then apply LSTM to predict it. I made time-series work with LSTM with removal of noise by eliminating outliers & it worked with nice further prediction.

RNNs tend to work better with time-series data especially bidirectional-LSTM due to their backwards learning capabilities.


Finding variable values to make equations behave like boolean gate

You can only have $w_0 < 0$. If $w_0 = 0$, then eq 1,2,3 imply that
$$ 0 < w_{1,2} \leq 0 $$
which is a contradiction. Hence option b) is incorrect.


Unsupervised binary classification on small dataset

Correct me if I'm wrong, but your unsupervised classification is not much different than clustering. This is because since it is unsupervised, you do not actually have a user-provided indication of what your classes should be. Thus an unsupervised machine learning system would guess its own classes, and an straightforward way would be to make these classes correspond to the clusters.

In your case you could represent each document as bag-of-words. Then, if necessary, you can do feature reduction (e.g. using PCA). Finally, you could use k-means (in fact 2-means, since you want binary classification). Assigning a new element to any of the two clusters is your classifier.


What is the minimum number of data to create a function?

There are two issues:

Linear regression might be simple to calculate, but it's not sure that the relation between your variables is actually linear. If not, sometimes this simplification is ok because there's not too much variation, but sometime it's completely wrong.
There's no general minimum number of instances valid for every case. Generally the more complex the relation one wants to represent, the more instances one needs.

I'd suggest you start by plotting your data: visualizing the relation between the variables should tell you whether linear regression is a good option, and seeing how scattered your points are should tell you whether you have enough instances.


Class label prediction in keras sequential model showing different results in confusion matrix

Model.predict_classes gives you the most likely class only (highest probability value), therefore it is of dimension (samples,), for every input in sample there is one output - the class.
To be more precise  Model.predict_classes calls argmax on the output of predict. See the code (of predict_classes below)
def predict_classes(self, X, batch_size=128, verbose=1):
  proba = self.predict(X, batch_size=batch_size, verbose=verbose)
        if self.class_mode == 'categorical':
            return proba.argmax(axis=-1)
        else:
            return (proba > 0.5).astype('int32')

As the warning suggest, Please use instead:

np.argmax(model.predict(x), axis=-1),   if your model does multi-class classification   (e.g. if it uses a softmax last-layer activation).
(model.predict(x) > 0.5).astype("int32"),   if your model does binary classification   (e.g. if it uses a sigmoid last-layer activation).


Number of leaves for lightgbm is smaller than categories in one feature

There are different ways to include categorical features, and in many of them a single leaf can combine multiple categories:

With the label, target, or frequency encoding the categorical feature is effectively replaced by a numeric one, so a leaf can include multiple original categories naturally. Conversely, any numeric feature can be thought of as an ordinal categorical feature with an infinite number of possible values.
With the hash or binary encoding a single bit is activated for different categories, grouping them together.
LightGBM also supports categorical features natively. If they are marked in the configuration, LightGBM will consider various ways to partition all the categories of a given feature into two subsets.
If the one-hot encoding is used, LightGBM can choose to combine multiple categories via the Exclusive Feature Bundling (EFB) algorithm. It is discussed in a related question.

Finally, it is also possible that only a few out of these 1000+ categories are really important (in reality or only seemingly due to overfitting).


Is there any way to plot ROC curve for Ensemble hard voting classifier?

For an ROC curve, you need the model to produce an ordering of the test points (ordered by likelihood of being in the positive class).  A few approaches:

A hard voting classifier usually just outputs the final class prediction.  This is an ordering, just with nearly all ties.  That will produce an ROC curve of two straight line segments, from the square's corners to a single point representing the model's specificity and sensitivity.  Not the most informative, but there it is.

While you probably are using a majority vote to produce the final class prediction, you can also interpret it slightly differently: produce a probability score by averaging the votes (as binary) from the base models.  The final class prediction is the same as thresholding this score at 0.5.  So now you have at least fewer ties, and you can produce an ROC curve that looks a little smoother.  This will work best if you have lots of base models, and in fact is pretty much exactly what you do when building random forests.

If your base models come with probability scores, then for each threshold you can get the classifications from all of the base models, and have those vote to produce classes.  Varying that threshold then produces an ROC curve that's just as smooth as any of the base models' ROC curves.  The catch is whether one threshold represents the same thing for all the base models: you need all the base models to be well-calibrated for this to really make sense.  You also lose insight into the potential power of setting different thresholds for each of your base models.


X Second samples taken on unevenly spaced intervals

It depends how you want to cluster the data, but here are some options....

FEATURE ENGINEERING
You could, for example, completely ignore the timestamps and just seek
to cluster the different modes of operation are based on the
magnitude of the feature alone. Here, simply extract the values into
a new feature list.
Otherwise, you have to consider what is important about the samples. For example, does the fact that one snapshot containing 5 samples vs 8 samples distinguish it between modes? If so they, build your feature vector based off counting the number of samples in a day.
In doing so, you will create single features for each day from various attributes in the day (e.g. magnitude, number of samples), enabling you to cluster them based off these features.

RESAMPLING
Conversely, you could resample all the data into uniform intervals, but this would probably need lots of assumptions, and would therefore introduce noise.


What is the formula and log base for idf?

There a a number of variation how to calculate inverse document frequency. Have a look at the wiki page (Tf-Idf) or scikit-learn's TfidfVetorizer class.


I agree with the referee. When your loss is composed from different terms, you should add a mechanism that enable you to adjust the contribution off those terms.
Just create a new hyperparameter (usually called lambda) and use it like that:
$L = L_1 + \lambda L_2$
You have to find the right lambda with cross-validation.
This is used in most of the papers.
For example : https://arxiv.org/pdf/1806.08462v1.pdf page 3


In a truly unsupervised setting, the only possibility is to cluster the documents. Typically this would be done with topic modelling. Then the result could be evaluated by inspecting the words most associated with every topic, and assign a sentiment class to the whole topic/cluster, i.e. all the documents labelled with it.


Interpretation of coefficient on interactive effects term in regression analysis

If the following statement is true, Then it means y should increase as either X1 or X2  increases.

I have 2 continuous variables; X1 and X2, both have a positive
correlation on the dependent variable y (continuous)

This is your final model:

$y=aX_1+bX_2-c(X_1*X_2)$

but I wonder why the regression coefficient c of the interaction term
is negative?  what does it mean?

you have a negative sign before the c coefficient which means it tries to decrease y as X1,X2 increases but the data necessitates that it should increase. Hence c is negative.

why if I add only the interaction term (X1*X2) to the model, the
coefficient is positive while if I add it with (X1) and (X2) the
regression coefficient becomes negative? it doesn't make sense!

By adding just X1*X2 to your does it look like the equation below?

$y=c(X_1*X_2)$

Then it makes sense only if c is positive if not then Y will decrease with increase in X1, X2. It makes sense.


What is the meaning of pct_trend_adjusted in FiveThirtyEight's polling data?

Looking at the methodology provided by FiveThirtyEight, it seems as though this figure (i.e. polling figures on state level) is adjusted for trends in national polls. See differentiator 1 on the linked page for more info:

Basically, in states that haven’t been polled recently, we make
inferences about what’s going on there using national polls or polls
from other states that have been surveyed recently


Yes, it is possible. In my case, with a specific reward function, my agent (snake) preferred to suicide instead of trying to reach the target, because of the "live penalty" (the agent receives a penalty each step, to speed up the exploitation phase over the exploration). Check this blog posts, they go through all of this in detail and they're so far the best ones I have found:
https://mpatacchiola.github.io/blog/2017/01/29/dissecting-reinforcement-learning-3.html

Also, since the author has a background in Cognitive Neuroscience, each algorithm is explained from a neuroscience point of view.


SVR - RMSE is much worse after normalizing the data

You normalized it after splitting into train / test / validation, but you're doing this wrong. You need to normalize the training set X_train_normalized = scaler.fit_transform(X_train), and then use the same statistics to normalize the test and validation sets: X_valid_normalized = scaler.transform(X_valid); X_test_normalized = scaler.transform(X_test).
If you don't do it this way, you're normalizing to different distributions (mean and variance are different). So if you're normalizing to a z-score, replacing each feature value with how many standard deviations from the mean it is, a value of 1.0 that the model sees in the training data has a different original value than a 1.0 in the validation set and test set.
So the right way is to calculate the statistics for the training set (the fit part of fit_transform) and assume the validation and test sets are from the same distribution (you're only supposed to know about the training set at training time). Then use those same statistics on the validation and test sets to make sure you're comparing apples to apples.


How to predict consumer purchase in next 6 months?

This is a forecasting problem, you are predicting a value in the future for a group of people which is the day they would make the next purchase. Sure, you would need the last time each person has bought something ( that's your target ) , periodicity of purchases;number of times that person bought something,obviously the id of the person as variables. And those are the things on top of my head, maybe someone who worked on forecasting problems could help you better.

On a second note : this is how you could approach the problem; maybe encode the dates they made the last purchase as date_num 1 to infinity, and each number in this target variable represents a consecutive day starting from a certain date(maybe the oldest date in your dataset). You would do a regression to try to predict that number using every potential predictor and round it to get an integer representing a unique day.


Replace human judgements with mathematical approach/theories

When it comes to choosing a threshold, I can see 3 approaches:

Make an educated guess

This is what you are currently doing. You pick a value and would need to argue why this is a reasonable threshold. Obviously, the argument is as strong as the assumptions you make.

Unsupervised way

If you compute the matching score for all pairs between A and B, you can analyze the histogram of these scores. Most likely, you will have quite a lot of scores near 0 and a few scores near 100. Then, you can decide to pick a threshold. This itself can be done in different ways. You can pick the median/mean matching score, this becomes close to option 1 but at least, the number comes from a specific mathematical concept. You can use the Jenks-Fisher algorithm (here is a Python implementation). In summary, the algorithm will find a threshold that splits your data into clusters which minimize the intra-cluster deviation. This would be better than the median/mean since it's expected that the data will be quite skewed.

Supervised way

If you somehow have access to which matches are confirmed/overturned by humans, you could use entropy/information gain to find the best threshold. This is similar to building a decision tree of depth 1 (a decision stump), where your input is the matching score and the target is a binary variable (whether or not a human says this is a match).

So your data would be something like this:

score, is_match
0.0, 0
0.1, 0
0.2, 1
0.3, 0
0.4, 1
0.5, 0
0.6, 0
0.7, 0
0.8, 1
0.9, 1
1.0, 1

My personal recommendation would be to use the Jenks-Fisher algorithm (option 2)


Efficient algorithm to find the lowest value cluster in a series of values

Sort your data. That is the fast trick that you cannot do on multivariate data. Then solid for r by three largest gaps or using KDE.

Clustering is more interesting for multivariate data.


CNN, which layer to choose for a similarity measure

For similarity computations, you should generally prefer the last (before softmax) dense layer. This layer will generally capture the most higher level representation of the images. It is also preferable to have the size of vector as small as possible when similarities are to be computed for efficiency reasons.


K-means is based on the assumption that the data is "translation invariant" (more precisely: variance does, and k-means is variance minimization).

In other words, it assumes that a difference of d=(x-y)^2 is of the same importance everywhere. Because of this, k-means does not work on skewed data. Furthermore, because of the square, it is sensitive to outliers and other extreme values.

For salaries and other monetary values, this usually does not hold. The difference between \$0 and \$1000 is massive, and not the same as a salary difference of \$100000 to \$101000. Salaries are usually rather skewed, and you often have some extreme values.

Converting the "user" attribute to a numerical value is outright statistical nonsense. What's variance worth in this attribute? K-means is for continuous numerical data only, and converting data does not chnage the nature, only the encoding - it's still inappropriate.


Dataset where svm performance is significantly different from random forest

Geographical datasets (e.g. when predicting population density from the built environment characteristics in my case) are, at least from my experience, one of the cases where SVM performs consistently worse than random forests. Although I have not investigated the issue in detail, it seems that random forests are more resistant to high amounts of noise and can learn to apply different rulesets in different regions or environments, something which SVM (SVR) fails to do.


Syntax error in pandas dataframe slice in my sklearn data prep?

It is not possible to write numpy style slices in dataframe, like :

observations = data[:6]

Try DataFrame.ix

df.ix[:,6:]

Cf panda indexing : Panda indexing


It seems like you want to reduce the records in the data if Col1 is a subset of another record in Col1. First you can sort the column alphabetically, and then use the function below.
import pandas as pd
df = pd.DataFrame()

df['Col1']  = ['X','X,Y','Z','Z,W']

def test(x):
    return df.Col1.apply(lambda y: y in x).sum()>1

mask = df.Col1.apply(lambda x: test(x))

df[mask]

Apply is the way to iterate through records in pandas. You can nest apply functions to efficiently solve your task.
Note: if there are duplicates in your dataset, it will also mask those. You will need to deduplicate the dataset first with df.drop_duplcates(subset='Col1')
OR
if you want to keep the variability in the other two columns, then you can pass use the apply function of the whole df, with axis=0. See the docs.


Test for feature dependencies in time series modelling

How do I test if an event happening in one location has an impact on events happening in another location (dependency)?

Pearson correlation between the two columns would already give you a simple indication of whether there is a dependency relation.
A $\chi$-square test would tell you whether there is a significant difference between an observed variable (e.g. count in one location) and an expected variable (count in the other location). In other words, it can tell you whether the variables are independent or not.
The conditional probability $p(A|B)$ of a variable A given the other variable B tells you how likely the event A is assuming the event B happens. $A$ and $B$ are independent if $p(A|B)=p(A)$ (note that it's unlikely to be exactly equal in the case of a real sample).

I believe that if an event happening in locations B and C are dependant, I should sum the 2 columns together as 1 feature in my model.

Unless you have a specific reason to do that (e.g. you want to consider a large area which includes locations B and C), this doesn't make a lot of sense:

first dependency is not "all or nothing", two variables can have a certain degree of dependency but it doesn't mean that they follow each other exactly. Therefore you would lose some information by merging them into one feature.
this would make it impossible to predict future events for a specific location, for instance B, if the two values for B and C are combined together.


No, there is not. You will have to use an alternative tool like dask, drill, spark, or a good old fashioned relational database.


Time series regression forecast next month from now (random forest,Lasso,Ridge)

You could try to predict difference $dy = Y_{t+1} - Y_{t}$, not the $Y_{t+1}$ itself. Intuitively, the weather today is good starting prediction for the weather tomorrow.

Secondly, with time series, the seasoning is quite important. You usually would like to use models like ARIMA that takes this into account.

Lastly, it could be just don't enough data to make good predictions. I don't know what is these indexes exactly, but I'm not sure you can use data from one fund to predict another. This leaves you with about 100 points of training data (8 years monthly) and it's not a lot.
I would recommend you to solve 1-d (predict one fund by only it's data) with $dY$ trick, then try ARIMA


How to create a social network like IBM's Watson News Explorer?

They've created a graph from the news articles, topics, and named entities (locations, persons, companies, organizations). There are a lot of things going on here, but k-means is not one of them. If I had to do this I would use a named entity recognition (NER) and document embeddings (doc2vec, etc.). Once you have the embeddings, and the edges (thanks to NER), use a graph layout algorithm like force direction. If the graph is too dense, thin out the weaker edges. If all that sounds Chinese to you, start by reading about "named entity recognition", and "word embeddings". The idea is to attach a number (or rather, a vector) to everything from a word to a document.

Twitter is a different beast. The textual content, such as it is, won't play nice with these embedding algorithms, but you have hashtags and strong social signals; mentions, retweets, and follows. This is pretty complicated, too, so I'll leave you with a relevant paper: Twitter-Network Topic Model: A Full Bayesian Treatment for Social Network and Text Modeling.


ELMo - How does the model transfer its learning/weights on new sentences

ELMo does not lookup the embeddings from a pre-precomputed table as Word2Vec and GloVe. Embeddings from ELMo are hidden states of an LSTM-based language model, i.e., they are computed on the fly when you give a sentence to the network.
ELMo even does use standard word embeddings as the LSTM input. Words are treated as character sequences and those are processed with a 1-dimensional CNN that provides vector representations of the words. The word representations are then passed into two two-layer LSTMs which are trained as a forward and a backward language model repsectively. What you get as a contextual word embedding is a an average of the LSTM states.


Test data for statistical t-test in Python

In Python, to generate random numbers from a certain distribution you would pick the corresponding distribution from np.random (documentation) and pass the corresponding parameters. Thus to draw from a normal distribution you would do

import numpy as np

# for reproducible results, seed the number generator
np.random.seed(42)

n = 100
mu_1, std_1 = 0, 1
mu_2, std_2 = 0.2, 1.5

dataset1 = np.random.normal(loc=mu_1, scale=std_1, size=n)
dataset2 = np.random.normal(loc=mu_2, scale=std_2, size=n)

And the output

print('dataset 1:')
print(f'mean: {dataset1.mean():.2f}')
print(f'std: {dataset1.std():.2f}')
print(f'shape: {dataset1.shape}')
print('--------------')
print('dataset 2:')
print(f'mean: {dataset2.mean():.2f}')
print(f'std: {dataset2.std():.2f}')
print(f'shape: {dataset2.shape}')

--------------
dataset 1:
mean: -0.10
std: 0.90
shape: (100,)
--------------
dataset 2:
mean: 0.23
std: 1.42
shape: (100,)

PS: You don't have to use np.random.seed that's just to make the random generator consistent with the output every time the code is run.

EDIT: Also, if you want to use a t-test on python you can use scipy.stats, thus if you want to calculate the T-test for the means of two independent samples use scipy.stats.ttest_ind, or if you want to calculate the t-test on two related samples use scipy.stats.ttest_rel


Can we calculate iou and mean iou with only images and labels and without train and test csv file

The IoU is a metric that measures the similarity of two sets. In the context of object detection or semantic segmentation, the IoU compares the similarity among the ground truths samples and their counterpart predictions. Thus, what you need (in a general approach, you don't give any details) is:

Samples and their ground truth values (labels) for a given set (train, test, validation...)
The predictions for the samples of the given set
Obvious: in order to get the predictions you need a pre-trained model

Typically, you get the prediction $\hat{y_i}$ of a given sample $x_i$ in a given set, and compare it with the ground truh label $y_i$ with $IoU(\hat{y_i},y_i)$. Then, you can average the IoU over the whole dataset.

Answers

  Are train and test CSV files necessary for calculating IoU (Intersection over Union)

At least one of them, but IoU does not compare the train with the test set. It compares the prediction for a sample with its ground truth.

  Can I calculate IoU with just images and labels?

No, you need the predictions


First I would say the fuzzy clustering is not necessarily a clustering algorithm which uses fuzzy logic.

In machine learning terminology, a "soft" clustering algorithm is called fuzzy clustering as the intuition behind it is not "crisp" (like hard clustering algorithms in which a data point belongs ONLY to one cluster and not others) i.e. for each data point, they output a vector of "probabilities" corresponding to the membership of that point to each cluster (please note that fuzzy memberships are not necessarily probabilities). Even in FCM algorithm, you are just doing a soft version of k-means and it has nothing to do with fuzzy logic (except of the very general idea i.e. not being crisp in making decision)

So you can apply them to any dataset.


This is normally called "Ensemble learning": wiki link

Bayesian Model Combination is a popular modern technique which usually delivers better results than Bayesian Model averaging, on which it was based. The latter had a tendency to give almost all the weight to one model.


How to generate bootstrapping samples in R?

Creating samples

You can create samples using sample and replicate functions.
I create smp with the first given row.

> smp <- c(rep('A', 8), rep('B', 4), rep('C', 2), rep('D', 11), rep('E', 21), rep('F', 0))
> table(smp)
smp
 A  B  C  D  E
 8  4  2 11 21
>
> samples <- replicate(1000, sample(smp)) # creating 1000 samples
> dim(samples) # 1000 samples of 46 votes
[1]   46 1000
>
> samples[,1:3]
      [,1] [,2] [,3]
 [1,] "E"  "A"  "E"
 [2,] "D"  "E"  "E"
 [3,] "E"  "D"  "C"
 [4,] "D"  "E"  "E"
 [5,] "A"  "D"  "D"
 [6,] "B"  "E"  "E"
 [7,] "E"  "A"  "D"
 [8,] "B"  "C"  "D"
 [9,] "B"  "A"  "E"
[10,] "E"  "E"  "E"
[11,] "A"  "E"  "E"
[12,] "D"  "E"  "D"
[13,] "D"  "E"  "B"
[14,] "D"  "B"  "E"
[15,] "E"  "E"  "D"
[16,] "E"  "E"  "E"
[17,] "E"  "B"  "B"
[18,] "A"  "D"  "E"
[19,] "E"  "E"  "E"
[20,] "E"  "E"  "C"
[21,] "E"  "E"  "E"
[22,] "E"  "D"  "A"
[23,] "D"  "D"  "B"
[24,] "E"  "E"  "E"
[25,] "D"  "D"  "A"
[26,] "E"  "E"  "D"
[27,] "C"  "A"  "E"
[28,] "D"  "E"  "D"
[29,] "D"  "E"  "E"
[30,] "D"  "A"  "E"
[31,] "A"  "D"  "E"
[32,] "E"  "E"  "A"
[33,] "E"  "E"  "D"
[34,] "E"  "E"  "A"
[35,] "A"  "E"  "E"
[36,] "D"  "C"  "A"
[37,] "A"  "D"  "A"
[38,] "E"  "D"  "D"
[39,] "E"  "D"  "E"
[40,] "A"  "E"  "A"
[41,] "B"  "A"  "A"
[42,] "C"  "A"  "D"
[43,] "A"  "A"  "D"
[44,] "E"  "B"  "E"
[45,] "E"  "D"  "B"
[46,] "E"  "B"  "E"

If all you need are samples, sample solves your problem.

Bootstraping

To bootstrap you need to compute a statistic. For example, I compute the weighted mean for a table of votes.
I'm using boot::boot to bootsprap.
You must pass the original sample and a handler which receives the original sample s and a vector with the indexes shuffled (idx).
This function returns a boot object that shows the bootstrap statistics.

> boot.smp <- boot::boot(smp, function(s, idx) {
+ tt <- table(s[idx])
+ weighted.mean(tt, as.numeric(factor(names(tt), labels=1:6, levels=LETTERS[1:6])))
+ }, 1000)
> boot.smp

CASE RESAMPLING BOOTSTRAP FOR CENSORED DATA

Call:
boot::boot(data = smp, statistic = function(d, w) {
    tt <- table(d[w])
    weighted.mean(tt, as.numeric(factor(names(tt), labels = 1:6,
        levels = LETTERS[1:6])))
}, R = 1000)

Bootstrap Statistics :
    original    bias    std. error
t1*     11.4 0.4126077    1.274827

The boot object has other methods to compute confidence interval and return much more information.


Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Two possible strategies:

Hard coded rules

Statistical learning

Hard coded rules means creating a mapping from original text to the normalized representations.  The text will have to be manually inspected to find commonly occurring patterns. Hard coded rules are often done with regular expressions.
A statistical learning approach looks for patterns in the data. Bigrams can be defined as the conditional probability of a token given the preceding token. When that probability is above a threshold the two tokens are merged.


What's the intuition behind nonlinear predictors in multivariate regression?

Not quite sure if this is what your question actually is. However, when you have a data generating process such as $y = 3 + 2x$ (with some zero-mean "gaussian noise"), your model could look like:
$$ y = \beta_0 + \beta_1 x + u,$$
and you will find the $\beta_0 = 3$ and $\beta_1 = 2$.
When you have a data generating process such as $y=3+2x+3x^2$ you need to do a "basis expansion" in your model to capture the additional complexity of the function and you could write a model such as:
$$ y = \beta_0 + \beta_1 x + \beta_2 x^2+ u.$$
The second model will capture the non-linear (quadratic) DGP "perfectly".
df = data.frame(x1=seq(-10,20,by=0.2))

df$v = df$x1^2
df$err = rnorm(nrow(df), mean = 0, sd = 1)
df$y = df$v + df$err

plot(df$x1,df$y)

reg1 = lm(y~x1,data=df)
summary(reg1)
lines(df$x1,predict(reg1,newdata=df),col="red")

reg2 = lm(y~poly(x1,2,raw=T),data=df)
summary(reg2)
lines(df$x1,predict(reg2,newdata=df),col="blue")


this can be done in multiple ways i am not one hundred percent understand the question because it is badly worded but there are 3 ways!
if you are doing this for a neural network you can use keras embedding layer
and to create the sequence to feed to this embedding layer you can use one hot and padding from the preprocessing packages of course the sequence can be used regardless of if it is a nn the second thing you can do is use gensim to create word to vector the create a list of words or if you want the most control you can do something like
for i in input_texts:
    for char in i:
        if char not in input_characters:
            input_characters.add(char)

for i in target_texts:
    for char in i:
        if char not in target_characters:
            target_characters.add(char)
            ```
and then

input_token_index = dict(
    [(char, i/num_encoder_tokens) for i, char in enumerate(input_characters)])
target_token_index = dict(
    [(char, i/num_decoder_tokens) for i, char in enumerate(target_characters)])

in your scenario you can just iterate through the sentence with a for loop and then create a dictionary after which you can feed it through a one hot and padding preprocessing from Keras.preprocessing and finally create an embedding


Cluster doc2vec using Affinity Propagation

Both of them convert the distances back to similarities, albeit using different methods. They will, if I recall correctly, also square the distances.

This may be problematic with the most common variant of cosine distance, which already is a squared distance. So it may be a good idea to modify the methods to be able to directly work with the similarities. But you probably need to modify the source code for this (and understand the methods!)


Why is my U-matrix visually not separating the classes?

Is the number of predictor variables in your new dataset large? If so, Euclidean distance can be unreliable due to the curse of dimensionality. You might want to try reducing the dimension of your dataset using Principal Component Analysis or feature selection. Alternatively, you might want to try using a different distance metric, like the cosine similarity.


Found input variables with inconsistent numbers of samples

It seems that I missed the word "scoring". In fact, the extra 3 was related to the number of characters of 'mae'.

def Ridgecv(alpha):
    return cross_val_score(Ridge(alpha=float(alpha), random_state=2),
                           X_train, y_train, scoring='mae', cv=5).mean()


instead of this:

print(clf.predict(df_train_att.head()))
score = clf.score (df_train_att, df_train_lbl)

better try this:

from sklearn.metrics import accuracy_score
test = clf.predict(df_test_att)
score = accuracy_score(test, df_test_lbl, normalize=False)


Calculate mean by decile in Svydesign object

In the Hmisc and reldist packages, you have the function wdt.quantile(). You can calculate your quantiles with this function, and then use cut() to make your groups and then your calculations.


One option is to give each model a unique identifier (e.g., a hash value or nickname). Then store all the metadata in another file.
Another option is using the PyTorch torch-model-archiver feature.


This seems to be a recommendation problem. You can find a nice overview here.


The difference is not related to the length of the named entities. Rather, it deals with how two adjacent named entities of the same type are labeled.

In IOB1 (IOB), B- is only used to separate two adjacent entities of the same type:

Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O  # or I-PER if pronominals are being tagged
ate      O
lasagna  O

In IOB2, all entities begin with B-:

Today    O
Alice    B-PER
Bob      B-PER
and      O
I        O  # or B-PER if pronominals are being tagged
ate      O
lasagna  O

See Wikipedia


How to compute denominator in Naive Bayes?

in the Examples section of the Wikipedia article there is a nice example.  The calculation of $p(\mathbf{x})$ can be done via

$$p(\mathbf{x}) = \sum_k p(C_k) \ p(\mathbf{x} \mid C_k)$$

Note that using the conditional independence assumption of the Naive Bayes one can write

$$ p(\mathbf{x} \mid C_k) = \Pi_{i} \, p(x_i \mid C_k) $$


What do you call the ratio of positive to negative samples?

You can use class ratio / sample class ratio. Which will make it more intuitive for any reader while going through the details.

As its not used for model performance analysis hence I think we don’t have a metric name for this.


How should clickstream data be prepared before user segmentation can be performed?

This sounds like a time-related predictive task...i.e. forecasting. The data you have available is event-driven, e.g. data is recorded when a user clicks on a link/refreshes a page/clicks on a Google ad etc. This means that you need to treat your data as if it were a time-series for each user. This would be easier if you had access to ESP (Event Stream Processing) but let's say that you do not. Therefore, you need to identify each user, aggregate their info/features so that you have features identifying the number of unique pages they visited, the number of sessions they had etc. etc. Then you can segment these users and you may or may not derive useful segments from them. You could even go a step further and do session analysis whereby you identify unique sessions and cluster/segment user sessions looking for nice visual patterns or anomalies.


What is a good objective function for allowing close to 0 predictions?

Please be aware that neural networks are normally badly calibrated. Essentially this means that for a binary classification, neural networks are good at keeping the prediction score of a sample in the right area (above 50% or below 50%, depending on the class), but the actual values do not have to represent plausible probabilites of the real world.
There is a lot of research going on about how to calibrate neural networks, e.g. this approach.


How to describe the process of using existing weights for training a deep learning model?

Yes it is definitely doable, and further more, its is recommended in many scenarios. For instance, if you wanted to train some generic image classifier it would be naive idea to start from scratch and, for instance, retrain ImageNets' dataset for 2-3 weeks simply trying to retrieve optimal weights that have already been previously computed.

if you simply take pre-trained weights and use them on a different domain, then that would be referred to as transfer learning. However, since you want to retrain the existing weights with additional training data then, yes, you would be "fine-tuning" the model. When you do this however, there are considerations that need to be taken. Do you want to fine tune every layer? Or only deeper layers that detect very specific features of the dataset? Considerations like these should be taken into account.

Also check out http://cs231n.github.io/transfer-learning/


Yes,gradient descent (GD) and Stochastic Gradient Descent(SGD) are Optimization functions.Objective of a Optimization function is  to give the best combinations of parameters for our algorithm which minimize the error.Support Vector Machine is the Classification algorithm and GD/SGD are the optimization functions use to find best optimal hyperplane(separator).

Gradient Descent’s cost-function(total error of the training set) iterates over ALL training samples

Stochastic Gradient Descent’s cost-function only accounts for ONE training sample, chosen at random


Sklearn Aggregating Multiple Fitted Models Into A Single Model? (binary classification)

Despite the downvote, the question is clear, and a common one I'm sure most stumble across after doing machine learning work for some time.

The goal was to make a stronger predictive model from multiple trained models.

Quote from my question:

  is it possible to aggregate these 30 fitted models into a single
  model?

Answer: yes but there's no good functionality that allows you to do this in sklearn.

Verbose answer:

Imagine you have 30 CSV files that contain 15,000 0 class and 15,000 1 class samples. In other words, an equally balanced number of binary responses (no class imbalance). I generated these files myself because 1) the size of the data I'm working with is too big to fit into memory (point #1 of my original question) and 2) contains 97% class 0 and 3% class 1 (large class imbalance). My goal is to see if it's even possible to distinguish between a 0 or a 1 if the class imbalance issue was removed from the equation. If there were distinguishing features found, I'd want to know what those features were.

To generate each batch, I grabbed 15,000 1 responses, and randomly sampled from the 97% 15,000 more samples, joined into one dataset (30,000 samples total), then shuffled them at random.

I then went through each batch and trained an XGBClassifier() using optimal parameters found from GridSearchCV() for each. I then saved the model to disk (using Python's pickle capabilities).

At this point, you have 30 saved models.

/models/ directory looks like this:

['model_0.pkl', 'model_1.pkl', 'model_10.pkl', 'model_11.pkl', 'model_12.pkl',
 'model_13.pkl', 'model_14.pkl', 'model_15.pkl', 'model_16.pkl', 'model_17.pkl',
 'model_18.pkl', 'model_19.pkl', 'model_2.pkl', 'model_20.pkl', 'model_21.pkl',
 'model_22.pkl', 'model_23.pkl', 'model_24.pkl', 'model_25.pkl', 'model_26.pkl',
 'model_27.pkl', 'model_28.pkl', 'model_29.pkl', 'model_3.pkl', 'model_4.pkl',
 'model_5.pkl', 'model_6.pkl', 'model_7.pkl', 'model_8.pkl', 'model_9.pkl']

Again, going back to the original question - is it possible to aggregate these into a single model? I wrote a pretty basic python function for this.

def xgb_predictions(X):
  ''' returns predictions from 30 saved models '''

  predictions = {}
  for pkl_file in os.listdir('./models/'):
    file_num = int(re.search(r'\d+', pkl_file).group())
    xgb = pickle.load(open(os.path.join('models', pkl_file), mode='rb'))
    y_pred = xgb.predict(X)
    predictions[file_num] = y_pred

  new_df = pd.DataFrame(predictions)
  new_df = new_df[sorted(new_df.columns)]

  return new_df

It iterates through each file in a directory of saved models and loads them one at a time casting its own "vote" so-to-speak. The end result is a new dataframe of predictions.

Since this new dataset is smaller (X.shape x 30), it will fit into memory. I iterate through each batch file calling xgb_predictions(X), getting a dataframe of predictions, add the y column to the dataframe, and append it to a new file. I then read the full dataset of predictions and create a "level 2" model instance where X is the prediction data and y is still y.

So to recap, the concept is, for binary classification, create equally balanced class datasets, train a model on each, run through each dataset and let each trained model cast a prediction. This collection of predictions is, in a way, a transformed version of your original dataset. You build another model to try and predict y given the predictions dataset. Train it all in memory so you end up with a single model. Any time you want to predict, you take your dataset, transform it into predictions, then use your level 2 trained model to cast the final prediction.

This technique performed much better than any other single model did or even stacking with UNTRAINED prior models.


Differences in scoring from PMML model on different platforms

The difference was, it appears, due to the different implementation of Random Forests in R and Cascading Pattern (as well as openscoring which I tried later) with respect to ties in the tree voting - i.e. when an even number of trees are built (say, 500) and exactly half classify an application as Good, and the other half as Bad, the handling of those situations differs. Solved it by growing and odd (501) number of trees.


redundancy or functional dependency of two columns

Well, it always depends, for example, on what model you might be training (i.e. some are robust to multicollinearity). I am pretty sure you are aware, but to have it said as a rule of thumb it is always helpful if you know what you are looking for, rather than hoping naively one function or method would give all the answers.

Said that, there are good progress and in fact a powerful Python package pandas-profiling that can simply takes a pandas dataframe and returns lots of useful information in a blick of an eye (well if it is not super large dataset). And yes, for redundancy, correlation is a good start, and pandas-profiling highlights of highly correlated variables based on Spearman, Pearson and Kendall matrices. Take a look at this post for quick read about pandas-profiling.

For categorical columns, often things get trickier. In past, pandas-profiling did not offer anything. I just double checked, and interestingly they implemented Cramer’s V at this Closed Issue at their repo (see this nice post 'The Search for Categorical Correlation' if you want to learn about Cramer’s V4). I cannot confirm its functionality, you gotta test it on your own, but I believe it should be reliable as many contribute to this project. Unfortunately their documentations isn't thorough, but it seems it is there, see correlations.html page.

Update: Just figured that they have an advanced tutorial under Examples i.e.  link to the Colab notebook Tutorial: report structure using Kaggle data (advanced), where the dataset is a combination of many data types including numerical and categorical, and they in fact show all the correlations elegantly, see a screenshot:

Good luck!


What is a good way to store processed CSV data to train model in Python?

With 100MB data, you can store it in any filesystem as CSV since read is going to take less than a second.

Most of the time is going to be spent by dataframe runtime in parsing data and creation of in-memory data structures.


What does these points mean in Naive Bayes?

Naïve Bayes is robust to irrelevant features. What does this mean? Can anyone give an example how does the irrelevant features cancels out and what are the irrelevant features?

Imagine a classifier for sentiment analysis. For a strongly positive word like $w=great$, the conditional probability $p(w|pos)$ is going to be quite high whereas $p(w|neg)$ is going to be quite low, so the posterior $p(pos|d)$ for a document $d$ containing this word is likely to be much higher than $p(neg|d)$.

Now what happens with a neutral word $w=today$? Neither $p(w|pos)$ or $p(w|neg)$ is going to be much higher than the other. So all other things being equal, the difference between the two posterior probabilities is not going to depend much on this word compared to other more relevant words, for instance "great".

  It is optimal if the independence assumption holds. Can anyone give an example of independence assumption not holding? I think it would be related to presence of words like Hong Kong, United Kingdom etc in a sentence.

In practice the independence assumption almost never holds with real data. For example words in a text actually depend on each other, that's how they make sense together in a sentence. This is true for entities like "Hong Kong" but also for virtually any sentence. For instance "I love chocolate but you hate it" doesn't mean that same as "You love chocolate but I hate it", or "it chocolate you hate love but" which means nothing. NB will treat all these variants the same way: basically the model assumes independence for the purpose of making things simpler and easier to compute, and it turns out that it works quite well in general, despite the massive simplification.


The first part before the underscore can be entered in this URL:

http://www.image-net.org/api/text/wordnet.synset.getwords?wnid=n00004475


Using the whole dataset for testing (not validation) in case of small datasets

Considering that you have not used a cross-validation strategy, you could try to use the LOOCV (Leave One Out Cross Validation) strategy, so you have several splits (as many as samples considered in your training dataset) and still leave some samples as a final never-seen-before validation set.
This is costly from a performance perspective, but interesting for small datasets as in your case. Another option, in case the dataset is not really small, is usin k-fold cross-validation (with k being 5, 10...).
A post with practical info can be found here and a related topic discussion can be found here


How to separate words that are together in a large data set

This commonly called a "word break" problem. There are a variety of approaches, the most common use dynamic programming or tries. You can recursively try candidates and keep the candidates if they can split the entire string.

Here is a version (inspired by this answer):

from functools import lru_cache

@lru_cache(None)
def wordbreak(string):
    if len(string) < 1:
        return " "
    else:
        for i in range(len(string)):
            substring = string[:i+1]
            if substring in valid_words:
                remaining_split = wordbreak(string[i+1:])
                if len(remaining_split) > 0:
                    return substring + " " + remaining_split
                else:
                    return ""
            if (i == len(string) - 1) and (substring not in valid_words):
                return ""

valid_words = {'man', 'boycott', 'bears'}
assert wordbreak('boycottbears') == 'boycott bears'
assert not wordbreak('abc')

However, creating a complete collection of valid words is impossible for Twitter data since there are always new words.


NLTK

A great starting point for keyword extraction is the NLTK (natural language toolkit) library.  To extract keywords, you probably need to tokenize your data, breaking each word out into a token, and ignore the most common or unimportant words known as "stopwords".  Assuming you're searching for keywords across a large number of query results, identify the most important terms in each document using TF-IDF (term frequency–inverse document frequency).  There are tools and tutorials for this in the NLTK documentation.  Sort the resulting token-scores, choose the highest scoring tokens, and these are a good start at your keywords.


Blast. It looks as though ARIMAX is currently a work in progress for the cloudera/spark-timeseries community. Although some work has been done on the model development side, it doesn't appear to have been merged in.


Data balance -before or after feature selection/engineering

Balancing is using which of the samples to consider in the data set (adding/reducing rows).
Features selection and feature engineering is removing and adding information about each sample (adding/removing columns).

Therefore, you have the liberty about how to combine them.
Many times, the order doesn't matter.

However, if you are normalising features, they will be normalised with respect to the data set and you might get different results before and after balancing.
Feature might change it's importance after balancing, influencing the feature selection.

In general, if you are modeling on a data set, it is better to adapt the feature engineering to it so balance first and do feature engineering later later unless you have a reason to do otherwise.


Rather than using MinMaxScaler, I think StandardScaler would be a better option if you have outliers that are quite different from most of the values, but are still valid data.

StandardScaler is used the same way as MinMaxScaler, but it will scale your values so they have mean equal to 0 and standard deviation equal to 1. Since those values will be found based on all the values in the series, it is much more robust against outliers.


In random forest, what happens if I add features that are correlated?

Random forests don't suffer from correlated variables like linear regression models do. Random forests randomly pick from a subset of variables at each split (hence the "random" in "random forests"). This means that correlated variables are less likely to show up together when the trees are being trained. But even when correlated variables show up in the same random subset of variables, it's still not much of an issue because the variables aren't assigned coefficients.

Correlated variables are mostly an issue for linear models that try to hold all other variables constant when calculating coefficients during training. The variable selection process is much simpler for trees and tree-based algorithms like random forests and gradient boosting. When a random forest is being trained and a tree's split is being evaluated, the algorithm will simply pick whichever feature most reduces error on that particular split of the tree. Once a variable is picked, there is no coefficient, just a greater-than/less-than split point, so the problem of "exploding coefficients" doesn't apply.


Does Naive Bayes Classifier require assuming a specific distribution for training?

The most harrowing problem would be "interpolation". That is, computing $p(x_i | y)$ for $x_i$ and $y$ you have not yet observed, but fall within the convex hull of your observations.

You have to bin events together, or come up with a wacky interpolation scheme. This does work. However, If you assume a distribution, $p(x_i | y)$ becomes continuous, and you end up with a more statisically sound means of interpolation.


How to solve this ? Value Error : operands could not be broadcast together with shapes (0,) (100,522)

This should be on stackoverflow, with a precise description of the error (line, traceback...), but the error seems to be here:

np.empty((0,3),float)

And then not appending the data as expected.

If you are doing this, use a list, and then at the end, transform your data in a numpy array:

data = []

for...
    data.append(new array)
data = np.array(data) # add the Fortran flags if you need to

There would be just one memory copy instead of O(n-squared) with np.append.

Also the shape for X or Y (?) seems to be 2D ((100,522)), where as you only have 1D. Check your data (which we don't have, so impossible for me to tell you exactly where to look for) so that you can ensure that the dimensions are correct.


Why use Gradient Descent when Gradient just solve the problem? (With Neural Nets)

You might be mistaking the gradient itself with the mathematical approach to find the critical points of a differentiable function.

In the latter approach, you take the derivative of the function with respect to its parameters and find the values of the parameters that make it zero. These points in parameter space are critital points, that is, where the function has either a local maxima, a local minima or a saddle point. In order to know the actual type, you have to take the second derivative.

The problem with applying such an approach to neural networks is not finding the gradient of the loss function with respect to the parameters; currently we have automatic differentiation software that computes the gradients for you. The actual problem is solving the equation where such gradient equals to zero. We don't know how to do that but in the trivial cases. Furthermore, this only gives you local optima, not global ones.

A solution to that problem is using numerical optimization techniques, like the gradient descent family. They basically explore the parameter space in the descending direction of the gradient of the loss function, hoping to reach a minimum. This cannot not assured for non-convex problems. Nevertheless, in practice it works quite well.

The reasons why gradient descent techniques work well in non-convex problems are an active line of research.


What does SpatialDropout1D() do to output of Embedding() in Keras?

Basically, it removes all the pixel in a row from all channels.
eg: take [[1,1,1], [2,4,5]], there are 3 points with values in 2 channels, by doing SpatialDropout1D it zeros an entire row ie all attributes of a point is set to 0; like [[1,1,0], [2,4,0]]

number of such choices would be 3C0 + 3C1+ 3C2 + 3C3 = 8

The intuition behind this is in many cases for an image the adjacent pixels are correlated, so hiding one of them is not helping much, rather hiding entire row, that's gotta make a difference. reference material


How to feed a table per timestamp to LSTM neural network?

Your data format is

                         feature1   feature2   target
             product1    1          12         2
 timestamp   product2    2          6          3
             product3    4          3          4

There are two designs for two assumptions:

Products are not related to each other. Therefore, each product could be modeled separately. That is, each timestamp is X(t) = [feature1, feature2] or, including the target, X(t)|y(t) = [feature1, feature2, target]. And we build a model for each product separately. In summary, LSTM receives two  1 x 3 sequences for t-1 and t, and outputs a 1 x 1 target for t + 1. In notation:
$$(\overbrace{X_{t-1}|y_{t-1}}^{1 \times 3}, \overbrace{X_{t}|y_{t}}^{1 \times 3}) \rightarrow \overbrace{y_{t+1}}^{1 \times 1}$$
Products are related to each other, meaning product1 can help product2 to predict its target. For this, we just need to flatten the 3 x 2 matrix to a 1 x 6 vector, where the order of values does not matter. That is,

X(t) = [product1_feature1, product1_feature2, ..., product3_feature2]

or

X(t) = [product1_feature1, product2_feature1, ..., product3_feature2]

We can also add the targets, for example

X(t)|y(t) = [product1_feature1, product2_feature1, ..., product3_feature2, target1, ..., target3]

This way, dimension of each timestamp would be 9 (6 + 3), and a sequence of two timestamps would be

[
  [product1_feature1, product2_feature1, ..., product3_feature2, target1, ..., target3], # t-1
  [product1_feature1, product2_feature1, ..., product3_feature2, target1, ..., target3]  # t
]

corresponding to a three dimensional target [target1, ..., target3] at t + 1.

In summary, LSTM receives two 1 x 9 sequences for t-1 and t, and outputs a 1 x 3 target for t + 1. In notation:
$$(\overbrace{X_{t-1}|y_{t-1}}^{1 \times 9}, \overbrace{X_{t}|y_{t}}^{1 \times 9}) \rightarrow \overbrace{y_{t+1}}^{1 \times 3}$$


Accuracy of KFold Cross Validation for Neural Network

For some context, the entire point of running K-Fold cross-validation to compute an estimate of the performance. If the obtained values vary a lot, it just means that your estimate is less precise (i.e. has a larger standard deviation)

In a neural network setting, your network evolves, so obviously, averaging all folds might give you a less accurate estimate if the network learned a lot in the last few iterations for instance.

Typically, you would always take the fold at the end of training, since this is the performance you currently have.

However, it is good to note that analyzing the variance of your performance within a short window can help you understand whether or not your model has converged or whether it is still "exploring".


Getting a balanced sample across many variables

If I understand correctly, you want a sample with along each feature the value below which a given percentage of observations in your population falls. If so, you might want to try something like this using np.percentile:
import numpy as np
import pandas as pd

data = [ np.random.randint(1,10,6) for i in range(20) ] # fake data
df = pd.DataFrame(data=data, columns=['A', 'B', 'C', 'D', 'E', 'F']) # fake dataframe

# 15 being your desired percentage
balanced_sample = [ np.percentile(df[col], 15) for col in df.columns ]


How do I deal with changing values in a categorical variable when I am aggregating customer records

The standard option would be to create one feature for each possible pair (variable, category) and use the frequency of this particular pair as value for this particular customer. If the number of times doesn't matter, it could be transformed into a binary feature, i.e. just indicating whether this customer has ever been seen with this category.

In case some categories appear very rarely, it's usually counter-productive to include them so you could have a minimum frequency in order to filter out rare pairs.


Why is my keras cnn model giving me the same result for every prediction?

You are applying a preprocessing step to both your train and test images: rescale=1./255. This normalises the values of the image pixels to be in the range of (0,1) instead of (0,255).

However, when you are doing your predictions you are not applying this rescaling to your test image. This translates into testing an image with pixel values 255 higher than your model is expecting, thus always predicting the same (always a 1 I gues...) You should rescale your test samples as well. Just try:

print(classifier.predict(test_image/255))


Overfitting results with Random Forest Regression

Where are you evaluating the performance of your algorithm?
Are you making a train test split and evaluating in the test split? It might be that you overfitted your train and you are just measuring the accuracy there.
If you have made correctly the train/test split and the evaluation it could be that the images that you are predicting do not have the same properties/configuration/topology than the with you are trainning


Which algorithms or methods can be used to detect an outlier from this data set?

You can use BoxPlot for outlier analysis. I would show you how to do that in Python:

Consider your data as an array:

a = [100, 50, 150, 200, 35, 60 ,50, 20, 500]

Now, use seaborn to plot the boxplot:

import seaborn as sn
sn.boxplot(a)

So, you would get a plot which looks somewhat like this:

Seems like 500 is the only outlier to me.  But, it all depends on the analysis and the tolerance level of the analyst or the statistician and also the problem statement.

You can have a look at one of my answers on the CrossValidated SE for more tests.

And there are several nice questions on outliers and the algorithms and techniques for detecting them.

My personal favourite is the Mahalanobis distance technique.


This is an open issue in Reinforcement Learning (and all of machine learning). Google has published a paper titled "Machine Learning: The High Interest Credit Card of Technical Debt" which addresses the many ways these systems can degrade over time.

One way is to follow fundamental production engineering techniques (e.g., test coverage and graceful exit).

Another way is to monitor the data, then retrain the agent if new data is statistically out-of-sample.


Should kernel size always be a prime number?

Odd numbers, usually, but not necessarily. This can depend on the problem at hand. The odd length of the side of the kernel is used in order to emphasize the center of the kernel. For a more detailed discussion of the geometry of kernels and convnets have a look at this thesis.


Intuition / Importance of intermediate supervision in deep learning

Unfortunately there is no mathematical proof whether or not the intermediate supervision improve the performance of the network.

However stack hourglass article shows that according to ablation study: using more encoder – decoder units with intermediate supervision improve the performance – please have a look at the figure below.

In that case – the remaining network doesn't perform identity transformation (although there is a residual connection between the remaining networks).

My intuition is the every encoder – decoder serves as a PCA,  that keep the most dominant feature maps.  A series of encoder – decoder let the network reassess the feature maps until it get its most discriminate feature maps at the end.


How can I restart model fit after x epochs if loss is high?

While you should be able to do that one way or another, it won't be a good solution to your problem described here and the other question. If the difference between runs are very high - ie it depends how you split your dataset, that probably means that there is something inherently wrong with the dataset or the way it is split into training and validation sets.

Somehow you might be able to get a good result on training and validation data and take that as your base model and use for making predictions, but that does not mean that it is the best result or even an accurate description of your model. For this purpose, you need to spare a third dataset to check the accuracy: splitting the entire dataset as (training/validation/test)

I know this does not actually answer your question, but I thought this would help your process.


Can GLM( generalized linear method) handle the collinearity between the predictor variables in a regression-analysis?

“It is said that the classic linear regression assumes there is no correlation between its independent variables”
Depending on your goal of doing the regression, this common statement is false.
Even with multicollinearity, you get that $\hat{\beta}=(X^TX)^{-1}X^Ty$ is the solution to the OLS optimization.
Even with multicollinearity, you get that $\hat{\beta}=(X^TX)^{-1}X^Ty$ is minimum variance linear unbiased estimator from the Gauss-Markov theorem.
What the Gauss-Markov theorem requires is that the error terms not be correlated. This is commonly confused for saying that the predictors are not correlated, but that is indeed a mistake.
There can be numerical instability when you do math on a computer, particularly as you approach perfect multicollinearity ($X^TX$ is close to singular, singular in the extreme case of perfect multicollinearity or correlation of $1$ between variables), but there is no inherent problem with multicollinearity if your goal is to predict.
Where multicollinearity can hurt is when you want to do inference on the parameters, which is rarely a goal in machine learning. When you have multicollinearity, the parameter standard errors are inflated, sapping away your power to tell that they are not zero. Philosophically, it also makes it hard to attribute an effect to a particular predictor if it is correlated with others. (Imagine a hospital wanting to know if it pays its neurosurgeons as much as its heart surgeons and sees that the heart surgeons make way more but also sees that the heart surgeons have much more experience. Do they make more because of their specialty or because of their experience?)
Multicollinearity also can mean that you might be able to use a smaller amount of variables to get almost as much information as the entire set of variables. For instance, if two predictors are highly correlated, it might not be worth including both; you might be better off leaving out one for the sake of model parsimony and having fewer parameters in your regression, but that is an empirical problem and up to the model-designer’s judgment.
Getting to the full GLM framework, the Gauss-Markov theorem does not apply, but the idea remains that there is no inherent issue with multicollinearity when your goal is to predict instead of to do parameter inference, which is the typical goal in machine learning.


What does the notation mAP@[.5:.95] mean?

mAP@[.5:.95](someone denoted mAP@[.5,.95]) means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

There is
an associated MS COCO challenge with a new evaluation
metric, that averages mAP over different IoU thresholds,
from 0.5 to 0.95 (written as “0.5:0.95”). [Ref]

We evaluate the
mAP averaged for IoU ∈ [0.5 : 0.05 : 0.95] (COCO’s
standard metric, simply denoted as mAP@[.5, .95])
and mAP@0.5 (PASCAL VOC’s metric). [Ref]

To evaluate our final detections, we use the official
COCO API [20], which measures mAP averaged over IOU
thresholds in [0.5 : 0.05 : 0.95], amongst other metrics. [Ref]

BTW, the source code of coco shows exactly what mAP@[.5:.95] is doing:
self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
References

cocoapi

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and
Recurrent Neural Networks

Faster R-CNN: Towards Real-Time Object
Detection with Region Proposal Networks

Speed/accuracy trade-offs for modern convolutional object detectors


Here is one example for you to consider.
from sklearn.neighbors import KNeighborsClassifier
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

url = "https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data"

# Assign colum names to the dataset
names = ['sepal-length', 'sepal-width', 'petal-length', 'petal-width', 'Class']

# Read dataset to pandas dataframe
dataset = pd.read_csv(url, names=names)

dataset.head()

X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, 4].values

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(X_train, y_train)

y_pred = classifier.predict(X_test)

from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

classifier = KNeighborsClassifier(n_neighbors=3)
classifier.fit(X, y)
testSet = [[5.1, 3.5, 1.4, .2]]
test = pd.DataFrame(testSet)
y_pred = classifier.predict(testSet)
print(y_pred)

The very last line of code will print this:  ['Iris-setosa']
Let's look at the very first row in the test set:  testSet = [[5.1, 3.5, 1.4, .2]]
Look at: dataset.head()
Do you see how y_pred is the class, given [5.1, 3.5, 1.4, .2]?

Hope that makes sense!  Now, let's change the testSet just a bit, and check the outcome.
classifier = KNeighborsClassifier(n_neighbors=3)
classifier.fit(X, y)
testSet = [[5.1, 3.5, 4.9, .2]]
test = pd.DataFrame(testSet)
y_pred = classifier.predict(testSet)
print(y_pred)

Result: ['Iris-versicolor']
The variable ('petal-length') changed from 1.4 to 4.9, and now we have a different 'Class'!!  Does it make sense?  If you look at the raw data, you will see that the testSet doesn't even exist, but nevertheless, the model can make a prediction about what the Class should be!
K-Nearest Neighbors Algorithm in Python and Scikit-Learn


Procedure for selecting optimal number of features with Python's Scikit-Learn

You may have a try on Lasso (l1 penalty) which does automatic feature selection by „shrinking“ parameters. This is one of the standard approaches to data with many columns and „not so many“ rows.
sklearn.linear_model.LogisticRegression(penalty=’l1‘,...

See also this post.
Edit:
The book „Introduction to Statistical Learning“ gives a really good overview. Here are the Python code examples from the book. Section 6.6.2 covers the Lasso.


If my data looks like this, is linear regression just never gonna be useful?

I tried running a RANSAC model on your data, but got worse results than a straight linear regressor. The ten-fold cross-validated mean absolute error over all three response variables (r,g,b) for the linear model was about 0.37. I also ran a random forest model for comparison, and got about the same score. This suggests that the linear model is not shabby, but it is up to you to decide if either is good enough.

import sklearn.linear_model, sklearn.cross_validation, sklearn.ensemble, pandas

labels = pandas.read_csv('labels.csv', header=None, names=['r', 'g', 'b'])
features = pandas.read_csv('features.csv', header=None, names=['area', 'length', 'index', 'complexity', 'lines', 'curves', 'intensity'])

sklearn.cross_validation.cross_val_score(sklearn.linear_model.LinearRegression(), features, labels[:len(features)], 'mean_absolute_error', 10, -1).mean()

I also tried visualizing the data with the outliers clipped, but did not find it very enlightening, so I did not include it here.


Can I use model which is trained using Keras to Scala?

Yep, check this blog: https://towardsdatascience.com/deploying-keras-deep-learning-models-with-java-62d80464f34a

Dump your model as .h5 like you normally would and load it in java using those libs from blog.

Another idea would be to create separate AI microservice in python(django/flask). That's what I usually do.


Algorithms that can determine whether a string is an English sentence?

There is a library for python

https://pypi.org/project/langdetect/

This field is called "language detection" and it is quite active.


If this is “big-endian”, then what is this?

That's an interesting question. I would argue that the answer to your question lies in implicit assumptions that you (are willing to) make and expecting others to make.

In the ISO 8601 example, the significance is well defined, as 1 year = 365 days, we could say that a year is more significant than a day. In the last data structure example, what do you mean with significant? There is no clear definition there. Without framing what context you are referring to (such as level of abstraction or chronological order), interpretations are bound to differ to the people you are communicating with (and that defies the purpose of using the communication strategy, doesn't it?).

Instead of compressing the idea to a single concept, you could consider to be as explicit and unambiguous in the relation between the data structures as possible.


Statistical learning for data-limited systems

It really depends on the data set in hand. For example for short time series you can look at this thread.

In general I would say that non-parametric statistics works pretty well for smaller datasets.


I am assuming that you have only 1 attribute (numeric).
What you can do is

modify the .arff file and set 0s to 1 and 1s to 0.
Then you can again run j48 classifier and visualise the decision tree.

It would give you inverted result.
Also, try watching this video.WEKA on MOOC


"Optimal range for loss function": Myth or Truth?

Very interesting statement you got there, i was not aware of this 'optimal loss value'.
the only way i could explain it is with the gradient computation : gradient is computed using :
$Grad = loss*lr$
So to me, changing the loss range is the same as changing the learning rate of your optimizer, it could be a good experiment to increase your lr instead of increasing the loss function range and see what happens to the training. I'm not aware of the subtilties of Adam optimizer so my explanation may be completly wrong.


How to predict whether the client will renew the subscription or not based on given data structure

That data structure is fine. You need to have a dataset of historic subscriptions (I.E. subscriptions that have now finished), where the output variable is "Renewed/ Not renewed". You can train your model on that dataset and then make predictions each day as to whether or not each "Active" subscription is likely to be renewed when it reaches the end date.

As to what kind of model you can train, the "usual suspects" should be fine for this. LogisticRegression, XGBoost, RandomForests and so on can all handle this kind of problem.

EDIT

So lets say for example I have a 3 month subscription that runs from 1st Jan to 31st March. I decide to renew for another 3 Months until 30th June. I then renew again until 30th September. YOU have the same subscription from 1st Jan to 31st March, and you also renew until 30th June. You then decide NOT to renew. In that case, your data should look like this:

The Status column shows subscriptions which are Finished, and which are active. You should train the model against Finished subscriptions only and make predictions against Active ones. The target variable is the "Renewed" feature. For testing purposes you should split the Finished subscriptions into a 75/25; train the model on the 75% and test it against the 25%

Hope that clarifies.


Should the depth on convolutional layers be set to a figure divisible by 2?

You are right, the way it says it, it means that the height and width should be divisible by 2.

While the argument he makes is invalid, it is a good practice for the sakes of memory efficiency to make some parameters like the batch size divisible by 2. I'm guessing you want to make more efficient use of memory blocks. The same logic could apply to the depth of convolutional layers.

This is no way as important as the author of the book says!


Why does adding a dropout layer improve deep/machine learning performance, given that dropout suppresses some neurons from the model?

The function of dropout is to increase the robustness of the model and also to remove any simple dependencies between the neurons.
Neurons are only removed for a single pass forward and backward through the network - meaning their weights are synthetically set to zero for that pass, and so their errors are as well, meaning that the weights are not updated.
Dropout also works as a form of regularisation, as it is penalising the model for its complexity, somewhat.
I would recommend having a read of the Dropout section in Michael Nielsen's Deep Learning book (freely available), which gives nice intuition and also has very helpful interactive diagrams/explanations. He explains that:

Dropout is a radically different technique for regularization. Unlike L1 and L2 regularization, dropout doesn't rely on modifying the cost function. Instead, in dropout we modify the network itself.

Here is a nice summary article. From that article:

Some Observations:

Dropout forces a neural network to learn more robust features that are useful in conjunction with many different random subsets of the other neurons.

Dropout roughly doubles the number of iterations required to converge. However, training time for each epoch is less.
With H hidden units, each of which can be dropped, we have
2^H possible models. In testing phase, the entire network is considered and each activation is reduced by a factor p.

Example
Imagine I ask you to make me a cup of tea - you might always use your right hand to pour the water, your left eye to measure the level of water and then your right hand again to stir the tea with a spoon. This would mean your left hand and right eye serve little purpose. Using dropout would e.g. tie your right hand behind your back - forcing you to use your left hand. Now after making me 20 cups of tea, with either one eye or one hand taken out of action, you are better trained at using everything available. Maybe you will later be forced to make tea in a tiny kitchen, where it is only possible to use the kettle with your left arm... and after using dropout, you have experience doing that! You have become more robust to unseen data.


Spark MLLib - how to re-use TF-IDF model

tf-idf will learn a vocabulary, idf, and some will also learn stop words (based on min_df, max_df, max_features). Read over sklearn's TfidfVectorizer and you can see the attributes that the fit method will set.

When you expose a trained tf-idf to new data it will transform that data into a vector of the same size as your original data using the vocabulary to construct Term_Counts which are then converted into your tf vector. The value in this is that you can use another model to predict an outcome based on the tf_idf as each new document will have the same size tf_idf vector as the documents you used to train the model. Otherwise you couldn't use it to make a prediction! For example with a naive bayes classifying tfidf:

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(X_train)
nb = MultinomialNB()
nb.fit(X, y_train)

# When you receive a new document
X = tfidf.transform(new_doc)
prediction = nb.predict_proba(X)

And I don't think you would want to refit the model. If you want some kind of continuous real-time update consider implementing a bayesian update


Why a restricted Boltzman machine (RBM) tends to learn very similar weights?

A restricted Boltzmann machine (RBM) learns a lossy compression of the original inputs or in other words, a probability distribution.

Those are 4 different weight matrices are all reduced dimension representations of the original face inputs. If you visualized the weights as a probability distribution, the distributions value would be different but they would have the same amount of loss from the original image reconstruction.


In short, Answer to your questions:

Yes
Movie A is near to center of the plot. Its the most balanced movie from your collection. (Biplot's center reflects the average of PCA scores: neither positive nor negative).
PC1 scores are positively correlated with with Romantic/ Comedy  and negatively with Action movies. PC0 is mostly negatively correlated with Drama movies.


Is color information only extracted in the first input layer of a convolutional neural network?

The quote that you have brought is speaking about something else. Zero means that most of your linear part of neurons have a negative value and after passing through the ReLU non-linearity, they become zero. That's the explanation of that part. About your question, not really. I don't know if you've ever visualized a convolutional network or not but you've done that you may have seen that there may be layers that get activated for special colors even in deep layers. The reason is that the convolutional layer just tries to do cross-correlation operation on the input which has three channels. Next layers of convolution also do this considering e.g. the input has 64 channels. To wrap up, there may be other deep-layers that use the information of colors, Although these operations really do not care about the colors, they care about the local patterns that may happen in the input. The reason for learning colors is that similar colors have similar patterns in their RGB channels. These patterns are what convolutional networks try to learn. Take a look at the tool box that have been discussed here.


Summarize and visualize a CSV in Java/Scala?

Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala.

I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.


What are the inputs to the first decoder layer in a Transformer model during the training phase?

Following your example:

The source sequence would be How are you <EOS>
The input to the encoder would be How are you <EOS>. Note that there is no <start> token here.
The target sequence would be I am fine <EOS> . The output of the decoder will be compared against this in the training.
The input to the decoder would be <start> I am fine <EOS>.

Notice that the input to the decoder is the target sequence shifted one position to the right by the token that signals the beginning of the sentence. The logic of this is that the output at each position should receive the previous tokens (and not the token at the same position, of course), which is achieved with this shift together with the self-attention mask.


What does this expression from gradient descent mean?

After reading up on multivariate calculus, I have understood what the above equation means. It is a simplified version of the multivariate chain rule - a rule used to differentiate multivariate functions. Here is what it looks like:
$$\cfrac{dC}{dt}≈\cfrac{∂C}{∂v_1}  \cfrac{dv_1}{dt}+\cfrac{∂C}{∂v_2}  \cfrac{dv_2}{dt}$$

Time applies here because we are studying how the cost changes over time/over the course of training. Multiplying each side by $dt$ gives:
$$dC≈\cfrac{∂C}{∂v_1}  dv_1+\cfrac{∂C}{∂v_2}  dv_2$$

And since the differential operator $d$ means a 'small change', we can replace this with delta to represent the same thing, thus arriving at the equation from the question:
$$\Delta C≈\cfrac{∂C}{∂v_1}  \Delta v_1+\cfrac{∂C}{∂v_2}  \Delta v_2$$
To summarise, this equation is a simplified version of the multivariate chain rule, which although means the same as the original, does a better job in communicating that a 'small change' in $v_1$ and $v_2$ results in a 'small change' in $C$.


What is the difference between concept class and hypothesis

If one requires that $H = C$, then this is called the "proper PAC" framework compared to "PAC prediction" where we don't care about the representation of $h$ as long as the prediction error is small enough (i.e. we allow $H$ to be the class of all time-polynomial programs).

You can think of a concept as the set of inputs that produce the same label (e.g. all the images that show a chair form the concept "chair" or all the points in the same half space form the concept "true/false").


Which hyperparameters of a neural network can be tunned independently?

Your question is indeed a little bit broad, but I will try to explain to give you an overview on the importance and on some specific topics.

Indeed, multiple hyper-parameters exist in the context of deep learning. At the same time, just like Andrew Ng mentions in his courses, some are of an bigger importance than the others.

For instance, if you see that your training progresses very slowly (i.e. your convergence is relatively slow), you may want to fine-tune your learning rate.

Learning rate is a quintessential example of hyper-parameter that is more important than the number of neurons in a FullyConnected layer or to change the Dropout rate on a layer from 0.3 to 0.5.

At the same time, there exist two well known techniques for hyper-parameter search: grid search and random search. While the former behaves exactly like you explained (keeping the values of N-1 hyper-parameters fixed and iterating over some specific values of the N hyper-parameter), the random search has proven to have a better positive impact, as it slightly modifies all your hyper-parameters after a search step; although it may not be intuitive at first sight, this can yield better and earlier results than the grid search.


Does partial transfer learning require a lot of computer power?

The dataset on which VGG16 is trained consists of images from different domains, and its purpose is to classify images by extracting images' feature maps and attributes. Unfreezing few botton layers will help in training them for our specific image usage, however, these will be few parameters to learn than the whole connected network as it will have already learned basic features.

When you choose to train only a few bottom layers of VGG16, it will definitely require less computation.


can I use t-sne or PCA to reduce number of classes?

No. t-Distributed Stochastic Neighbor Embedding (t-SNE) and Principal Component Analysis (PCA) are dimension reduction techniques, aka fewer columns of a tidy dataframe.

Clustering will reduce the number of observations, aka fewer rows of a tidy dataframe. In particular, you might be looking for hierarchical clustering.


Prevent model from over-focusing on strong features

Use dropout.
Dropout is a form of regularization where you zero out the input components with a certain probability.
If you want your model to learn representations that rely less on some features, just place a dropout layer before those features enter the rest of the network. This will force the network to learn more robust representations that depend less on the strongest features.


Do weights of keywords for each topic add up to 1 in topic modeling?

Yes, the weights would add up to 1. This is a dirchlet random variable - review documentation on scipy here - https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.dirichlet.html


How can we identify potential customers for a new list of customers?

You are absolutely right. This method is called customer segmentation. Here we cluster customers based on many features like their demographic and income.
Suppose we get to know a particular set of people who belong to high income group/rich demographic are not spending much then we can do promotions to increase sales to get potential new customers hence it help drive business decision.
Having said that it's not best to run the algo once on past data and make decision as we know that customer behaviour changes frequently .So,companies usually run clustering algorithm daily to take advantage of the new customer behaviour.
So, to make our approach robust we need to find optimal no of clusters, choose best features hence it is a iterative process.
Refer this to know more.


Suggestion for stacked modelling in machine learning

Would that be called as stacked modelling?

Yes, this is exactly what stacking models means.

I didn't understand what is the role of clustering in your design?

The standard approach consists in using only the predictions of the $N$ individual learners as feature for the meta-model. The training/testing data split is a bit more complex: the training of the meta-model requires predictions from the individual models, so the data could be split like this:

training data for individual models
training data for the meta-model (on which individual models are applied)
testing data for the full system

Generally it's safer to choose a simple learning algorithm for the meta-model, because there is a higher risk of overfitting (for instance linear regression or majority voting).

[Detailed answer to comments below]

The standard setting would be like this:

Train $N$ individual models $m_1,...,m_N$ using first-level training set $T_1$
Apply these models in order to obtain $N$ features for the meta-classifier. At this stage it's a bad idea to use instances from $T_1$, because the individual classifiers have been trained on $T_1$. so one needs a second set of instances $T_2$: for every instance $x\in T_2$, let $m_1(x),...,m_N(x)$ be the predictions resulting from applying classifiers $m_1,...,m_N$.
Train the meta-classifier model $M$ using $T_2$ as training set, with $m_1(x),...,m_N(x)$ the features for any $x\in T_2$.
Testing: as usual one needs a fresh set of instances, say $T_3$. For any instance $x\in T_3$ calculate the prediction by applying $M$ to the $N$ predictions obtained by the individual models, i.e. $M(m_1(x),...,m_N(x))$


Which data science model is best for explainability for prediction problems?

Personally I think linear (through model's coefficients/weights) and tree-based models (gain importance) are the best for explainability
But this is not restricted to those models since you can use model agnostic techniques to explaine any model, even those consider as "black-box"
Like:

SHAP Values
Partial Dependence plot
LIME

You can check this good resource to learn more on that.
A final comment I'd say is that you cannot forget the fact that an essential part of model explainability is model performance, It does not make sense to have a model that is easily explainable but with a very low performance, since those structures found by the model do not generalize well. So a good model in performance will lead you to the "correct" conclusions when explaining it.


For certain models, it can be important to make sure the inputs are linearly independent. I'm pretty sure this is never the case for the outputs. If we have more than two classes, it's extremely difficult if not impossible to output class-probabilities if we don't have a specific dimension for each class in the output vector. In other words, your suggestion has much more explanatory power than what was suggested by your interviewer, which is solving a problem that exists on the inputs for some models but not for the outputs.

In multi-class problems, it is indeed the norm that each class has their own dimension in the output vector as you suggested. Either the interviewer was trying to look smart and confused input and output encoding (which still would only apply to certain families of models), or you aren't remembering the events of the interview properly. If you're confident that the interview went as you described, you should consider complaining to your point of contact that you were criticized for giving a correct answer.

Feel free to cite the following as popular/canonical examples of multi-class classifiers:

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba

  Returns: array of shape = [n_samples, n_classes]

https://keras.io/losses/#categorical_crossentropy

  when using the categorical_crossentropy loss, your targets should be in categorical format (e.g. if you have 10 classes, the target for each sample should be a 10-dimensional vector that is all-zeros expect for a 1 at the index corresponding to the class of the sample)


Run python script using spark-submit on windows 7

use this command
.\bin\spark-submit \guruorders\pcakinetics\files\kineticPCA.py


What are the differences between MLP and DNN?

You explained it already quite well. An MLP is a type of neural network, the same way CNNs, RNNs, and other types exist. DNN is an umbrella term for all types of neural networks.
The reason some literature may be using these terms interchangeably is likely due to the fact that MLPs are some of the oldest forms of neural networks and therefore, at the time the literature was written, MLPs and DNNs were ubiquitous.


Sklearn tfidf vectorize returns different shape after fit_transform()

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this -

>>> x_train_tf.toarray()
array([[ 0.70710678,  0.        ,  0.70710678],
       [ 0.        ,  1.        ,  0.        ]])

If you check what features you are getting, you will see -

>>> tfidf.get_feature_names()
[u'marital', u'relationship', u'status']

What's happening is, while passing dataframe, the TfidfVectorizer is only taking the column names and converting them into numeric form.

I don't think you need to use tfidf here. As far as I understand, your data is categorical text, so use pandas.get_dummies() instead of tfidf. This will convert your categorical data to numeric form which you can use for modelling.

>>> pd.get_dummies(x_train.iloc[:10,])
       marital-status_Divorced  marital-status_Married-civ-spouse  marital-status_Never-married  marital-status_Widowed  relationship_Husband  relationship_Not-in-family  relationship_Own-child  relationship_Unmarried
1521                   1                          0                              0                        0                       0                            1                         0                        0
2274                   0                          1                              0                        0                       1                            0                         0                        0
20209                  0                          1                              0                        0                       1                            0                         0                        0
5529                   0                          1                              0                        0                       1                            0                         0                        0
27639                  0                          1                              0                        0                       1                            0                         0                        0
26670                  0                          0                              1                        0                       0                            1                         0                        0
16635                  0                          0                              0                        1                       0                            0                         0                        1
30824                  0                          0                              1                        0                       0                            1                         0                        0
15181                  0                          0                              1                        0                       0                            0                         1                        0
9119                   1                          0                              0                        0                       0                            0                         1                        0

Here, for demo purpose, I have taken only 10 rows.