Hierarchical clustering with precomputed cosine similarity matrix using scikit learn produces error

According to sklearn's documentation:

  If linkage is “ward”, only “euclidean” is accepted. If
  “precomputed”, a distance matrix (instead of a similarity matrix) is
  needed as input for the fit method.

So you need to change the linkage to one of complete, average or single. If you try this it works:

import numpy as np
from sklearn.cluster import AgglomerativeClustering

X = np.array([[0,0.3,0.4],[0.3,0,0.7],[0.4,0.7,0]])

#cluster = AgglomerativeClustering(affinity='precomputed', linkage='complete')
#cluster = AgglomerativeClustering(affinity='precomputed', linkage='average')
cluster = AgglomerativeClustering(affinity='precomputed', linkage='single')
cluster.fit(X)


In order to validation you can choose different methods such as k-fold cross validation, repeated splits and Bootstrapping. It seems what you aim to do is 2 fold cross validation and yes it should be random. You should be careful, and it seems your training data and testing data are very different.In this case, you;ll not have a model with a good performance.Try to find a way and make your training data look more like the data on which you want to implement your model. In this case you might want try 10 fold cross validation (for instance)


Is there a quantitative way to determine if a class of algorithms tends produce low bias or low variance models?

It's more a matter of complexity of the model than of the class of algorithms. Of course some classes of algorithm produce more complex models than others by construction, but this is not always the case. For example the complexity of a Decision Tree usually depends on the options/hyper-parameters: maximum depth, pruning, minimum number of instances in a branch. If these parameters are set to produce a small tree then the risk is bias (underfitting), but if they are set to produce a large tree then the risk is variance (overfitting).
The complexity of a model depends mostly on the number and nature of its parameters, so as a first approximation the number of parameters is a reasonable quantitative measure of complexity (see this closely related question). Also keep in mind that most models try to use all the features they are provided with, so the number of features also has a high impact on the model complexity.


Prepare data for SVM, Is it valid to normalise the data before and after PCA dimension reduction

Depending on the language of your choice, you may or may not have to normalise the data yourself - for example the e1071 package in R does this automatically for you. As this is built off the 'libsvm' library, it might be that this is also the case there - the library documentation is your best source. As for the normalisation values, you should definitely use the same [min,max] values from the training set for the test set as well. Also, I would reccomend reducing the dimensionality of your data first, than normalising before running a svm. hth


Predict the salaries of customers in 2019 from 2016 data

Welcome to the site! First off, I will start by saying that you might be better off asking this in a statistics SE because this isn't really data science. But I will still try to help you.

I will assume that age and salary are the only items you have and that salary is the only thing you are trying to predict. If those assumptions are correct then there really is no algorithm, you can just do some distribution "tricks" to get you there.

You're on the right path of thinking about things on a proportional basis and I would proceed accordingly. However, I wouldn't think of it in whole numbers (like your 20k example), I would think about using some scaling. So, take the max value of all your salaries and create scales with something like salary / max salary. From there, you should should know how many people will have X salary (in a scaled, decimal format).

Now, you need to decide how much salary inflation (or deflation) has happened in the 3 years since your data was available. Take that percentage and assign it back to your scaled values and now you can have a pretty good idea of how much each person makes in today's dollars (otherwise, you'd end up with salaries in 2016 dollars).

Good luck!


Gradient descent parameter estimation Package for R

Ok, after a lot of looking I found the "optim" routine which is in "stats", one of the packages that is always loaded. It has quite a few methods including conjugate gradients and BGGS and a few others and worked well on the first few examples I tried. It doesn't seem to get a lot of attention strangely. I guess optimization people tend to use Matlab.

I knew there had to be something.


Binary classificaiton for weather data if its class 1 or class 0 alert

Which model you choose in the end depends on your data. We cannot really answer this for you. Only practice and trials and errors will actually help you doing this.

For binary classification, you can choose among a very diverse range of models, from Logistic Regression to SVMs to Random Forests to Neural Networks. As a rule of thumb, you can use Occam's Razor => for two models giving the same performance, always choose the simplest one.

So my advice is: try out a few model, compare their performance on a testing set and select whichever works best for you...


Decision tree with final decision being a linear regression

I think the easiest way to do this would be to have a decision tree where the final decision results in a linear formula.

Setting aside whether this is actually easiest/best, this type of model does exist, usually called "model-based recursive partitioning".  See e.g. https://stats.stackexchange.com/q/78563/232706
There are several packages in R for this: partyfit (and the older simpler party), mob, Cubist; unfortunately, there don't seem to be any in Python yet.  Here's a discussion on including it in sklearn, from mid-2018.


Is random forest better suited than neural network in my scenario?

I'd use Random Forests for your scenario because RF is well known for determining the importance of different features; the methods are implemented in many Machine Learning frameworks, and it's very simple to understand.

While neural network are powerful, the methods for extracting the importance of individual features are less clear. Neural networks are more appropriate if you are more focused on just results.


How to interpret two continous variables output using GAM?

What you do here is to use smoothing splines regression. Have a look at the book "Introduction to Statistical Learning" Chapter 7.5 for a very good overview on the method.

The s-function in GAM allows you to specify how the GAM is fitted. You have not supplied k, so some default is chosen:

  the dimension of the basis used to represent the smooth term. The
  default depends on the number of variables that the smooth is a
  function of. k should not be less than the dimension of the null space
  of the penalty for the term (see null.space.dimension), but will be
  reset if it is. See choose.k for further information.

GAM basically does several linear regressions (specified by k) along the x-axis if you want to say so. Thus, GAM allows to model wild non-linearity. If you want to check linearity in your data, you should check different values of k and look at the plot.

GAM Example:

library(gam)
library(ISLR)
df = ISLR::Auto

# GAM with 10 knots
gam5= gam(horsepower ~ s(mpg, 10), data=df)
summary(gam5)
plot(gam5, se=T)

GAM Result:

Would you conclude that your model is linear? No. In the range between about ]mpg=0 to mpg=20[ there is a linear relation as well as betweeen ]mpg=20 to mpg=40[. But linearity does not hold for the entire range of data. So I would differentiate these segments, e.g. by dummy encoding and interaction terms.

Note that the y-axis is rescaled. So there is no natural interpretation of the y-axis here.

Comparison to non-parametric (NP) estimation:

To deal with non-linearity non-parametric regression is an obvious alternative. What happens if we do NP?

# Nonparametric regression
library(SemiPar)
fit <- spm(df$horsepower ~ f(df$mpg))
plot(fit)

NP Result:

As you can see NP delivers almost the same result. However, the y-axis in the figure has a natural interpretation, which can be useful.

What about your Problem:

First make sure that you check different values for k in s(...,k), so check different number of knots and see how the figure changes. Also have a look at the book to understand the background.

In your current figure, I see some kinks at about x=10 and x=20. However, I would not say that this is severe non-linearity (but there is non-linearity in the data). Generally, if you can draw a line over the plot range (along the x-axis), and if this line is not outside your confidence bands, you can claim for a linear relationship.


Strange Behavior for trying to Predict Tennis Millionaires with Keras (Validation Accuracy)

I can't find your script hidden in your repo which is your dataset
  which is full of stuff...you should organize your work better if you
  plan to ask on SO.

This would be the reply from a veteran but I am not.

You should use binary_crossentropy as loss, and then I think you would be able to not use to_categorical.

Then the last layer should have activation sigmoid.

More suggestion

Try to use only a neuron in the last layer: Dense(1, activation='sigmoid')

The accuracy problem may be on how you split the data: maybe only samples from one class go into the validation or in training data. I suggest you use train_test_split from sklearn with fixed random_state or something similar.


how to get probability in classification problem

Thanks a lot for your help. I tried model.summary() and looked at the outputs of intermediate layers. My mistake was not normalizing the test input.


Agglomerative Hierarchial Clustering in python using DTW distance

Use a precomputed distance matrix, and distance="precomputed".

HAC will compute a distance matrix anyway.


How to generate image using deep learning

DCGAN(Deep convolutioanl GAN) or WGAN can be used to generate images by training them on already existing images. GAN are very tricky to train. DCGAN and WGAN are the techniques that make it somewhat easier.

A GAN consists of a Generator and a Discriminator. Generator's job is to generate images and Discriminator's job is to predict whether the image generated by the generator is fake or real. If the generator succeeds in fooling the discriminator, we can say that generator has succeeded.

I have generated MNIST images using DCGAN, you can easily port the code to generate dogs and cats images.


1) You don't, at least initially. Use all of the variables you have, and through proper cross validation/model tuning (in particular, tuning the number of variables you randomly select to try in each split of the tree, and maybe stuff like minimum node size and maximum size of individual trees) let the random forest find the most predictive variables for you.

If you have a lot of candidate variables, then maybe try an wrapper based method like permutation importance, recursive feature elimination, etc. though for now I'd avoid this especially if you are a beginner (these methods tend to take a long time, and may not be useful/offer marginal improvement).

2) What is the purpose of this? As a means of feature selection? Random forests already kind of do this; they randomly select a subset of your variables at each split in a tree. Again, I would stick to letting the random forest do its feature selection for you for now.

3) Both. Nominal attributes are often one-hot encoded/dummy encoded (create k-1 dummy columns, k = number of categories in the nominal variable) which I encourage you to search up. Basically, you are encoding a nominal attribute into a numerical one.

You can also encode nominal variables through other means (like target encoding) but again, if you are a beginner let's avoid that for now because you can severely screw up your model validation by doing this.


How do write the following piece of code in R?

If you already have the names in R, you can iterate over the vector of names, and use the paste function. Here's the help page for it. sprintf would be another option, if you prefer that approach.

An example of this would be

names <- c("a", "b", "c")
querystring_a <- "https://...../api/dataservice?sql = select * from db where name IN "
querystring_b <- '\\'
querystring_c <- "\'"
for(n in names){
    cat(paste(querystring_a, querystring_b, querystring_c, n, querystring_c, sep=""), sep="\n")
}

Return:

https://...../api/dataservice?sql = select * from db where name IN \'a'
https://...../api/dataservice?sql = select * from db where name IN \'b'
https://...../api/dataservice?sql = select * from db where name IN \'c'


This is a controversial subject with no clear best answer and myriad options, some of which are model-specific. You can drop them, replace them with extreme values, interpolate, replace with median, impute with nearest neighbors, etc. In all cases besides dropping them altogether, you make assumptions and create data where no data exists.  I can't tell you the best approach, but I can advise you to try multiple approaches and hope the end results are robust to choice.


Make use of relationships on recommendation systems

There are a number of ways to make use of this information, including various graph-based methods of recommendation.
Collaborative Filtering using Weighted BiPartite Graph
Projection
A Recommendation System for Yelp
It's also common to use multiple approaches to build a hybrid recommendation system.


Perform classification on market basket analysis

In case that the number of items is quite small, turning the problem into a classification problem will be the most convenient solution. Use each item as a feature, the class as a concept. Now you can apply the methods that @OvisAmmon recommended on or other classification methods.

However, usually in market basket analysis there are many items and each user buy few of them.  That leads to a very sparse matrix and most classification methods have problems dealing with them.

A common method to analyze market basket data is using
Association rules which are implemented by first locating frequent item sets.

You can use association rules on order to learn on market basket in the following way:

Add item for every class
Add the owner class to its basket (e.g, transform {milk, beer} basket of a student in to the basket {mlik, beer, student}
Use a frequent item set algorithm to find them
Remove all frequent item sets that don't have a class member in them. Such sets can't help deduce the class.
From the frequent items set find rules that imply the classes (with good enough support and confidence
Use the these rules that match a basket in order to predict class

You should note some points:

Some association rules with different implication might apply to the same basket. You might have the rules milk -> X and beer -Y and the basket {milk, beer}. If the set {milk, beer} is frequent enough you might have a rule specifically to it. Otherwise you'll need a met rule in order to base by the rules. Such rules might be taking the rule with the highest confidence or the most likely class.

Association rules are based only on the existence of item, not on the lack of them. If students never buy diapers, you will not find such association rule. You can add "no diapers" as an item to your data set but that will turn it from a sparse one to a full one and the storage might be unfeasible.

The complexity of association rules is exponential. The is simply since the output it self is exponential , as can be noted when you have a Clique in the data.
Usually people don't try to find all association rules due to that problem, which is the practical thing to do.
However, it means that there might be association rules in your data that you won't find.


What is the best way to visualize the relationship two categorical variables

You can use a confusion matrix to generate the heat map of your data.

Suppose you have:

labels = ['cardiac arrest', 'choking', 'seizure']
dispatch_code = ['cardiac arrest', 'choking', 'seizure', 'choking', 'seizure', 'seizure', 'cardiac arrest', 'cardiac arrest']
hospital_code = ['cardiac arrest', 'choking', 'cardiac arrest', 'choking', 'seizure', 'seizure', 'seizure', 'cardiac arrest']

Then you can plot with:

from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt

cm = confusion_matrix(hospital_code, dispatch_code)
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(cm)
plt.title('Confusion matrix of the classifier')
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('True')
plt.show()

This example shows that often there is a correct code coming from the dispatcher (yellow stripe in the middle). Cardiac arrest and seizure are often confused (blue) and chocking is never misclassified (purple).


predicting loan default based on transaction history

Some context:
The problem you are trying to solve can be defined as a classification problem.
Where based on features $x^1...x^n$ we are trying to predict if $y$ is class $default$ or class $no$ $default$.
There are several algorithms that are able to learn the different functions between the features and the class, functions that can be used for prediction purposes.
Neural network is one such function.

Answering the question(s):

1) handling variable input size: this is common when looking at time based related data, where each instance might have different history length and or frequency.
One simple approach is deducing the feature space to the biggest common space(I.E, truncate the history lengths to shortest existing in all). Be aware that in this approach you wont use all of data you have.
Another approach is aggregating the history into one or more features:
'total deals', 'average deal amount', 'average time between deals', etc..

2) Handling the 'string' columns: This data can be probably treated as a categorical feature.
Encoding it into several different variables(each representing one(or more) possible value(s) this feature have) is a common way including it as a feature in the learning algorithms.

There are other methods handling each question, but these should be a good starting point.


How do you visualize Decision boundaries?

To visualize a decision boundary of a classifier, specifically a binary classifier as in your case, you can instantiate a grid of points that spans the domain of interest that you want to classify. Then use your trained classifier to predict the class of each and every point in the grid. If the grid resolution is fine enough, when you plot the contours of the grid points with color corresponding to its class, you can then visualize the decision boundary. It is helpful to overlay the original training data points so that you can see how your decision boundaries are generated based on various training sets. See the following example


How to find False discovery rate of a ML model? Situation explained below

In your case, you can estimate the FDR by
$$110 / 400 = 27.5\%$$
the approach of sampling some data and estimating the empirical FDR in there is valid, it's the same as estimating the mean of a population using a sample, it makes sense, and the more data you use the more confident you're going to be on your estimate, as long as you're sampling randomly.
400 seems a pretty decent number to estimate FDR, so your estimate is going to be pretty good.


Obvious optimisation would be converting 'Date received' to pandas datetime once, for example (if you need to retain original column):

geoset['pddate_received'] = pd.to_datetime(geoset['Date received'])
geoset['MonthInt'] = geoset['pddate_received'].dt.month # Add a month
geoset['YearInt'] = geoset['pddate_received'].dt.year # Add a year
geoset['DayInt'] = geoset['pddate_received'].dt.day # Add a day of month
geoset['DayOfWeekInt'] = geoset['pddate_received'].dt.dayofweek # Add a day of week


What tools do you use to clean corrupt data?

If you know the nature of data you require as suggested by @Emre for manual cleaning you can use Apache Spark or Pandas. Utilizing the data frame aspects of both Spark and Pandas will help you preprocess data with a much easier convenience. Because data frames account for any data type it's easy to access and clean specifically the segments you require. If the corrupt data is of repetitive nature you can use several Map reduce techniques to clean it fast as well. If it's the case of big data.


Python: How to test a RandomForest regression model for Overfitting?

Predicting on x_train and comparing the error with the prediction error of x_test is a good approach. Here, it seems that you are overfitting since you model is better to predict what it learned than new data. It would be a good thing to use a cross-validation approach to ensure that the problem does not come from your split.
Here are some leads I would follow:

Apply a cross-validation to evaluate your model
If using cross-validation in hyperparameters tuning, it is important to keep a validation dataset to evaluate your final model after the tuning. Doing that, you ensure to test your model on unseen data.
Have a look at your hyperparameters. You could for instance decrease max_depth, that may lead to reduce overfitting.
Plot your outputs: plot the excepted values and the predicted ones. MAE is the Mean Absolute Error, so you may discover unexpected effects (such as your model having high troubles to predict specific range values and not others)

Hope it helps.
Note
I would not say overfitting is a problem, rather its consequences could be a problem. Overfitting occurs when the model learns too much from the training data and so does not generalize well. If you constructed your pipeline well, the metric that should lead your performance is the score you choose (in your case MAE)


Visualizing decision tree with feature names

hstack preserves the order of the columns, so you can piece together the feature names for each of your component arrays.  OneHotEncoder (if that's what you used) and CountVectorizer both support get_feature_names, so concatenating the lists of feature names should be possible.  To give full details would require more details about how each of the arrays was generated.  You might consider using ColumnTransformer in the future, which handles all that concatenation for you and also provides its own get_feature_names.


Why does Naive Forecast,Moving average,Simple Exponential Smothing model predictions flatten?

The naive forecast = random walk with no drift will always give a flat straight line as your forecasts. This is just the last observed data point in your time series extended forward n steps ahead.

The moving average model with m =50 will be the average of the most 50 recent data points. As we can see, your time series is mostly flat with some erratic peaks that have been smoothed away by the relatively large window (50) that you have chosen. In general, the larger the window the more flat and smooth your forecasts. Try changing the window to be smaller and you will probably get more erratic behavior in your forecasts.

Simple exponential smoothing always gives a flat forecast since all forecasted values are equal to the first forecasted value (i.e. y(t+k) = y(t+k-1) =....y(t+1), for all k > 1). This can be proven quite easily using basic induction.

Holts Linear Trend model breaks up the forecasts into three different components; the trend, level, and seasonality. If your time series exhibits no seasonality and no level changes then all you will be left with is your trend which is a flat line in your case.

Your time series looks to be a flat line with unpredictable random fluctuations. Hence, in terms of forecasting error I am not too surprised to see that a flat line may give the best forecasts. Try calculating RMSE, MAE, or MASE (which compares to the naive method directly) and see for yourself. Often, a flat line is the best forecast especially for financial time series in particular.


Is what I did supervised or unsupervised Machine learning?

The basic idea behind supervised and unsupervised learning is:
In supervised learning, one possesses a training data set that includes the desired target output feature. Examples of popular supervised learning algorithms are linear regression, and support vector classification etc.
In unsupervised learning, we do not possess a training data set with the target output feature i.e. we do not know what our output or results look like. An example is clustering. There are really good explanations out there about this.

LDA is an unsupervised learning algorithm and the process you described can be classified as unsupervised learning. The filtering step that you describe does not make the algorithm supervised because the target smartphones have not been directly correlated to the training data and therefore is only serving as a guide to restrict the cardinality of the target variable.


Hybrid Convolutional and Conventional Neural Networks

If I had to do it, I would use a transfer learning strategy:

I would train a deep learning model just with the images to solve a classification problem. In order to do this, I would need of course to have tags for the image classes. If this tags do not exist, then this approach does not make sense. It is relatively easy to make very accurate image classification, as there are lots of tools. Of course, I would do that using a CNN.

Once I have my CNN trained, I would take the last fully connected layer out. I would concatenate the inputs of this layer with the cateogrical/numerical data that you have, and obtain a feature vector using this concatenation. I would input this feature vector to another ML algorithm to carry out the objective that you need.

Did I just make this up?

No. A similar thing is done in fast ai deep learning course for text. An RNN is trained to predict the next word of a sentence and then the weights of that RNN are used to build a sentiment analysis machine. I think the idea is very similar and this is why I call it transfer learning.

What if I don't have tags for the images?

Then my advise is to use a CNN trained on image-net, take the last fully connected layer out and concatenate the inputs to that layer with the other features.


intuitive understanding of (1+parameter)/parameter

I'm not sure, without knowing more, but this is of course equal to $1 + \frac{1}{parameter}$. Out of curiosity I searched for something about fish mortality and growth params and found http://www.fao.org/tempref/docrep/fao/008/a0212e/a0212E08.pdf which sets this up as a log-log regression in terms like a growth param $L_\infty$, which makes sense if there's some power-law distribution.

Maybe the parameter being used is its inverse, and it may be that this is fudged with the +1 just to make sure the value is greater than 1 so that the log is nonnegative. Maybe there's a better domain-specific reason for tweaking it.


Splitting large multi class dataset using leave one out scheme into train and test

Never mind! Found it!

just doing a,indices,counts=np.unique(y) will return the starting indices of each unique value in indices, and then you can just slice this off as x_test=X[indices], and remember to delete it using np.delete(X,indices)


You can't really talk about significance in this case without standard errors; they scale with the variables and coefficients. Further, each coefficient is conditional on the other variables in the model, and collinearity actually seems to be inflating the importance of hp and disp.

Rescaling variables should not change the significance of results at all. Indeed, when I reran the regression (with the variables as is, and normalized by subtracting off the mean and dividing by the standard errors), each coefficient estimate (except the constant) had exactly the same t-stat as before scaling, and the F-test of overall significance remained exactly the same.

That is, even when all of the variables have been scaled to have a mean of zero and a variance of 1, there is no one size of standard error for each of the regression coefficients, so just looking at the magnitude of each coefficient in the standardized regression is still misleading about significance.

As David Masip explained, the apparent size of the coefficients has an inverse relationship with the magnitude of the data points. But even when the coefficients on disp and hp are huge, they are still not significantly different from zero.

In fact, hp and disp are highly correlated with each other, r=.79, so the standard errors on those coefficients are especially high relative to the coefficient magnitude because they are so collinear. In this regression, they're doing a weird counterbalancing, which is why one has a positive coefficient and one has a negative coefficient; it seems like a case of overfitting and does not seem meaningful.

A good way to see which variables explain the most variation in mpg is the (adjusted) R-squared. It is literally the percentage of the variation in y that is explained by the variation in the x variables. (Adjusted R-squared includes a slight penalty for each additional x variable in the equation, to counterbalance overfitting.)

A good way to see what's important--in light of the other variables--is to look at the change in adjusted R-squared when you leave out that variable from the regression. That change is the percentage of variance in the dependent variable that that factor explains, after holding constant the other variables. (Formally, you can test whether the left-out variables matter with an F-test; this is how stepwise regressions for variable selection work.)

To illustrate this, I ran single linear regressions for each of the variables separately, predicting mpg. The variable wt alone explains 75.3% of the variation in mpg, and no single variable explains more. However, many of other variables are correlated with wt and explain some of that same variation. (I used robust standard errors, which might lead to slight differences in standard error and significance calculations but will not affect coefficients or R-squared.)

+------+-----------+---------+----------+---------+----------+-------+
|      |   coeff   |   se    | constant |   se    | adj R-sq | R-sq  |
+------+-----------+---------+----------+---------+----------+-------+
| cyl  | -0.852*** | [0.110] |        0 | [0.094] |    0.717 | 0.726 |
| disp | -0.848*** | [0.105] |        0 | [0.095] |    0.709 | 0.718 |
| hp   | -0.776*** | [0.154] |        0 | [0.113] |    0.589 | 0.602 |
| drat |  0.681*** | [0.123] |        0 | [0.132] |    0.446 | 0.464 |
| wt   | -0.868*** | [0.106] |        0 | [0.089] |    0.745 | 0.753 |
| qsec |  0.419**  | [0.136] |        0 | [0.163] |    0.148 | 0.175 |
| vs   |  0.664*** | [0.142] |        0 | [0.134] |    0.422 | 0.441 |
| am   |  0.600*** | [0.158] |        0 | [0.144] |    0.338 | 0.360 |
| gear |  0.480*   | [0.178] |        0 | [0.158] |    0.205 | 0.231 |
| carb | -0.551**  | [0.168] |        0 | [0.150] |    0.280 | 0.304 |
+------+-----------+---------+----------+---------+----------+-------+

When all the variables are in there together, the R-squared is 0.869, and the adjusted R-squared is 0.807. So, throwing in 9 more variables to join wt just explains another 11% of the variation (or merely 5% more, if we correct for overfitting). (Many of the variables explained some of the same variation in mpg that wt does.) And in that full model, the only coefficient with a p-value under 20% is wt, at p= 0.089.


It seems that you are training on a very small dataset. A dataset of size 10000 samples will bring out less than 5k words after preprocessing, which I think isn't enough for your model to learn the relations. Try increasing your dataset and then train. Also, you can tune your parameters while training. Try changing window size (default is 5), embedding size and min count. Generally, 100 is a good embedding size, but you can tweak it and see what gives you better results.

You can try using already trained word2vec model released by Google. It's 1.5GB and includes word vectors for a vocabulary of 3 million words and phrases that they trained on roughly 100 billion words from a Google News dataset.


Working ofLSTM with multiple Units - NER

Welcome to StackExchange DataScience!

1) The same input $x_1$ and $h_0$ will be fed to each 100 neurons parallel (independent) and respective outputs will go to next time step. Notice that there are two 100-dimensional vector going to the next time step: $c_1$ and $h_1$, which are denoted by the arrows on the right side in your second figure.

2) This is a complicated question. Firstly, there are usually no one-to-one mapping between a neural and its meaning like neural 1 -> gender.

3) Those neurons do share information by the matrix multiplication operations. For example, the cell value at time step $t$ is usually calculated as
$$c_t = f_t c_{t-1} + i_t \tanh(W_{xc}x_t+W_{hc}h_{t-1}+b_c)$$
Notice that the hidden value in the previous time step $h_{t-1}$ is multiplied by a matrix $W_{hc}$, therefore each value in $W_{hc}h_{t-1}$ is a linear combination of all the values in $h_{t-1}$.

4) The number of neurons in the LSTM cell, the dimension of hidden vector $h_t$, the dimension of the cell vector $c_t$, and the dimension of the output vector $o_t$, are all the same (in your case, 100). This is known as the "hidden size", "output size" or "number of units/neurons". This is not the same as the dimension of input vector $x_t$, which is usually known as the "input size".

If you have further questions about the dimension of the LSTM variables, I suggest you to look at the first answer here where the author listed the dimension of all variables and parameters.


Tagged dataset with photos for race detection

Alright, I believe I found one:
https://github.com/dchen236/FairFace


how to read emotion label of MSP-IMPROV dataset samples?

I asked this question from Prof. Busso as the leader of the MSP-Improv dataset team. He answered:
"
The intended emotion is included in the file name. The emotion annotated by the workers is provided in the file Evalution.txt. You should not use the intended emotion as the actual emotion.
Example:
MSP-IMPROV-S01A-M01-P-FM01
intended emotion was anger, but if you see the evaluation you will see the following information:
UTD-IMPROV-S01A-M01-P-FM01.avi; N; A:3.666667; V:3.166667; D:2.833333 ; N:4.166667;
51o11nal985p2oq49qmnnqivm2-p3b; Neutral; Neutral; A:3.000000; V:3.000000; D:4.000000; N:4.000000;
6lrk33e18tetg934je64ale7r6-p3b; Neutral; Neutral; A:4.000000; V:3.000000; D:3.000000; N:4.000000;
cv970rs8rn8v23p3f3tec5bgb4-p3b; Happy; Happy,Neutral; A:5.000000; V:4.000000; D:3.000000; N:5.000000;
n76ifqflrdaao10o0bg8js05o3-p3b; Neutral; Neutral; A:3.000000; V:3.000000; D:2.000000; N:4.000000;
puumgduqdt8hfiffuvts8m3241-p3b; Neutral; Neutral; A:4.000000; V:3.000000; D:2.000000; N:4.000000;
qlvg8teulu3c7mcgvds07eqte1-p3b; Neutral; Neutral; A:3.000000; V:3.000000; D:3.000000; N:4.000000;
This file was annotated as neutral.
"
so I write a code for this purpose that reads the file name and finds it in the Evaluation file and read real annotated emotion.
this is the main function (please put MSP database  in "DBs" folder):
 function [DB, samplesN, fs]= mspDBread(sesNum)
%%
emoEvl  = 'DBs/MSPEvalution.txt';

rootdir     = 'DBs/MSP-IMPROV';
sesList    = dir(rootdir);
sesList    = sesList(3:end);
samplesN =1;
for iSes= sesNum
session = sesList(iSes).name;
sentenceRoot = [rootdir,'/',session];
sentences = dir(sentenceRoot);
sentences = sentences(3:end);

for iSentence = 1:numel(sentences)
    scenarioRoot=[sentenceRoot,'/',sentences(iSentence).name];
    SentName = sentences(iSentence).name;
    scenario = dir(scenarioRoot);
    scenario = scenario(3:end);
    for iScen = 1:numel(scenario)
        fileRoot = [scenarioRoot,'/',scenario(iScen).name];
        fileList = dir(fileRoot);
        fileList = fileList(3:end);
        for iFile = 1:numel(fileList)

            emoList={};
            i=1;
            textFileID = fopen(emoEvl);
            tline = fgetl(textFileID);

            while ischar(tline)

                tline = fgetl(textFileID);
                if numel(tline)~=0
                    if strcmp(tline(1),'U')
                        words = strsplit(tline);
                        destLineHeader = words(1);
                        destLineHeader = destLineHeader{1,1};
                        destLineHeader = destLineHeader(5:end-5);
                        waveFileName = fileList(iFile).name;
                        if strcmp(destLineHeader,waveFileName(5:end-4))

                            emotion = words(2);
                            emotion = emotion{1,1};
                            emotion = emotion(1,1);

                            compRoot= [fileRoot,'/',waveFileName];

                            [x, fs] = audioread(compRoot);

                            DB{samplesN,1} = samplesN;                          % sample Number
                            DB{samplesN,2} = str2double(session(end));          % session number
                            DB{samplesN,3} = str2double(waveFileName(13:14));   % scentence number
                            DB{samplesN,4} = scenarioMap(waveFileName(21));     % scenario type
                            DB{samplesN,5} = labelMap(emotion);                 % emoton label
                            DB{samplesN,6} = genderMAP(waveFileName(17));       % speaker gender
                            DB{samplesN,7} = str2double( waveFileName(18:19));  % speaker ID
                            DB{samplesN,8} = genderMAP( waveFileName(23));      % listener gender   1:male 0:female
                            DB{samplesN,9} = genderMAP( waveFileName(24));      % seaker gender     1:male 0:female
                            DB{samplesN,10} = str2double(waveFileName(25:26));  % turn number
                            DB{samplesN,11} = waveFileName;                     % complete name  of voice
                            DB{samplesN,12} = x;                                % data

                            disp ([num2str(samplesN), '   session : ',session(end), '     label:',waveFileName(15),'  detect: ',emotion,'   EmoIndex: ',num2str(labelMap(emotion)),'    ',  waveFileName])
                            samplesN =samplesN+1;

                        end
                    end
                end
            end
            fclose('all') ;
        end
    end
  end
end

other functions that are used in the above function is listed below:
1- scenario mapping function:
function [code]=scenarioMap(character)

if character=='T'
    code = 0;
elseif character=='S'
    code = 1;
elseif character=='P'
    code = 2;
elseif character=='R'
    code = 3;
else
    code = 4;
end

2- label mapping function:
function [code]= labelMap(character)

if character=='A'
    code = 0;
elseif character=='H'
    code = 1;
elseif character=='S'
    code = 2;
elseif character=='N'
    code = 3;
elseif character=='O'
    code = 4;
    disp('--------zzzzzzz-----')
elseif character=='W'
    code = 4;
    disp('--------zzzzzzz-----')
else
    code = 4;
end

3-Gender mapping function:
function [gender ]= genderMAP(str)

if strcmp(str,'F')
    gender = 0;
elseif strcmp(str,'M')
    gender = 1;
end


Multiple digits MNIST and transfer learning

You need to manually or automatically draw bounding boxes around each digit. Then compute the area for each box and take the largest.

Given they are non-overlapping black digits on a white background, template matching would work.


What are limitations of decision tree approaches to data analysis?

Simple decision trees have some limitations listed below. Fortunately, some of these can be fixed used ensemble learning techniques (think bagging, boosting...).
Concerning limitations :

Trees tend to overfit quickly at the bottom. If you have few observations in last nodes, poor decision can be taken. In this situation, consider reducing the number of levels of your tree or using pruning.

Trees can be unstable because small variations in the data might result in a completely different tree being generated.

Decision trees perform greedy search of best splits at each node. This is particularly true for CART based implementation which tests all possible splits. For a continuous variable, this represents 2^(n-1) - 1 possible splits with n the number of observations in current node.

For classification, if some classes dominate, it can create biased trees. It is therefore recommended to balance the dataset prior to fitting.
Also, Some distributions can be hard to learn for a decision tree. An example below (XOR) :


Relation mining of multivariant categorical timeseries without excluding the temporal nature

This problem is one of estimating the lag. Once that is estimated, you could create additional features representing the lagged values and move forward with "sequence mining" as you have already suggested in the question itself.

For each variable, Var_i, you will have to estimate its lag l_i. This lag can be calculated by estimating the order of a Markov chain with seven symbols (you could use either BIC or AIC to estimate this order; both would require calculating likelihood of candidate orders and pick the order that maximizes either of these criteria). Once you are done calculating the order of the Markov chain for each of the variables, then you could represent your dataset such that each row will have the current value of Var_i and its preceding values, all the way back to its estimated lag l_i. While this methodology is laborious, it pays rich dividends as its automated and parsimonious way of representing the necessary information.


Solve a pair of coupled nonlinear equations within certain limits

As a workaround, you could minimize another function that includes both the objective and the constraints, then check if sol.fun is (numerically) equal to zero.

from scipy.optimize import minimize
import numpy as np

f = lambda x: np.sin(x).sum() #the function to find roots of
L = np.array([-1,-1]) #lower bound for each coordinate
U = np.array([1, 1])  #upper bound for each coordinate
g = lambda x: f(x) **2 +  max(0, (L - x).max(), (x - U).max())
sol = minimize(g, [0.5,0.5])

Also, scipy.optimize seems to have some optimisers that support rectangular bounds, i.e. differential_evolution (since version 0.15.0).


Attempt at a Conceptual Approach

I will attempt to answer this question focusing primarily on the concept itself, rather than the mathematics which have little explicit relevance. Therefore, correct me if I am wrong, but I believe that when you state "Localized Filter" you are referring to  what is also, commonly known as a "Convolution Layer" or "Neuron" even occasionally a "Kernel". In a manner, of which I hope will be suitable to your application, I would have one visualize this as a sort of mask, who's depth MUST equal that of the source image depth, that is to be masking the source input data, centered at some point (x,y,z), then sequentially shifted one position at a time in a manner such that, the area covered by the filter in the second position will be overlapping the first all but one column. To better understand this concept, imagine the way the data input to a CNN would be "seen" by the machine, the following images attached to this guide, show the general concept; although the picture above this paragraph is AN ACTUAL GRAPHICAL REPRESENTATION OF A NEURON, the same principle applies to both:

  Some array -> N1xN2xN3 of numbers.

After successfully shifting, the algorithm will then perform an element wise multiplication across the values in the original matrix(X0) that the "conv layer" currently Filters and its own numeric values(X1) then stored in some other array (A), referred to as the "Activation Map", that becomes some degree smaller than the original data in relation to the filter size:

  np.multiply(X0,X1)

Of which the result is stored in some Matrix, of numerals, Pt, position total. Upon completion of the sequence of the entire original input data, the algorithm
then computes the sum of all elements and stores them, so to gain the ability to pass to the next layer of the CNN:

  Ptfinal = sum(Pt)

The filter itself, being representative of some pattern, for example a circle, would then result in the outcome of a large number after completing the second step if there was another circle within its area as the pixels that have any data within them will result in an element of the original array having some positive number. As an example reference the below picture's and notice the difference in the resulting Integers for both instance's:

Where as the opposite is true, of an area's values of which do not match those of the "conv layer". This scenario will result in a very low number, relative to a confirmed match, or even a zero if the element happens to be empty of data other than zero. This method is what comprises the "secret layer" in a CNN, which generally alternates in a sequence between different variants, "Convolution", "ReLU" as well as "Pool" layers.

I hope that this explained better what is going on in the localized filters that you came upon in your research! If I was to ambiguous on any part of the explanation, please feel free to espouse and I will try to extrapolate.


How to write custom de-identification algorithm in Python?

Unless you want to build your own, try the Faker Library for anonymity of PPI info.

pip install Faker


Which supervised machine learning algorithms assume normally distributed feature variables?

If you are distinguishing between Statistics and Machine Learning, then you need to define boundary between the two, and that boundary is going to be opinion-based. It is a matter of definition which algorithms fall into Machine Learning and which under Statistics.
I'll try to give some examples without committing to one or the other.

Linear regression expects the errors (residuals) to be normally distributed. This comes from the maximum likelihood and the $x^2$ term in the normal distribution's PDF.
Logistic regression expects the log-odds of class membership to be linear. This is given for two normally distributed classes with equal variance. It follows from the Bayesian probability.
Linear discriminant analysis expects two normal-multivariate distributed classes with the same covariance matrix.

You can apply these algorithms even if your data violate the assumption, but then the results (in statistics: parameter estimates) won't be (quite) correct. How much they'll deviate from the real values depends on the extent of the assumption violation.


Modeling when the response variable has too many 0's and few continuous values?

How about

Ordinary Least Square (OLS) regression? Since you have a class imbalance, you might want to combine that with boosting algorithms.
If you have a function to quantify the cost involved with FP's and FN's, use any optimization technique you can find. My favorite is genetic algorithms. You may also try linear programming.


Help training Back-propagation Neural Network With 150k training pairs

Typical things to look for when training neural networks:

Learning Curve: a plot of the error on the validation set X size of training set. The error must decrease as the network is trained, so if you see fluctuations or an increase something is wrong.
Learning Rate: the learning rate influences the convergence of gradient descent. A small learning rate will make the model overfit the data while a big one may make it underfit.
Precision: of the classified examples in the training set, how many did the network got right, or more precisely, how many did it classify as expected?
Recall: of all the positive examples, in your case the paid ones, how many did it classify as paid?

If you notice the performance of the network is bad you may:

Add other features that make the model fit the data better - for example add average family income to the features
Get more training examples
Some papers suggest deeper networks are able to catch subtleties about the model they are trying to fit, so experimenting with the network architecture is an option.


As a research area, Deep Learning is really just a sub-field of Machine Learning as Machine Learning is a sub-field of Artificial Intelligence.

1) Unsupervised Feature Learning

Conceptually, the first main difference between "traditional" (or "shallow") Machine Learning and Deep Learning is Unsupervised Feature Learning.

As you already know, successfully training a "traditional" Machine Learning model (ex: SVM, XGBoost...) is only possible after suitable pre-processing and judicious feature extraction to select meaningful information from the data.
That is, good feature vectors contain features distinctive between data points with different labels and consistent among data points with the same label.
Feature Engineering is thus the process of manual feature selection from experts. This is a very important but tedious taks to perform!

Unsupervised Feature Learning is a process where the model itself selects features automatically through training.
The topology of a Neural Network organized in layers connected to each other have the nice property of mapping a low-level representation of the data to a higher-level representation. Through training, the network can thus "decide" what part of the data matters and what part of the data doesn't. This is particularly interesting in Computer Vision or Natural Language Processing where it is quite hard to manually select or engineer robust features.

(picture credits: Tony Beltramelli)

As an example, let's assume we want to classify cat pictures. Using a Deep Neural Net, we can feed in the raw pixel values that will be mapped to a set of weights by the first layer, then these weights will be mapped to other weights by the second layer, until the last layer allows some weights to be mapped to numbers representing your problem. (ex: in this case the probability of the picture containing a cat)

Even though Deep Neural Networks can perform Unsupervised Feature Learning, it doesn't prevent you from doing Feature Engineering yourself to better represent your problem. Unsupervised Feature Learning, Feature Extraction, and Feature Engineering are not mutually exclusive!

Sources:

http://deeplearning.stanford.edu/tutorial/
https://arxiv.org/abs/1404.7828
https://arxiv.org/abs/1512.05616 (Chapter 2, Section 2)

2) Linear Separability

Deep Neural Networks can solve some non-linearly separable problems by bending the feature space such that features become linearly separable.
Once again, this is possible thanks to the network topology organized in layers mapping inputs to new representations of the data.

(picture credits: Christopher Olah)

Sources:
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

3) Statistical Invariance

Lastly, Deep Neural Networks are surpassing traditional Machine Learning algorithms in some domains because some architectures are showcasing Statistical Invariance (ex: Spacial Statistical Invariance with Convolutional Neural Networks and Temporal Statistical Invariance with Recurrent Neural Networks)

Check this Udacity video for more details:
https://www.youtube.com/watch?v=5PH2Vot-tD4


Why dont we use 2d cnn filters for Nlp tasks?

Text is a 1D sequence, but is typically treated as a sequence of embedding vectors. So yes it is in some sense 2D input. But the embedding dimension doesn't really have any spatial meaning; adjacent dimensions aren't any more related than any others. There is no invariance across the embedding dimension either; the same values in one part of the embedding don't mean the same thing. So the assumptions a 2D convolution don't make sense for this type of input.


What you are looking for is a probabilistic classification for 2+ classes (multi class) i.e. evaluating a probability of being associated to a list of k classes given the observation. k = 4 blocks/classes for your example. Its a one vs many situation.

Binary classifiers (like logistic regression) have been extended to multi class problems.

One of the strategy used is to train/test your model k times, each time keeping one class in predicted block intact and turning other three predicted class to a single dummy class. Then use a binary probabilistic classifier(eg. logistic regression) to predict class probabilities. The order of class preference will be in decreasing order of class probabilities.
For logistic regression and for other classifiers, this is inherently built in by scikit and applied by default:

lr = linear_model.LogisticRegression(multi_class = 'ovr')
# ovr is "one vs rest"
lr.predict_proba(test_features)

refer to Multi-class capability built
 in scikit

Yes, you can even use a pruned decision tree to get the class probabilities. But most probably you will not be able to get 2nd, 3rd... best predictions for most of your observations from a single tree due to the underlying splitting mechanism of algorithm.

 dt = tree.DecisionTreeClassifier(min_samples_split=25)
 dt.predict_proba(test_features)

Therefore, rather then relying on a single decision three, better option would be to use a randomforest classifier to obtain proportion of votes going to each class while making a prediction for an observation.

In short, any multi-class classifier (or ensemble) which spits out a likelihood over multiple classes will do. Another eg. XGBoost as pointed out in below comment by bradS.


Identify credit card shape using machine learning

I am only concerned about finding the outline of card in an image and isolating the outline of card from rest of the image.

This can be efficiently solved by semantic segmentation (aka dense prediction) - problem in which every pixel must be labeled with class.
In your case, you will have 2 classes: credit card and background. And you will need to have annotated dataset in such way: for every image, every pixel should have a class label. If you will be annotating it manually, I guess that credit cards (because of their simple shape) can be annotated easily.
Here is a good solution in Keras for semantic segmentation models. It offers a lot of different architectures and backbones and it will be straightforward to apply it to your problem.
There are also implementations in other frameworks on the web.


BiGAN, Bi-directional GAN(variation of GAN(Generative Adversarial Network)) can be explored for anomaly detection.
I have used for my use case. Since I dont have large image dataset, my results are not that good.


Maximum likelihood estimation vs calculating distribution parameters "manually"

I think you are asking about how estimates for these parameters from your data sample (obtained by calculating moments) would be different from maximum likelihood estimation. If so, check out:
https://stats.stackexchange.com/questions/252936/what-is-the-method-of-moments-and-how-is-it-different-from-mle
http://gradquant.ucr.edu/wp-content/uploads/2013/11/Maximum-Likelihood-and-Method-of-Moments-Estimation.pdf


ARIMA could work, I think it's the right approach. It's simple enough to be used on a small dataset, but sufficiently flexible at the same time. If you are using Python, library statsmodels allows you to implement ARIMA regressions. You have to grid search and find the right parameters to find the best fit, and run the prediction.
If you want to know how to do it, take a look here and here.
Alternatively, even simpler models could work correctly, such as a simple moving average (MA), or auto-regression (AR). But that's something you can find from the ARIMA grid search above.


How can we create neural net to detect false predictions?

One thing you could try is to change the softmax activation on your final layer to sigmoid. This will give a value between zero and one for each class, the sum of which may not equal one. This means that for a given example there may be more than one output neuron activated, or no output neurons activated, in which case you can treat the example as 'unsure' and manually check it out. In the case that there is exactly one output neuron activated (should be most of the time), that should be the classification given.


What's the difference between keras api augmentation and data augmentation definition?

The Augmentation api apply different augment defined by you (horizontalflip, verticalflip, rotation, zcawhitening etc) to your original image rather than making a copy of it. It apply different augmentation on the same image on different epoch which reduce the generalization error.


Best way to scale across different datasets

Your observation stems from two facts

Model performance deteriorates when data distribution changes (shifts). For example, when model is trained on Boston dataset but tested on Paris dataset, or trained on summer dataset but tested on winter dataset.
Model evaluation (wrongly) shows an improvement when information leaks from test set. For example, when test set is scaled with must-not-be-known statistics of the test set like (min, mean, std, max).

If a data set for other cities like Paris is not available at the training time (1), obviously there is nothing we can do. In real world cases, when we see a noticeable difference between validation score and test score, we hypothesize that the test set has a different distribution than the training set, thus, we try to incorporate the new data and re-train (probably not a full re-training in an online learning setting). Otherwise (2), we can

Combine all the cities into one dataset, then split it into train and test sets (later, validation set will be peaked from train set), i.e.

  train = A_train U B_train U ...
  test = A_test U B_test U ...

This way, the information from all cities will be used for scaling. We may under- or over-sample some cities to have a data set with equal representatives from each city.
Or, build a separate (specialized) model for each city (or a group of similar cites) which bypasses the "train on A_train, and  test on B" problem all together.


Loss given Activation Function and Probability Model

At the output we can have an activation that takes a weighted linear sum from the previous layer and transforms it into a new range. Here are a few activations you could use:

$x = x$, the identity activation which is what is usually used at the end for regression models

$\sigma(x) = \frac{1}{1+e^{-x}}$ which maps to $(0, 1)$

$exp(x) = e^x$ which maps to $(0,\infty)$

These activations are differentiable which means backpropagation will keep working. Now that we can manipulate our output to turn into ranges that we want, we can use them to use them for probability distributions. In the $\sigma$ case we can use them directly as the probability at event $Y$ as opposed to not $Y$, so $P(Y|X)$. We can also use multiple output nodes to parameterize a probability distribution. For example let's try to approximate a Gaussian distribution conditioned on our input features $x$. Instead of regressing directly on $y$ we can regress $\mu$ and $\sigma$ that define the Gaussian distribution. For $\mu$ we can use the identity activation because $\mu$ is not constrained, $\sigma$ needs to be strictly positive so we could use $e^x$ for that.

However, in our training data, we don't have $\mu$ and $\sigma$, we only have a $y$, a sample from this probability distribution. What we do know is that given a specific $\mu$ and $\sigma$ what the likelihood is of that $y$. A value of 100 is extremely unlikely from a normal distribution with $\mu = 20$ and $\sigma=20$, but a value of $20$ has a much higher density. The density of a Gaussian is $f(y|\mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(y-\mu)^2}{2\sigma^2}}$. This means we can check the likelihood of a $y$ given our current estimate of $\mu$ and $\sigma$ given $x$ and try and make the likelihood higher. Or minimize the negative log-likelihood, which has less numerical issues as opposed to normal likelihood.


Multiclass classification on imbalanced dataset : Accuracy or micro F1 or macro F1

There are two not so widely known in the data science community metrics that work well for imbalanced data and can be used for multi-class data: Cohen's kappa and Matthews Correlation Coefficient (MCC).

Cohen's kappa is a statistic that was designed to measure inter-annotator agreement, but it can be used to measure agreement between the ground truth and a prediction. There are number of explanations online e.g. on Wikipedia or here and it is implemented in scikit-learn.

MMC was initially designed for a binary classification but then generalized for multi-class data. There are also multiple online sources for MCC, e.g.  Wikipedia and here, and it is implemented in scikit-learn.

Hope this helps.


I found an intermediate solution between minimal representation and full rotation matrix, which is using two unit vectors indicating two orthogonal directions (e.g. forward and right) with respect to the orientation. These can be learned simply through squared differences and are simple to use (in my case the animation runtime already had functions to compute an orientation given two directions, taking into account that they may not be perfectly orthogonal). This representation takes 6 values per orientation which is not as good as 3 or 4 but not too terrible either, and in my experiments it seems to work just as fine, only without the unstable cases.


Using Machine Learning to play the game SET

So you would like to play this game using images?

If that's the case, I suggest dividing your problem into 2 sub-problems: Image classification (supervised learning) and reinforcement learning.

The first sub-problem will detect the right card and the second sub-problem takes that as an input.

When it comes to game, it's definitely worth trying reinforcement learning. You will only need to create a reinforcement learning environment for this game and use some off-the-shelf algorithms (like OpenAI baselines).


You are right that LSTMs work very well for some problems, but some of the drawbacks are:

LSTMs take longer to train
LSTMs require more memory to train
LSTMs are easy to overfit
Dropout is much harder to implement in LSTMs
LSTMs are sensitive to different random weight initializations

These are in comparison to a simpler model like a 1D conv net, for example.

The first three items are because LSTMs have more parameters.


Predicting similarity between nouns like university names and tech companies?

This is a case of entity resolution for which a standard method is not available. You will have to write your own method also using abbreviation resolution. The python Dedupe package has some distance metrics which you could use to calculate the similarities.


Can we train a neural network to tell if an object is present or not in an Image?

A standard computer vision and deep learning dataset for this problem was developed by the Canadian Institute for Advanced Research (CIFAR). The CIFAR-10 dataset consists of 60,000 photos divided into 10 classes (hence the name CIFAR-10). Classes include common objects such as airplanes, automobiles, birds, cats and so on. The dataset is split in a standard way, where 50,000 images are used for training a model and the remaining 10,000 for evaluating its performance. Other datasets are MNIST, CIFAR-100 STL-10, SVHN, ILSVRC2012 etc

State of the art results are achieved using very large Convolutional Neural networks. You can learn about state of the are results on CIFAR-10 on Rodrigo Benenson’s webpage. Model performance is reported in classification accuracy, with very good performance above 90%.

Here is a sample tutorial on convolutional neural network with caffe and python


I feel that is something more fundamentally wrong if your test scores are good but your predictions on new data are bad. I think it sounds like your new prediction data is not coming from the same distribution as your train/test data. That means it is different in some way which will cause you model to perform more poorly on the new data.


Is a CNN architecture more susceptible to adversarial attack if it is built on ResNet?

No.  Both standards CNNs and ResNets are highly vulnerable to adversarial examples; neither one is particularly better than the other.  "Open source" is not really relevant here.

If you're asking about ability to craft adversarial examples when the attacker doesn't have access to the model itself, you might be interested in black-box algorithms for generating adversarial examples.  There are ways known that an adversary can find adversarial examples without having access to the model.  Broadly speaking, there are two common attack approaches:

Exploit transferability: Generate adversarial examples for some other model that is known; then they will often fool the secret model as well.
Exploit API access: if you can make queries to the model (send it an input and get back the label that the classifier outputs), then there are new attack algorithms that are able to find adversarial examples.  ZOO is one example, but there are more recent advances as well.

Broadly speaking, there is no known defense for adversarial examples that seems to be very effective in general.


I don't know precisely what's happening under the hood, but your list comprehensions on lines 31-33 are wrong and causing numpy to produce some funky arrays.

If you pull each of them out of the np.array calls and attempt to run them, you'll generate a nice SyntaxError (which is nice because it'll loudly tell us what's wrong!).
Right now, the error is getting suppressed by the fact Python is taking that funky list comprehension and creating a generator expression (I'm guessing due to how Python manages argument packing).

I'm unsure of the intent of lines 31 and 32 (are they supposed to be individual records from the data?), but line 33 can be rewritten as:

rand_y = np.array([binary_target[y_i] for y_i in rand_index])


The first issue seems to be in the following block of code:
# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 0)
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 1)
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(iris_y_test, y_test_prob[:,1], pos_label = 2)
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

You are feeding y_test_prob[:,1] to each roc_auc calculator (when you should be feeding column 0, 1 and 2). In this case the predict_proba method outputs one column per class. Additionally you are feeding iris_y_test into this calculator which is numerically encoded with distinct values 0, 1 and 2. This function wants one column with the probabilities of each class on it's own. See a possible alteration:
import pandas as pd

# Calculating the roc curve for each class changing the pos_label value
fpr_cl0, tpr_cl0, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 0], y_test_prob[:, 0])
roc_auc_cl0 = auc(fpr_cl0, tpr_cl0)
fpr_cl1, tpr_cl1, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 1], y_test_prob[:, 1])
roc_auc_cl1 = auc(fpr_cl1, tpr_cl1)
fpr_cl2, tpr_cl2, _ = roc_curve(pd.get_dummies(iris_y_test).loc[:, 2], y_test_prob[:, 2])
roc_auc_cl2 = auc(fpr_cl2, tpr_cl2)

Secondly - looking at your output probabilities they are all either distinctly 0 or 1. Your decision tree seems to have "perfectly" classified each example into a class with 100% certainty.
The ROC AUC curve will only show interesting results if there are multiple possible thresholds to classify your samples into different classes. In your case - any threshold you pick between 0 and 1 exclusively will yield the exact same False Positive and True Positive rate. The ROC AUC curve helps to determine thresholds for classification - and in your case selecting thresholds is quite trivial.


Is it right to impute Train and Test set?

Yes, it is possible to impute both the train and the test set. You have to be careful not to introduce information leakage by splitting - if you impute for the train set, then use the same imputation process for the test set as well. I believe that was mentioned in a comment as well. Here is some further information:

See https://stats.stackexchange.com/questions/95083/imputation-before-or-after-splitting-into-train-and-test


how would I have to format my data to make it work with Keras?

Your training labels in the output layer should be a binary vector that is 1 for class which is present and 0 for class which is not. For example, let us assume you have 3 classes of genres - comedy, romantic and horror. There are many ways to make it and Scikit-learn has a method which makes it very easy which I show below.

>>> from sklearn.preprocessing import MultiLabelBinarizer
>>> mlb = MultiLabelBinarizer()
>>> y = mlb.fit_transform([[0,2],[1]])
array([[1, 0, 1],
   [0, 1, 0]])

  I initially considered using a softmax layer as my output layer, but since a movie can have multiple genre labels, how should my output be?

This is a simple Keras example I suggest.

>>> from keras.models import Sequential
>>> from keras.layers import Dense, Activation

>>> model = Sequential([
    Dense(32, input_dim=784),
    Activation('relu'),
    Dense(10),
    Activation('sigmoid'),
    ])
>>> model.compile(optimizer='rmsprop', loss='binary_crossentropy')
>>> model.fit(X_train, y_train)

Refer this for more info. I used sigmoid because it is better for multilabel classification.


Data normalization in nonstationary data classification with Learn++.NSE based on MLP

Batch normalization is critical technique for fast learning
speed and generalization [8]. In this paper, batch temporal
normalization layer is proposed for stationarity of input
time series.


Organizing a csv file of multiple datasets into a list of Pandas dataframes

It is possible to solve this problem procedurally by thinking line-by-line or streamline-by-streamline.
For each streamline that can be matched with IntegrationTime == 0.0, extract the slice from the data frame, and append the slide to an output list (if it has more than 1 data point). A code like the following should address this problem:
import pandas as pd

# read the dataset
df = pd.read_csv("vel.csv")

# get row indexes where integrationtime is zero (start of each streamline)
start_index_list = df.loc[df['IntegrationTime']==0.0].index.values
stream_line_list = [] # the output list

for i in range(len(start_index_list)):
    # for each stream line, obtain the slice of the original dataframe that corresponds to it
    start_index = start_index_list[i]
    end_index = None
    if i+1 < len(start_index_list):
        end_index = start_index_list[i+1]-1
    stream_line_df = df.loc[start_index:end_index]
    # only append streamline with more than 1 data point
    if len(stream_line_df) > 1:
        stream_line_list.append(stream_line_df)

print(f"number of complete streamlines found: {len(stream_line_list)}")
```


t-SNE: Why equal data values are visually not close?

You're correct that the same values in T-SNE can be distributed across different points, the reason this happens is clear if you take a look at the algorithm that T-SNE runs across.

To address your first concern about the points actually not being the same after the algorithm has been applied to the dataset. I will leave you with an exercise to verify it for yourself, consider a simple array $x_1 =
 [0,1]$ and $x_2 = [0,1]$ and run actual algorithm against it and see for yourself that the resultant points are not actually identical.You can cross reference your answer against this.

import numpy as np
 from sklearn.manifold import TSNE
 m = TSNE(n_components=2, random_state=0)
 m.fit_transform(np.array([[0,1],[0,1]]))

You'd also observe that changing the random_state actually modifies the output co-ordinates of the model. There isn't any real correlation between the actual co-ordinates and their output. Since the first step of TSNE computes the conditional likelihood.

Let us now try to rationalize, using the algorithm the reason this happens, just using math, without any intuition, for the moment. Note, that $x_i$ and $x_j$ are both vectors in this situation. $p_{j | i} = \frac{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}{\sum_{k \neq i}{exp(\frac{-||x_j - x_i||^2}{2\sigma^2})}}$. Now, if we calculate the $p_{ij} = \frac{p_{i|j} + p_{j | i}}{2N}$, we can see that the value is 1. After applying KL divergence, we obtain the values stated above. Now, let us apply some intuition to this. $p_{ij}$ is, informally, the conditional probability that $x_i$ would pick $x_j$ as it's neighbor. This justifies the result, 1, for two reasons. The first, being that there is no other neighbor, so it must pick the only other vector in the list of co-ordinates. In addition, the points are identical and the chances that they are picked as the others neighbor should be high, as we see.

Now coming to whether the absolute co-ordinates in $\mathbb{R}^2$ have any significance. They really don't. The randomness can redistribute the points wherever you want them to go. However, what is more interesting is the ratios of the distances between the points and these are relative and are relative even when we project it onto higher dimensions which is pretty interesting.

So, the truth is, rather than looking at the two clusters, look at the distances between them, because that conveys more information than the co-ordinates themselves.

Hope this answered your question :)


Measuring similarity for sets with same cardinality

Yes, the Jaccard similarity score is normalized by the union to deal with sets of different cardinality. Without this normalization (if you used just the intersection), very small sets would always have very low scores.

When the cardinalities of all your sets are the same, the union of any two sets will be a straightforward function of the intersection (this is easy to visualize - as the two sets intersect more and more, their unions get smaller and smaller). The formula is:

union = 2 * cardinality - intersection

So the Jaccard score in your case would be:

intersection / (200 - intersection)

If you plot this, you'll see it's monotonically the same as what you did.


Noise Elimination with majority vote filtering

I have a few questions on which I can't find the answers elsewhere

It's probably because there is no simple answer to these three questions :)

I doubt there's any state of the art approach, in such cases I simply try to determine the answer to these questions empirically. Basically I create a list of hyper-parameters including the type of algorithm, the algorithm-specific hyper-parameters and any other potentially relevant option. The goal is to determine the optimal combination of values for the set of parameters. If practical I run all the combinations and select the best one. If not practical, I use a simple genetic algorithm to find an optimal combination. Of course it's suitable only if you have a dataset large enough and if the training/testing process is not too computer-intensive. You also need to be very careful about overfitting by using cross-validation and re-sampling.


Principal components analysis need standardization or normalization?

I believe Normalization refers to scaling the variable in between 0 and 1. Standardization refers to making the empirical distribution $Y\sim N(0,1)$. Principal component analysis, and similar methods such as Ridge Regression and Partial Least Squares regression, require standardization before training, i.e. $y_{i}=\frac{y_i-\mu_y}{\sigma_{y}}$, reference: Elements of Statistical Learning, Ch. 3.4


What is the error rate for random guessing?

Random guessing of value of whcih column? For example random guessing the gender of each row is: probability of it is male but guess female + vice versa which is $\frac{7}{10}\times\frac{1}{2} + \frac{3}{10}\times \frac{1}{2} = \frac{1}{2}$ and it means the error of random guessing the value of Gender is $50\%$.
Also you can compute for the others by the possible value for each column. For example for the Hair Length the error would be $\frac{2}{3}$ or $66.66\%$.


A Siamese network (a network with multiple outputs) will work for such a case.


Constrained k-means algorithms in R (must-link constraints)

For the given problem I would apply constrained Hierarchical clustering in R. That is more efficient in this case. If you like to apply Hierarchical clustering the package is "rioja" and the function that you can use is chclust().
This is not constrained K-means implementation in R but this should solve your actual problem.

There are two types of constrained in chclust() function, you can set those with the help of method parameter in function. This will add the constraint on the basis of sum of square.You can read in detail about these constraint with simple search.

This chclust function will produce the constraint object and you can check the proper number of group by using other functions in "rioja" package.

A practical implementation of same can be checked on below link:
https://gist.github.com/benmarwick/7097120


How to handle negative integer values in a data set?

There are no problem with negative inputs values, as long as it's mean something. As Ankit Saith said in the comment, temperatures can be negative, money too (positive is money I earn, negative the money I lose) and so on. Of course, inputs like distances should not be negative !

Furthermore, generally in deep learning, you normalize your dataset to have inputs with 0 mean and a std of 1. Then you have "small" value around 0 which can be positive or negative !


Check out Code Folding and Sections:

  Code sections allow you to break a larger source file into a set of
  discrete regions for easy navigation between them. Code sections are
  automatically foldable—for example, the following source file has
  three sections (one expanded and the other two folded):

  To insert a new code section you can use the Code -> Insert Section
  command. Alternatively, any comment line which includes at least four
  trailing dashes (-), equal signs (=), or pound signs (#) automatically
  creates a code section. For example, all of the following lines create
  code sections:

  # Section One ---------------------------------
   # Section Two =================================
   ### Section Three #############################

  Note that as illustrated above the line can start with any number of
  pound signs (#) so long as it ends with four or more -, =, or #
  characters.

(highlights by myself)


Creating an LSTM NN for Fault Classification with Keras

It is very simple.

Just add a Dense layer (Keras-wise) with one unit after your LSTM network, with sigmoid activation. If you don't use Keras, Dense layer is simply a fully-connected neuron with one unit (in your case), which you can easily implement in the deep learning framework of your preference.

Then you can train your network to directly classify the input sequence using soft-labeling, so that it outputs a probability from 0 to 1 that a sequence is anomalous. Needless to say, you can have an arbitrary number of inputs, not just one.

Bottom line: use a fully-connected layer with one unit (neuron) and sigmoid activation after the LSTM output


Fully-Configured Deep Learning Virtual Machines in Python (VirtualBox or VMware)

downnload here.

Python 3.5
OpenCV 3.2 with Python 3 bindings
dlib 19.9 with Python 3 bindings
TensorFlow 1.5 for Python 3
Keras 2 for Python 3
face_recognition for Python 3 (for playing around with face recognition)
PyCharm Community Edition already set up and ready to go for all these libraries


What is the underlying pattern, or algorithm(s) used in these numbers?

11 instances of data is insufficient for discovering the behavior (even finding patterns) for such presumably complex systems. You need at least thousands of them.
These numbers are encoded versions of numbers generated by Serial Key Generation Algorithms, which in my experience are usually implemented by Python. So it's not at all strange that there may be a correlation between some digits of them. As it is necessary for the system to recognize its created numbers from the numbers created by other lotteries.
Even if there was a correlation or correlation between all the instances of lottery numbers, correlation doesn't imply causation and these patterns, if found any, couldn't really give any information about the key generation system itself.
If you are really serious about it. you could go to this page and enter your data as a sequence it would show you the pattern, if there is any:
The On-Line Encyclopedia of Integer Sequences


How to extract the reason behind a prediction using TensorFlow?

[updated]

You are looking for GradCam.

Code:

https://fairyonice.github.io/Grad-CAM-with-keras-vis.html
https://tf-explain.readthedocs.io/en/latest/index.html

Paper: https://arxiv.org/abs/1610.02391

This is a good resource I found
https://christophm.github.io/interpretable-ml-book/


Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in commercial products, you have to acquire commercial license.
Another interesting set of open source libraries, IMHO suitable for this task and much more, is parallel framework for machine learning GraphLab, which includes clustering library, implementing various clustering algorithms. It is especially suitable for very large volume of data (like you have), as it implements MapReduce model and, thus, supports multicore and multiprocessor parallel processing.
You most likely are aware of the following, but I will mention it just in case. Natural Language Toolkit (NLTK) for Python contains modules for clustering/classifying/categorizing text. Check the relevant chapter in the NLTK Book.
UPDATE:
Speaking of algorithms, it seems that you've tried most of the ones from scikit-learn, such as illustrated in this topic extraction example. However, you may find useful other libraries, which implement a wide variety of clustering algorithms, including Non-Negative Matrix Factorization (NMF). One of such libraries is Python Matrix Factorization (PyMF). Another, even more interesting, library, also Python-based, is NIMFA, which implements various NMF algorithms. Here's a research paper, describing NIMFA. Here's an example from its documentation, which presents the solution for very similar text processing problem of topic clustering.


Calculate Similarity using User's Personal Data?

You can use the scipy module to calculate similarities. For your example, this would work like this:

import pandas as pd
users = ["U1", "U2", "U3", "U4", "U5", "U6"]
X = pd.DataFrame({
"Dep1": [0, 1, 1, 0, 0, 1],
"Dep2": [1, 0, 0, 1, 1, 0],
"Comp1": [0, 0, 1, 0, 0, 1],
"Comp2": [1, 1, 0, 1, 1, 0],
"Site1": [0, 1, 1, 0, 1, 0],
"Site2": [1, 0, 0, 1, 0, 1],
"Sitex": [1, 0, 0, 1, 0, 1]

}, index =users)
X

Out[2]:
    Comp1  Comp2  Dep1  Dep2  Site1  Site2  Sitex
U1      0      1     0     1      0      1      1
U2      0      1     1     0      1      0      0
U3      1      0     1     0      1      0      0
U4      0      1     0     1      0      1      1
U5      0      1     0     1      1      0      0
U6      1      0     1     0      0      1      1

from scipy.spatial.distance import pdist
from scipy.spatial.distance import squareform

# calculate the jaccard distances of the users, you can change jaccard to 'cosine'
distances = pdist(X.values, metric='jaccard')

# pdist returns a non-redundant vector of distances
# distances matrices are always symmetric, and diagonals are always zero
# so it is unnecessary to save all the information
# however if we want a matrix, we can use squareform()

distance_matrix = squareform(distances)

# the distances are bewteen 0 and 1, we get the similiarities by using 1-distances
similiarities = 1-distance_matrix

pd.DataFrame(similiarities, columns = users, index = users)

Out[3]:
          U1        U2   U3        U4   U5        U6
U1  1.000000  0.166667  0.0  1.000000  0.4  0.333333
U2  0.166667  1.000000  0.5  0.166667  0.5  0.166667
U3  0.000000  0.500000  1.0  0.000000  0.2  0.400000
U4  1.000000  0.166667  0.0  1.000000  0.4  0.333333
U5  0.400000  0.500000  0.2  0.400000  1.0  0.000000
U6  0.333333  0.166667  0.4  0.333333  0.0  1.000000


I’m intentionally skipping reviews with 3 stars; could this be an issue?

It's not a problem, it's just your definition of your question. It assumes rate 3 as neutral and blow/above that as negative/positive. One may say I assume every rate is positive so each number is a level of positiveness then rate 3 works for him. The main point is that you get the answer for the question you define so be careful about what you want from your classifier to set up your question correctly.

  When using a percentage split to generate training and testing sets, should both of them contain the same share of positive and negative reviews?

Nice question. Before specifically taking Naive Bayes into account, it is a general machine learning problem when the population of classes are imbalanced. If this is the case then better to balance them, not only in train/test split but also during train itself as the dominating class will bias your result. If the classes are not that imbalanced then you can split things randomly and it's fine. Regarding Bayes Classifier itself the balance inside training set should be more important as NB learns from the statistics of your training set. On test set, it just uses the already learned statistics so portion of classes should not impact.

  I observed that negative reviews tend to contain longer texts on average. would this have an impact on the way my classifier tries to predict?

Again nice question. Yes it does but that is not necessarily that bad. Longer text means that the probability of having larger number for term frequencies are higher. This is a feature itself! (Just think again that you already found a difference between classes. Well ... you want NB to do the same right?!!) The only thing is that if the difference in lengths of comments is very huge, then it might affect the value of features in a way that you need normalization (e.g. instead of counting terms as features, you can use TF-IDF which is a bounded score)

Hope it helps. Good Luck!


Keras Classifier returns similar output for all Predictions

As you are using sigmoid as activation function in last layer. It will output generate output based on if probability above 50% then it belongs to "W" class and if it is less than 50% belongs to "A" class.
If you can print output probability of different images & share it then it will little helpful for us for understand problem


How can I adjust the legend when visualizing clusters in two dimensions?

Yeah this is an annoying weirdness of seaborn. Just pass legend='full' as a parameter to sns.scatterplot(). So your code becomes:
kmeans = KMeans(n_clusters=10)
y2 = kmeans.fit_predict(scaled_data)

reduced_scaled_data = PCA(n_components=2).fit_transform(scaled_data)
results = pd.DataFrame(reduced_scaled_data,columns=['pca1','pca2'])
sns.scatterplot(x="pca1", y="pca2", hue=y2, data=results, legend='full')
#y2 is my cluster number

plt.title('K-means Clustering with 2 dimensions')
plt.show()

I have no idea why you'd ever want half a legend! But the parameter defaults to 'brief'. Refer to the seaborn documentation:

legend : “brief”, “full”, or False, optional
How to draw the legend. If “brief”, numeric hue and size variables will be represented with a sample of evenly spaced values. If “full”, every group will get an entry in the legend. If False, no legend data is added and no legend is drawn.

edit: To have the clusters named in the format you're after, first create a list in the right format, and then call plt.legend() and pass that list as an argument to legend():
legend = []
for i in np.unique(y2):
    legend.append('Cluster {0}'.format(i))

sns.scatterplot(x="pca1", y="pca2", hue=y2, data=results, legend='full')
#y2 is my cluster number

plt.title('K-means Clustering with 2 dimensions')
plt.legend(legend)
plt.show()

To change the location of the legend, use the loc= parameter in the plt.legend() call. For example plt.legend(legend, loc='upper right') or plt.legend(legend, loc='upper left')


You are looking for Entropy. The higher the entropy, the more imbalanced it is. You can use this function for calculating it.


How to Access a Text File from IBM DSX Scala-Spark Notebook

In order to access files that are in Cloud Object Storage, you need to provide the path of the file in the form of a CloudObjectStorage instance. Add following lines, after the file credentials, to your code:

var soc = new CloudObjectStorage(sc, credentials, null, “bluemix_cos”) var myFile = sc.textFile(soc.url(“your_bucket”,”data.txt”))

In the second code line above, “your_bucket” is the value of “BUCKET” in the file credentials.

There is one last thing we need to do before we are good to go. The CloudObjectStorage constructor accepts credentials in a way that is for the CSV files. Luckily the main difference between the two formats is the HashMap key names; hence, we can easily manage this problem by changing the above credentials manually, as mentioned below:

“IBM_API_KEY_ID” as “apiKey”

“IAM_SERVICE_ID” as “serviceId”

“ENDPOINT” as “endPoint”

“IBM_AUTH_ENDPOINT” as “iamServiceEndpoint”

For further details, you can visit the link below:
http://rehmans.org/2018/01/31/how-to-access-a-text-file-from-ibm-dsx-scala-spark-notebook/


Caret is designed to:

  streamline the model training process for complex regression and classification problems. The package utilizes a number of R packages...

(https://cran.r-project.org/web/packages/caret/vignettes/caret.html)

Caret's train has two draws: a common API for many different models, and performing hyperparameter tuning by default.

Specifically, caret calls the xgboost package. It does some checking of your data before calling xgb; most notably, it uses that to select the objective function (which may cause some confusion). And it provides some hyperparameter distributions to search over (which you can modify if desired). However, it doesn't seem possible to search over the hyperparameters of xgboost that aren't built into the caret list of parameters.

So, it seems to me: use caret for a convenient wrapper and hyperparameter tuner, and use the original package for finer control.


After one hot encoding, instead of columns, my number of rows are increasing

get_dummies function of pandas can also be used for this. Just create a pandas dataframe with your data and subject it to this function. There is no need to list categorical variables:

X_train_df = pandas.DataFrame(data=X_train)
X_train_df_1hot = pandas.get_dummies(X_train_df)

You can specify prefix etc also for new columns is required:

pandas.get_dummies(data, prefix=None, prefix_sep='_',
     dummy_na=False, columns=None,
     sparse=False, drop_first=False,
     dtype=None)

Data as a list of lists is available with .values attribute which can be converted to a numpy array:

housing_cat_1hot = numpy.array(X_train_df_1hot.values)


Regression on multiple datasets with a per-dataset variable

I believe adding the meta-variable X to you dataset and combining them is a good option.
However - why do you have 10 different datasets ? and not a single dataset to start with ?
Is there a special meaning attached to each individual dataset - which is why they are not combined in the first place ? Does that sound like an additional categorical variable you need so as to differentiate between individual datasets ?


Is SuperLearning actually different to stacking,or are they essentially the same thing?

So Ensemble Learning is essentially using multiple learning algorithms and providing the best predictive performance considering all of them. This gives a better detailed description. Now, Ensemble Learning can be broadly divided into multiple types like

Boosting
Bagging
Stacking / Super Learning

Stacking contains a bunch of algorithms along with a learner to ensemble a group of base learners. Going deeper, the term Stacking was used way before Super Learning. Later, when the algorithm was actually developed theoretically and made popular in 2007, it was given the name 'Super Learner'.


In which cases should Genetic Programming be preferred over Artificial Neural Network trained with Genetic Algorithm

One key benefit of genetic programming is that it can generate models that are capable of extrapolation. Neural nets tend to be really good at interpolating, but fall apart when trying to extrapolate.
As an example to highlight this we can think of data fitting.
We can start by using just a very simple example. Let the input data be x={1,2,3} and y={1,4,9}.
In genetic programming the proper choice here would be to use symbolic regression, which evolves an algebraic expression to fit data. Given this training data it would likely evolve the expression $y=x^2$. As you can see this expression is capable of taking new values outside of the training data range and mapping it to a response outside of the training responses. For example, you could supply as input 10 and it would use the evolved expression to give a predicted value of 100.
Alternatively, a neural net is very good at interpolating since it creates a network of connections to connect inputs to responses, but when given new data, it tries to fit the new data into its set of rules built to fit only the training data. So likely, the trained neural net would be able to perfectly map $x\mapsto y$, but if you were to give it the number 10 as input, the closest training value it saw was 3, so it would likely give a predicted response of 9 instead of 100.


Negative log-likelihood not the same as cross-entropy?

I think the issue is that while
$$
\frac{1}{m} \sum_{i=1}^{m}\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}) \rightarrow E_{}(\log p_{model}(\mathbf{y} | \mathbf{x} ; \boldsymbol{\theta}))
$$
is true, the right side is not what is being referred to as the cross-entropy.  Indeed, we have no access to the expectation over the true population distribution / data generating process; a few paragraphs down, we find:

We can thus see maximum likelihood as an attempt to make the model distribution match the empirical distribution $\hat{p}_{data}$. Ideally, we would like to match the true data-generating distribution $p_{data}$, but we have no direct access to this distribution.

They also distinguish throughout that the expectations are over $\hat{p}_{data}$, the training data empirical distribution, not $p_{data}$, the underlying population distribution.
The left side of your limit is already an expectation, but over the finite training data, and that is what is referred to as the cross-entropy.  ("Cross-entropy" is a broader term, for any pair of probability distributions.  Goodfellow et al note this (my emphasis):

Any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution deﬁned by the training set and the probability distribution deﬁned by model.

So, the answer to your questions is that the premise is incorrect: (this) cross-entropy is the same as negative log-likelihood.  Taking your questions with the limiting and population cross-entropy instead, the answer is "we don't have access to the latter".  It would be the better target to be sure, but our lack of that information is the point of modeling in the first place.


Accuracy improvement for logistic regression model

Try:

  mylogit <- glm(VALUE ~ POINT1 * POINT2 * POINT3 * POINT4, data = data1, family ="binomial")

with about 72% accuracy.


Is the number of iterations in gradient tree boosting just the number of trees?

Your first interpretation is correct. One base learner will be added per boosting iteration/round and that is probably what people are referring to when talking about iterations.

From wiki:

  One natural regularization parameter is the number of gradient
  boosting iterations M (i.e. the number of trees in the model when the
  base learner is a decision tree).

Iterations take place in other parts of the algorithm, for instance in the gradient descent, but I don't think that is what is discussed if it is simply referred to as "iterations".

This was really helpful when I was trying to understand GBMs