Access keys of pandas dataframe when using groupby

You may use
df.groupby(['BirthDate', 'ZipCode']).size().reset_index().rename(columns={0: 'n'})

and now you have a data frame that you can easily manipulate.


Random Forest command in R for selected variables

You have to do

RF <- randomForest(sale ~ v1 + v2 + v3, data = TrainSet, importance = TRUE)

this is the formula notation for R. It doesn't make a lot of sense for random forest models, but it is how it works.


How to make overfitting (powerful) model?

Here are some general hyperparameter adjustments to increase model capacity:

Architecture-related:

add additional layers
increase the number of hidden units per layer
remove dropout or decrease dropout rates

Optimizer related:

find the optimal learning rate
train for more epochs

Section 11.4.1 in the Deep Learning Book provides a good overview. The paper "Practical recommendations for gradient-based training of deep architectures" is another good source to understand the effect of different hyperparameters (though it is a bit older).

Moreover, there are specific hyperparameters to increase model capacity for different network types, e.g.

kernel size for CNNs,
sequence length for LSTMs and
embedding dimension for embedding layers.


Understanding one of the assumptions of linear regression: Multicollinearity

Interaction effect and Colinearity have two different meanings.
Multi-colinearity simply tells if two or more predictors are correlated i.e. change in one changes the other. I believe there is no confusion about this as you have also mentioned in your question.
But Multi-colinearity doesn't need the response variable for it to be figured out.
Interaction by definition, are always in the context of how predictors relate to the outcome.

More formally, two or more predictors are said to interact if their combined effect is different (less or greater) than what we would expect if we were to add the impact of each of their effects when considered alone.

A simple and intuitive example can be - consider the effects of water and fertilizer on the yield of a field corn crop. With no water but some fertilizer, the crop of field corn will produce no yield since water is a necessary requirement for plant growth. Conversely, with a sufficient amount of water but no fertilizer, a crop of field corn will produce some yield. However, the yield is best optimized with a sufficient amount of water and a sufficient amount of fertilizer. Hence water and fertilizer, when combined in the right amounts, produce a yield that is greater than what either would produce alone

It is represented as -
$y =β_0+β_1x_1+β_2x_2+β_3x_1x_2+error$
$β_3$ explains the interaction between $x_1$ and $x_2$
Interaction can be Additive (no interaction), Synergistic(+ve), Antagonistic(-ve), Atypical depending on different parameters values.
Reference
Quotes, example and the equation is from -
Feature Engineering and Selection: A Practical Approach for Predictive Models, Max Kuhn and Kjell Johnson
Good read for an in-depth explanation of different Feature engineering concepts.
Internet link


Majority voting in scikit-learn Random forest

To quote from the documentation:

  The predicted class probabilities of an input sample are computed as
          the mean predicted class probabilities of the trees in the forest. The
          class probability of a single tree is the fraction of samples of the same
          class in a leaf.

The predictions are then simply the class with the highest average class probability.

You can find the source code here.

Note that this is different from classical majority voting which is usually understood to be the most common class prediction among trees whereas here the voting happens on the class probability level.

The source code for regression is here:

  The predicted regression target of an input sample is computed as the
          mean predicted regression targets of the trees in the forest.


XGboost with group-specific and individual-specific data

You can use all executives if you believe their compensation is governed by the same function. That is, all things equal, you expect all executives to have equal pay.
If not, perhaps you should add a feature that tells the function of each executive (CEO, CFO...).
Anyhow, the fact that part of the features is the same for multiple examples is not a problem.
It would be a problem if you had very many examples from one company and you're looking for a model that applies to all companies. If you used loads of examples of one company, your model would be biased towards the pay at that company, but since you have 5 from each company, I don't see the problem.


Deconvolutional Network in Semantic Segmentation

There are two main functions they undo.

The pooling layers in the convolutional neural network downsample the image by (usually) taking the maximum value within the receptive field.  Each rxr image region is downsampled to a single value.  What this implementation does is store which unit had the maximum activation in each of these pooling steps.  Then, at each "unpooling" layer in the deconvolutional network, they upsample back to a rxr image region, only propagating the activation to the location that produced the original max-pooled value.

Thus, "the output of an unpooling layer is an enlarged, yet sparse activation map."

Convolutional layers learn a filter for each image region that maps from a region of size r x r to a single value, where r is the receptive field size. The point of the deconvolutional layer is to learn the opposite filter.  This filter is a set of weights that projects an rxr input into a space of size sxs, where s is the size of the next convolutional layer.  These filters are learned in the same way that as regular convolutional layers are.

As the mirror image of a deep CNN, the low-level features of the network are really high-level, class-specific features.  Each layer in the network then localizes them, enhancing class-specific features while minimizing noise.


Same TF-IDF Vectorizer for 2 data inputs

You can use something like this

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
data['tf_idf_q1'] = tfidf_vectorizer.fit_transform(data['q1'])
data['tf_idf_q2'] = tfidf_vectorizer.fit_transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identical'].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)

Combined Model - here you actually learn the transformation for all questions. Then transform each one(question) separately to create features for your model training.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
tfidf_vectorizer = TfidfVectorizer()
raw_data = pd.DataFrame(*raw_data, columns = ['id', 'is_identical', 'q1', 'q2'])
tf_train_data = pd.concat([data['q1'], data['q2']])
trained_tf_idf_transformer = tfidf_vectorizer.fit_transform(tf_train_data)
data['tf_idf_q1'] = trained_tf_idf_transformer.transform(data['q1'])
data['tf_idf_q2'] = trained_tf_idf_transformer.transform(data['q2'])
data_for_model = data[['tf_idf_q1', 'tf_idf_q2', 'is_identical']]
X = data_for_model[['tf_idf_q1', 'tf_idf_q2']].as_matrix()
Y = data_for_model['is_identica'l].as_matrix()
model = Sklearn.LogisticRegression()
model.fit(X, Y)


Imputing missing values by mean by id column in R

If you want the mean, you could use dplyr syntax:

df = structure(list(ID = c(101L, 101L, 101L, 102L, 102L, 102L, 103L,103L, 103L),
                    rating = c(60L, 78L, NA, 60L, NA, 75L, NA, 68L, 70L),
                    Sur = c(0.7687, NA, 0.765, NA, 0.654, 0.435, 0.576, 0.875, NA)),
               .Names = c("ID", "rating", "Sur"), class = "data.frame", row.names = c(NA,-9L))

library(dplyr)
df %>%
  group_by(ID) %>%
  mutate(rating = ifelse(is.na(rating), mean(rating, na.rm = TRUE), rating),
         Sur = ifelse(is.na(Sur), mean(Sur, na.rm = TRUE), Sur))

     ID rating     Sur
  <int>  <dbl>   <dbl>
1   101   60.0 0.76870
2   101   78.0 0.76685
3   101   69.0 0.76500
4   102   60.0 0.54450
5   102   67.5 0.65400
6   102   75.0 0.43500
7   103   69.0 0.57600
8   103   68.0 0.87500
9   103   70.0 0.72550

for other imputations you can look at packages like mi, mice, caret, amelia or simputation. And I'm probably forgetting a lot of others. Personally I like the simplicity of simputation.

For example median imputation is just one line:
impute_median(df, rating + Sur ~ ID)


Advise on making predictions given collection of dimensions and corresponding probabilities

Aggregate Data

Since you're only given aggregate data, and not individual examples, machine learning techniques like decision trees won't really help you much. Those algorithms gain a lot of traction by looking at correlations within a single example. For instance, the increase in risk from being both obese and over 40 might be much higher than the sum of the individual risks of being obese or over 40 (i.e. the effect is greater than the sum of its parts). Aggregate data loses this information.

The Bayesian Approach

On the bright side, though, using aggregate data like this is fairly straightforward, but requires some probability theory. If $D$ is whether the person has diabetes and $F_1,\ldots,F_n$ are the factors from that link you provided, and if I'm doing my math correctly, we can use the formula:
$$ \text{Prob}(D\ |\ F_1,\ldots,F_n) \propto \frac{\prod_{k=1}^n \text{Prob}(D\ |\ F_k)}{\text{Prob}(D)^{n-1}} $$
(The proof for this is an extension of the one found here). This assumes that the factors $F_1,\ldots,F_n$ are conditionally independent given $D$, though that's usually reasonable. To calculate the probabilities, compute the outputs for $D=\text{Diabetes}$ and $\neg D=\text{No diabetes}$ and divide them both by their sum so that they add to 1.

Example

Suppose we had a married, 48-year-old male. Looking at the 2010-2012 data, 0.73% of all people get diabetes ($\text{Prob}(D) = 0.73\%$), 0.77% of married people get diabetes ($\text{Prob}(D\ |\ F_1)$$= 0.77\%$), 1.02% of people age 45-54 get diabetes ($\text{Prob}(D\ |\ F_2) = 1.02\%$), and 0.70% of males get diabetes ($\text{Prob}(D\ |\ F_3) = 0.70\%$). This gives us the unnormalized probabilities:
$$ \begin{align*} P(D\ |\ F_1,F_2,F_3) &= \frac{(0.77\%)(1.02\%)(0.70\%)}{(0.73\%)^2} &= 0.0103 \\
P(\neg D\ |\ F_1,F_2,F_3) &= \frac{(99.23\%)(98.98\%)(99.30\%)}{(99.27\%)^2} &= 0.9897 \end{align*}$$
After normalizing these to add to one (which they already do in this case), we get a 1.03% chance of this person getting diabetes, and a 98.97% chance for them not getting diabetes.


Why are my Decision Tree Leafs not pure?

With default settings the DecisionTreeClassifier does not have any restrictions in terms of complexity as described in the Scikit Documentation.

Therefore, it will stop further branching the tree if either a given node is pure (all examples have the same classification) or there are no further attributes to branch on.

So if your final tree contains leaves which are not pure (for the data set it has been trained on) the algorithm did not have any attributes to further split on.

In case you applied any kind of randomization on your data like a random split of training and test data this might turn out differently when splitting again and getting a different training set.


Yes, it's possible, especially using Deep learning.
To get an idea, you can see this github project:
 https://github.com/HuaizhengZhang/Awsome-Deep-Learning-for-Video-Analysis


Pandas - read CSV with spanish characters

read_csv has an optional argument called encoding that deals with the way your characters are encoded.

You can give a try to:

df = pandas.read_csv('...', delimiter = ';', decimal = ',', encoding = 'utf-8')

Otherwise, you have to check how your characters are encoded (It is one of them).

You can read the doc of read_csv here


Should I have "normal" sampled data in my dataset?

In short, yes.

If your goal is to understand the drivers behind poor school performance (or even model + predict school performance), you will need both "positive" and "negative" outcomes in your dataset.

Of course, you need to be careful that the positive outcomes are "similar" enough to the negative outcomes to avoid introducing distortions into your data - i.e. ensure that the data you are adding are from middle-class families with access to facilities etc.

You should also consider the ratio of positive:negative outcomes in your final data set, as you will ideally like to avoid imbalanced data. However, there are techniques and approaches to handle imbalanced data if necessary.


How to apply Stacking cross validation for time-series data?

TL;DR

Time-series algorithms assume that data points are ordered.
Traditional K-Fold cannot be used for time series because it doesn't take into account the order in which data points appear.
One approach to validate time series algorithms is with Time Based Splitting.

K-Fold vs Time Based Splitting

The two graphs below show the difference between K-Fold and Time Based Splitting. From them, the following characteristics can be observed.

K-Fold always the all data points.

Time Base Splitting uses a fraction of all data points.

K-Fold lets the test set be any data point.

Time Base Splitting only allows the test set to have higher indexes than the training set.

K-Fold will use the first data point for testing and the last data point for training.

Time Base Splitting will never use the first data point for testing and never use the last data point for training.

Scikit-learn implementation

Scikit-learn has an implementation of this algorithm called TimeSeriesSplit.

Look at their documentation, you find the following example:

from sklearn.model_selection import TimeSeriesSplit
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=5)

for train_index, test_index in tscv.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

>> TRAIN: [0] TEST: [1]
>> TRAIN: [0 1] TEST: [2]
>> TRAIN: [0 1 2] TEST: [3]
>> TRAIN: [0 1 2 3] TEST: [4]
>> TRAIN: [0 1 2 3 4] TEST: [5]


Is there a standardized way to do data analysis?

If you only consider the process of making something from your data (excluding data extraction etc), I'd say something like :

Visualising your data to understand it
Cleaning your data, removing useless features etc
Feature engineering : keeping only relevant features, creating new features from existing ones (combining a few one, making ratio of 2 features for example)
Preprocesses (including simplifying your data if it's too precise, like too many classes in a category variable; encoding your variables so it can be taken by a model)
Testing and chosing the most adapted model
Tuning your model (using cross validation to have the best view of what your model does)
Try your model in brand new data (validation set) so you have the best estimation of how good it works, gather your results

This is pretty hard to give 'precise steps' since it differs from one project to another, depending on constraints, type of data, ...


standardize dataset with numerical and dummy features

1) Dummy variables doesn't need to be standardized, just numerical ones, but if you use MaxMin scaler you can pass both numerical and dummy variables because this scaler doesn't change the values of dummy variables, try to apply the equation to dummy variables to see.

$$ z_i=\frac{x_i-\min(x)}{\max(x)-\min(x)} $$

2) That's a trick question and the response is: it depends. If you have some outlier in your data MaxMin doesn't fit well, because this outlier will be replaced by 0 or 1 and the rest of data will be confined in a very restricted range of values, in this case you should use a normal scaler.


Coefficients of Linear regression for minimizing MSE

For a linear regression we have the loss function

$$J(a,b)=\sum_{n=1}^N(y_n-a-bx_n)^2.$$

The partial derivatives are

$$\dfrac{\partial J}{\partial a}=2\sum_{n=1}^N(y_n-a-bx_n)(-1)$$
$$\dfrac{\partial J}{\partial b}=2\sum_{n=1}^N(y_n-a-bx_n)(-x_n).$$

If we set both derivatives to zero and divide by the sample size $N$ we obtain

$$0=\overline{y}-a-b\overline{x}$$
$$0=\overline{xy}-a\overline{x}-b\overline{x^2}.$$

Now, solve the first equation for $a= \overline{y}-b\overline{x}$ and plug this into the second equation

$$0=\overline{xy}-\overline{x}\overline{y}+b\overline{x}^2-b\overline{x^2}$$

and solve for $b$ to obtain

$$b=\dfrac{\overline{xy}-\overline{x}\overline{y}}{\overline{x^2}-\overline{x}^2}.$$

The espression in the numerator is the covariance for a sample and the expression in the denominator is the variance of $x$.


How to use Scikit-learn's affinity propagation clustering with my own datasets?

Clustering algorithms usually assume that you have positions of objects, and want to find dense groups of observations.

If I understand you correctly, you have a 2d grid of sensor readings, and you want to segment them into regions. That is a slightly different problem. If you'd just put your sensor readings into a clustering, then the clusters will not be spatially coherent: clustering assumes there is no particular order to the points.

So you'll need to look into segmentation.

A naive way would be to use (sensor.x, sensor.y, sensor.value) tuples. Including the sensor positions will cause the results to be somewhat spatially coherent. But that makes it very sensitive to scaling, and there is no "correct" way of scaling this. There is a trade-off between spatial coherence and measurement coherence.


Training multiple keras models and combining outputs to determine losses

In keras a model can have multiple outputs, for example:

model = Model(inputs=[a1, a2], outputs=[b1, b2, b3])

Aand if you have multiple outputs you can have multiple losses, for example:

model.compile(loss=[loss1, loss2, loss3], optimizer='sgd')

By doing this, when you train and backpropagate your model you are using all the losses to train your model since you are minimizing all the losses. In this way you can couple your two models as you suggested.

Additionally compile has the parameter loss_weights in which you can change the weights of the different losses.


Here are some things to try.

Plot a bar graph. The bar graph will clearly show jobless people are often choosing NO. Try an 1-way ANOVA test. If the p < delta (i.e. delta=0.05), try a post-hoc test (i.e. Tukey's HSD) to do a pairwise comparison.
Like I said earlier, try a multiple comparison test (1-way ANOVA) first, if there is a statistically significant difference, you can try a pairwise comparison test (post-hoc test).
Maybe try a clustering algorithm? Be careful, because the marginal sums (by rows or columns) are not equal. Maybe create a similarity matrix by profession? To me, it seems that Employees and Businessmen are in one group (very similar), while Workers and Jobless are each in their own group. If you turn those frequencies into proportions, then you might just have 2 groups; one for employees + workers + businessmen, and one for jobless.
Use contingency table analysis to see if the responses (yes/no/don't know) are associated with profession.


The short answer is:
You are correct, in order to compute node embeddings you only need the "true" train edges. These edges span a training graph which is the input we give NE methods to learn the node embeddings. Non-edges are only used in the binary classification problem.

The longer answer is:
The most effective way to compute link predictions from embeddings is via binary classification. The node embeddings first need to be transformed into edge embeddings (e.g. averaging the embeddings of i and j to get the embedding of e=(i,j)) and then solve a binary classification problem. The binary classifier (usually Logistic regression) will take the edge embeddings as examples and try to learn a mapping to their corresponding "label" where these labels are 1 for the "true" edges and 0 for the non-edges. Therefore, in addition to the edge embeddings of "true" graph edges, we need to compute the edge embeddings of a set of non-edges in the graphs (those with label 0). This is what we usually refer to when we talk about negative sampling for LP.

I believe the confusion here comes from the way some popular NE algorithms compute the embeddings. For instance, methods base on matrix factorization directly take the train graph you give as input, compute the adjacency matrix, factorize it and return the node embeddings. Other methods e.g. Deepwalk, Node2vec or LINE have a special way of learning embeddings (Skip-gram from work embedding literature) and require "negative sampling" in the embedding learning process itself. These methods will select, from that train graph you give them, a set of non-edges or paths of non-connected edges to learn which node vectors should be put close together and which should be far. The NE methods will select these samples of non-edges themselves and in different ways, so the user generally doesn't have any control over it.

I hope this clarifies your question.


Is it necessary to normalise data for XGBoost?

Your rationale is indeed correct: decision trees do not require normalization of their inputs; and since XGBoost is essentially an ensemble algorithm comprised of decision trees, it does not require normalization for the inputs either.
For corroboration, see also the thread Is Normalization necessary? at the XGBoost Github repo, where the answer by the lead XGBoost developer is a clear:

no you do not have to normalize the features


When to use which multiple testing correction?

The Bonferroni and Holm methods lead to the same FWER and disjunctive power when analysing multiple primary outcomes. This is because both methods adjust the smallest p-value in the same way. Similarly, the Hochberg and Hommel methods lead to same FWER and disjunctive power when two primary outcomes are analysed and differences between these methods arise when analysing three or more outcomes
You can read about in this article Link


comparison of t-SNE and PCA and truncate SVD

PCA and truncate SVD do not differ much, since they are based on
the same theory that the eigenvectors with the less eigenvalue are
discarded. As mentioned here the difference:

TruncatedSVD is very similar to PCA, but differs in that it works on
sample matrices  directly instead of their covariance matrices. When
the columnwise (per-feature) means of  are subtracted from the feature
values, truncated SVD on the resulting matrix is equivalent to PCA. In
practical terms, this means that the TruncatedSVD transformer accepts
scipy.sparse matrices without the need to densify them, as densifying
may fill up memory even for medium-sized document collections.

T-SNE attempts to find the underlying structure of the data by taking
into account the neighbors of a sample. By prioritizing the neighbor
points gives an advantage when data structure is a manifold.
A way to understand the meaning of your figure clusters is described here,
using radar charts. From my point of view, what i would say is that

The information, which predicts the labels Fraud/not Fraud, is not
on    a structure manifold
The samples seems to be linear separated and and that the main information is given by only two features. If you wanted, you could check how much of the eigenvalues energy is retained when keeping only two features


Why autoencoders work well for outlier detection?

It helps to first understand why outlier detection is generally a difficult problem and why other methods struggle with it.
By their very nature outliers are rare and most data we have is heavily imbalanced. Quite likely you might have not enough "positive cases" / outliers to train a model at all.
Autoencoders solve this problem because they do not try to identify outliers per se. As you have described they basically learn to down- and upsample input with a high resolution performance. However when an input is very different from usual inputs the upsampling might result in more errors than usual which then helps us in identifying outliers.
The reconstruction errors are higher because the autoencoder has been trained mostly / almost exclusively on non-outlier data so when they encounter an outlier they cannot deal with it as well.
Imagine an autoencoder trained to downsample pictures of an orange and then upsample it. If we feed the picture of an apple into this autoencoder it will not produce a very accurate result and that helps us identify that the input is actually an outlier.


python - What is the format of the WAV file for a Text to Speech Neural Network?

Speech data is made up of unique acoustic units called phonemes. Any audio file can be represented as a sequence of phonemes. Both automatic speech recognition (ASR) and speech synthesis (SS) systems model these phonemes. In ASR, speech signal (wav file) is used as input and phoneme labels are predicted and in SS, phoneme labels can be input and speech signal is output.

You can use a phonetic dictionary for converting your text files into sequence of phonemes. e.g play -> P L EY

If you have phoneme boundary marked data e.g. in audio file file1.wav 0.1s to 0.5s phoneme x and 0.5s to 0.9s is phoneme y. Now you have You can use a NN to learn the mapping between phoneme labels and speech signal (400 data points as output and phoneme label of these 400 points as input).

But there are many things that affect the pronunciation. Some of them are listed below:

Context: 'to' and 'go' have the same phoneme 'o' but have very different pronunciations.

Pitch: Female speakers usually have higher pitch than male speakers.

Speaking rate: Speaking rate varies across speakers. It also depends on speaking mode while reading a text we tend to have less number of pauses as compared to conversations.

length_of_output_phoneme: The length of wav file to generate

So in the end input to your NN will look something like this
[left_context, phoneme, right_context, specking_rate, pitch, length_of_output_phoneme] and output will be corresponding speech signal. You can either use MFCC features or raw wav data as NN output. There are many other factors that affect the pronunciation.

If you don't have time marked data. You can use Hidden Markov model HMM for speech synthesis. A separate model will be learned for each phoneme. Input for HMM will be text files (sequence of phonemes) and output will be specch signal. These learned models can be used for generating speech data later.

Some speech synthesis resources are listed below:

1) CMU festvox

2) wavenet

3) Deep Learning in Speech Synthesis

The biggest challenge will be to make it sound like human voice.

Edit 1: Some of the problems with "whenever you see this alphabet, pronounce it like this" are listed below

1) context: From the above to and go example which pronunciation of 'o' will be used

2) discontinuity: Vocal tract vibrations (lip, tongue motions) produce phoneme. Phonemes diffuse into neighboring phonemes since vocal tract doesn't stop vibrating immediately. If you copy-paste phonemes then there will be an abrupt change. You can copy-paste sounds for words since they are independent (kind of, 'can not' becomes 'can't' while speaking) and insert small pauses in between words. But then you have to store pronunciations for all the words.

3) stress phoneme: While speaking one or more phonemes are stressed (more focused, longer), Copy-pasting leaves out the stress information since everything is same

If you can store pronunciations of all the phonemes in all the possible contexts and can smooth the transition between adjacent phonemes then you can copy-paste sounds.
With a generative model we try model the human vocal tract system and depending on input context, speaking rate audio data is generated. Go through the first and third resource for detailed explanations.

All these problem arise because we want to make it sound more human. Phoneme transition and diffusion are the major hurdles.

todo: Pick any word and record it in different context, different speaking rate, different speakers and plot waveforms. You'll see each time it has a different waveform. Even if you don't change any condition each time there will be a slight variation in pronunciation of same word. Save pronunciations of few phoneme and copy paste them to generate a word and listen to it.


Custom Named Entity Recognition using deep learning

The CNN and RNN are quite popular for custom NER. I will suggest you try to implement CNN first and then RNN. Also, when I did custom NER, I found out that for my dataset stacking is giving really good results. I used Random Forest, XGBoost and Linear Regression.


Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training?

Yes the ReLU second order derivative is 0. Technically, neither $\frac{dy}{dx}$ nor $\frac{d^2y}{dx^2}$ are defined at $x=0$, but we ignore that - in practice an exact $x=0$ is rare and not especially meaningful, so this is not a problem. Newton's method does not work on the ReLU transfer function because it has no stationary points. It also doesn't work meaningfully on most other common transfer functions though - they cannot be minimised or maximised for finite inputs.

When you combine multiple ReLU functions with layers of matrix multiplications in a structure such as a neural network, and wish to minimise an objective function, the picture is more complicated. This combination does have stationary points. Even a single ReLU neuron and a mean square error objective will have different enough behaviour such that the second-order derivative of the single weight will vary and is not guaranteed to be 0.

Nonlinearities when multiple layers combine is what creates a more interesting optimisation surface. This also means that it is harder to calculate useful second-order partial derivatives (or Hessian matrix), it is not just a matter of taking second order derivatives of the transfer functions.

The fact that $\frac{d^2y}{dx^2} = 0$ for the transfer function will make some terms zero in the matrix (for the second order effect from same neuron activation), but the majority of terms in the Hessian are of the form $\frac{\partial^2E}{\partial x_i\partial x_j}$ where E is the objective and $x_i$, $x_j$ are different parameters of the neural network. A fully-realised Hessian matrix will have $N^2$ terms where $N$ is number of parameters - with large neural networks having upwards of 1 million parameters, even with a simple calculation process and many terms being 0 (e.g. w.r.t. 2 weights in same layer) this may not be feasible to compute.

There are techniques to estimate effects of second-order derivatives used in some neural network optimisers. RMSProp can be viewed as roughly estimating second-order effects, for example. The "Hessian-free" optimisers more explicitly calculate the impact of this matrix.


I think you need to distinguish between training and execution of the model. During training, you can use batches, which in your case will be different fragments from Shakespeare. So, a batch will be a list of fragments, and the language model will start from the first character on each element of the batch and do the backward and forward pass.

When you execute the model, once it is trained, you would like to see one single example, in which case you can set the batch size to one.

I believe this answers your three questions.


How do I use NLTK Text object with Re library?

You are passing the variable text (where you stored the nltk.Text object) to re.sub (which expects a string str object, not a nltk.Text object).

Contrary to what that class's name may suggest, Text is not a string-like object itself, it simply stores strings. That may be what misled you.

Those strings are probably what you need when calling the re-related functionality below. You can access them normally and in a pythonic via standard enumeration methods, e.g. for-looping over that instance, for word in theText.... See the code below for a full example.

import re
import random
from nltk.text import Text

# Initialize a dummy text with integers as its words.
# With random proability, we add a non-alphanumeric
# character to the word so that we can test the regular
# expression in the original example.
text = Text([
    str(random.randrange(1, 100)) + (
        '' if random.random() >= 0.5 else
        random.choice(['!', ',', '?', ':', ';'])
    )
    for _ in range(20)
])

# The tokens in the `Text` object can be accessed
# via enumeration (the class implements a Python iterator
# underlyingly)
print(text)
print(text.tokens[:10])
print(list(text)[:10])
print([x for x in text][:10])

# To avoid the exception, we comment the original line
# We can't pass the variable `text` to `re.sub` because
# it expects a string and `text` stores an instance of
# NLTK's `Text` class.
#lowerx = re.sub('[^a-zA-Z0-9]','',text)

# Test for the expected behavior:
for token in text:
    lowertoken = re.sub('[^a-zA-Z0-9]', '', token)
    print('<input="%s"  output="%s">' % (token, lowertoken))


Is it acceptable not to transform() test data after train data is being fit_transform()-ed

No, it does not make sense to do this.

You model has learned how to map one input space to another, that is to say it is itself function approximation, and will likely not know what to for the unseen data.

By not performing the same scaling on the test data, you are introducing systematic errors in the model. This was pointed out in the comments by nanoman - see that comment for a simple transformation example.

To exaggerate the point, imagine you have a language model, translating text from English to French. You apply a single transformation on your English data: you first translate it to Spanish. Now the model is trained to translate Spanish to French, with some accuracy. Now you move on to the  test data - still in English - and you do not apply the same transformation as you did to your training data. You are asking the model to translate directly from English to French, instead of Spanish to French, and it is obvious that the results won't be good.

In principal, this idea is the same as with any other model and transformations, just that the impact might not always be so visible i.e. you might get really lucky and not notice a large impact.

The language model might have learnt some elementary linguistics common to all three languages (e.g. overlapping vocabulary or sentence structuring), but we cannot expect the model to perform well, translating English to French.

Practical Note

You should only compute the transformation statistics (e.g. mean and variance for normalisation) only on training data and use these values to then transform the training data itself, and then the same values to transform the test data.

Including the test dataset in the transform computation will allow information to flow from the test data to the train data and therefore to the model that learns from it, thus allowing the model to cheat (introducing a bias).

Also, it is important not to confuse transformations with augmentations. Some "transformations" might be used to synthetically create more training data, but don't have to be used at test time. For example, in computer vision, deleting regions of an image. Test time augmentation is something you could read about.

Extra discussion

More complicated models (ones with many many more parameters) might be able to perform some kind of interpolation, especially if your dataset if N-dimensions with a large N (e.g. > 10).

This has recently been seen with extremely large models, such as Open AI's GPT-3, which has 175 BILLION parameters, and is therefore even able to perform quite well on completely different tasks, let alone the given problem in the training set range.


What is the Space & Time Complexity of Mini-Batch K-Means clustering algorithm?

Infinite.

Mini-batch k-means never converges, you need to use an iteration limit or similar heuristic, and you can never guarantee to have found a local optimum.

In essence, mini-batch k-means is:

draw a random sample
perform one iteration of k-means using this sample
repeat

Assuming that your sample size is N, 2 takes O(k N m t) time.


Calculate probability vector from sample data

Probability vectors are meant for discrete random variables. Your data is drawn from a continuous distribution. In order to get a probability vector from data, you need to bin it into a normalized histogram. This gets you a probability vector. You probably want to make sure though, that bin center and width for samples from both distributions are the same.

I guess you could use the pdf to calculate the probability vectors too, if you knew the pdfs of the distributions. But if you only have access to data, i.e. samples from the distributions, this is not the case.


Why is IoU said to be non-differentiable?

Intersection over Union (IoU) can be differentiable. The paper "UnitBox: An Advanced Object Detection Network" by Yu et al. describes it detail.


Does Tensorflow uses vectorization in its operators

Yes, tensorflow works with tensors, which are like vectors but more generic, so dont worry about loops. Just make sure the matrices you are multiplying have compatible dimension. Its completely tensor arithmetic, which for your case will be vector algebra.


Different results every time I train a reinforcement learning agent

A part of the agent consists of taking random actions. So there is a % chance that the agent will take a random action instead of an action based on the training. This is called "exploration". This page describes this as "The amount of randomness in action selection depends on both initial conditions and the training procedure. Over the course of training, the policy typically becomes progressively less random, as the update rule encourages it to exploit rewards that it has already found. "
This is normal. The agent's network is initialized with random weights, and part of the actions it takes during the training are also random (see above). So different training runs will produce different results. If you want to circumvent this issue, you could use a fixed seed for the random-number generator.


There are a lot of different possibilities. It is a bit hard to get the message through here in the forum. But I try and focus on one thing that is relatively easy to implement:

One of the easiest things you can do is to do a linear regression. You dependent variable (y) would be „fire trucks working“ (or the share of working ones). You can estimate this dependent on time. So you can (for example) generate indicator variables (=1 if something is true vs =0 if not) which indicate things like weekday, hour of the day. Use this indicators as explanatory (x) variables in the regression.

The result will be, that each indicator shows you the average truck use at some point in time. Say indicator A is „monday“, B is „16:00“, you will see how many trucks are on average dispatched in this time window. You can also easily see if the difference is significant or not.

In the past such models were used to estimate the number of callers in callcenters. The thing is called „dummy variable regression“ (at least in some statistic fields). https://en.m.wikipedia.org/wiki/Dummy_variable_(statistics)

Note that you observe effects from each „dummy“ variable in contrast to a baseline. Imagine you have truck use (y) and only 1 indicator/dummy, which is „monday“. Say the intercept of the model (the result with all dummies =0) is 5, meaning on average 5 trucks are employed. If the effect/coefficient of your dummy is 2, it means there are 2 more trucks employed (on average) on mondays. So each dummy/indicator (viz in case it is 1) shows you an effect in contrast to the base category (where all the dummies are 0).

Here is some script in Python which I guess refers to a similar problem, maybe it helps for orientation: https://stackoverflow.com/questions/50733014/linear-regression-with-dummy-categorical-variables

Also this may help: https://songhuiming.github.io/pages/2017/01/21/linear-regression-in-python-chapter-3-regression-with-categorical-predictors/

Another note: In principle I would try to get many different dummies/indicators from the timestamp, e.g weekday, hour of day, special days or seasons (Christmas, winter etc). You need to try combinations which work for you and the model. But don‘t plug too many indicators into the model. Keep things easy in a first step and try to improve the model later.

In order to check the „quality“ of your model you can look at R2 (a measure that shows how well you are able to explain the data which is between 0,1). You can also randomly (!) set aside some 10-20% of the data (do not use them to estimate the model in the first place) to check how well you model can explain real truck use at some point in time.

I would use R for such a task (I find it more intuitive than Python). Here is some example code for R: https://stats.idre.ucla.edu/r/modules/coding-for-categorical-variables-in-regression-models/

Hope this helps: Good luck with your interesting project!


Model selection: large mean and variance vs small mean and variance

This is exactly a question I asked Sebastian Raschka (author of the great Python Machine Learning book), here you can find his answer saying "I also recommend the 1-standard error method, which basically means that you select the best model from k-fold based on pure performance, and then you select the simplest model that is within 1 standard error of that model".
The more extended explanation of his answer can be found on his github link.


How to present the final model (e.g. random forest)?

The probability threshold can be changed and you would obtain different results for accuracy / recall / precision / F1 score (the objective). You could see this as an aspect of the parameters you have tuned to obtain the best objective.

Similarly for CV, the out-of-training fold objective would be reported together with the other parameters, such as number of trees and predictor subset size. One reason I could think of regarding the instability of threshold across CV folds is that the CV folds were not split up using stratified sampling, resulting in different proportions of positives observed across the folds.


Reinforcement Learning uses a simple logic of learning in which the network tries to learn from the feedback it obtains. This tries to optimise the overall reward in the long run instead of the current reward.

This is one of the best platform to read about it. It also contains some useful links.

As stated by the wiki,
The basic reinforcement learning model consists of:

a set of environment states S;
a set of actions A;
rules of transitioning between states;
rules that determine the scalar immediate reward of a transition; and
rules that describe what the agent observes.

The rules are often stochastic. The observation typically involves the scalar immediate reward associated with the last transition. In many works, the agent is also assumed to observe the current environmental state, in which case we talk about full observability, whereas in the opposing case we talk about partial observability. Sometimes the set of actions available to the agent is restricted (e.g., you cannot spend more money than what you possess).


Modeling pixel intensity with the normal distribution

The first thing to consider is that, even continuous measurements are actually discrete. For instance, temperature is always considered as continuous but every time you process temperature data, these are discrete data to the precision of the sensor used to measure them. Same is true for every physical, measurable quantity (including intensity). That being said, with our modern physics models, it makes no sense to consider that temperatures follow discrete mass probabilities with several thousands possible output and everyone studies them under continuous assumptions.

So where do we set the boundary? When do we consider that there are "too many" possible outputs to work with probability mass functions and choose to work with continuous distributions? This is an open question! :)

Here is the thing, we do not lose information when processing discrete data in a continuous way so there is no restriction for doing so!

Now consider the pixel intensities of an image, most of the time, we would need to consider discrete mass probabilities with 256 possible outputs, meaning 256 parameters (or weights) to be estimated or tuned. If we model the intensity by a continuous distribution, the number of parameters is drastically reduced! For a Gaussian for instance: 2 parameters. Even using more complex distributions or mixture models, we will not need to tune hundreds of parameters. This is the main advantage! Compact representation!

About the choice of the Gaussian specifically, let say that it has been empirically proven to work great. The Central Limit Theorem also supports this common choice. And now, on the top of this, many algorithms are readily available under the assumption that the data follows Gaussian distributions making them even more used by people in the field. Doesn't mean that nothing else could be used but this is the reality of things.


What could be some Classification techniques to classify a tree of webpages given the category of each webpage

There are of course a number of ways this can be done, such as majority voting or some other rule-based algorithm, however it can also be done through supervised learning since you have some labels for the trees.

I would make the input space of my model the normalized frequency of the categories for a tree. This means you will need a dictionary of possible categories for the nodes, usually obtained from your training set. Then you can tabulate the frequency of instances.

For example if we have a website with the following node classes:

News: 5
Opinions: 9
About: 1

Then we can formulate our input vector as $[0.33, 0.6, 0.067]$.

You can then train this model using your already labeled trees. The model will then be capable of classifying future trees in this same way.

To determine the top $K$ classes for a tree you will need a model which can do this (most can). If you use K-NN (different K) then you can pick the $K$ closest neighbourhoods. With Random Forests or Naive Bayes you can pick the $K$ classes with the highest probabilities.

To consider the fact that you have a list of $K$ categories for each node you can add a weighting when calculating the normalized frequencies. For example let's say we have 3 classifications and the following webpages (nodes).

Page 1: News, Opinion, Commentary
Page 2: News, Advertisement, Opinion
Page 3: Commentary, News, Adverisement
Page 4: News, Opinion, Advertisement

Then the input vector can be calculated by awarding 3 points to the first category, 2 to the next and 1 to the last. This results in $[0.49, 0.21,0.17, 0.17]$. Alternatively, if you have a probability for these classifications you can use that as the weighting factor.


term for data that compares a day to same day of a week/month/year ago

A repeated cross-sectional study. Cross-sectional refers to data collected at one specific point in time or over a short period. Repeated since you compare at least two time frames. If there are several time frames, one could talk about time-series cross-sectional (TSCS) data. Repeated cross-sectional studies may become pseudolongitudinal, or longitudinal if carried over a longer lapse of time.


Accuracy on Validation and Test set, Overfit?

If the training set, validation set and the unseen test set (as you put it) have the same score, but lower than you expected then the model has not overfitted.
An overfitted model would have higher scores for the training data at least, and depending on how you optimised the hyper parameters on the validation data but lower for the unseen test set.
The more likely outcome is that your model has underfitted i.e. low but consistent scores across all 3 sets of data.


which machine learning technique can be used?

I think these are the methods that you can try out (Please feel free to add more to this list):

Highly precise with a little low recall is to use a dictionary with almost all possibilities (manual effort, but must be worth it.).
Using Word2Vec. Mikolov has already trained text data and created word vectors. Using this vector space, you can figure out which words are similar. You can try out and find a threshold above which you can say which words are similar (for example, yoga and exercise would have decent similarity.)
Train custom W2V, if you have enough data(This is an unsupervised model, so you don't need to worry about tagging the data but finding huge amounts of data relevant to the working domain.)
You can use an RNN to find the most similar words in a corpus and use it for queries. This gives a bit more flexibility than W2V.


Why we can't Remove features with missing values in Data Preprocessing

Missing values doesn't necessarily mean missing information. Sometime missing value represent an information in itself. For example: we have a data set which have features such as pool area, no. Of rooms and area. Now pool area have 90% of its value missing. You can create a new column called is_pool, which tells if the house has pool or not, from pool area column by using condition that if pool area is missing, make is_pool =0 otherwise 1.
This is one basic example, in my experience most difficult thing while doing EDA is identifying is missing value is really means no information or is it represents something else entirely. In short, understand why the value is missing.


Your approach is not correct.

Some hints that should help you to understand your mistakes

the sum over all elements of the confusion matrix should be 1000 in your case.
TPR=TP / Num_positives

Good guide to confusion matrix terminology


Use a GPU to speed up neural net training in R

If you're able to convert the code into python, then you could use the Google colab environment or Kaggle kernels. These online platforms provide free GPU's that you can utilize.
Kaggle kernels also support R directly.


Some algorithms, such as SVM or Logistical regression, have possibility to add a weight to certain class, therefore fix the unbalanced issue.

This really sounds like a job for Survival analysis, which is especially designed to answer questions like "When machine X fail" or "Which attribute influence the most the failure". You can simply start by plotting the Kaplan-Meier curve and then further stratify it by some attribute. Then you can try Cox regression model - it is useful to see the influence of an attribute on survival - the hazard ratio.
But don't forget to verify the assumptions (functional form and proportional hazard).

In R the Survival analysis is implemented very well, so don't be affraid.
There is simple and short tutorial which might help.


Speed Date or Dating Conversation Dataset / Corpora

I have a feeling that you are never going to come across this, most of this data is closely guarded. It's no secret that dating sites like OK Cupid use data science on their app, so I think that most of these sites aren't quick to make their data public.


time series prediction with several independent variables using RNNs and Keras

Please go through this tutorial:

https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/

The author focuses on LSTM-RNNs with explicit instructions on how to re-shape your dataset into the format that Keras requires, which can be VERY tricky if not properly understood.


Considering ocam's razor I would recommend to use the simplest model first and increase the complexity if the simple models fail:

Exponentially Weighted Moving Average, allows only for auto correlation with lag one
ARIMAX, allows for several lagged autocorrelation, seasonal adjustments and external regressor.
Fourier transformation, allows to fit more shapes time series but they are often more complex to explain to users

All these models can predict one step ahead and you can repeat this up to 60. Of course the confidence intervals become much wider these many steps ahead. Base R has many time series available.

These two models work well for many time series data. I would only start to use neural nets when these are not working or if you are looking for say edge detection.

Neural nets

If neural nets are the only option to go you can check this post or if you have matlab available this post.

I recently found this very interesting article on medium that explains how to fit a Neural Net to time series data. A more detailed implementation guide can be found here. I have not tried either approaches but I thought that the may be useful for future reference.

Time series modus operandi

Note that in any case, before staring off modeling your time series it is highly recommended to take some preparatory steps:

Make your series stationary as explained here
Understand what elements you can decompose in your series as explained here

Hope this helps


How to do data augmentation for Machine Learning on statistical data?

You can do mainly two things: bootstrap or oversampling.
With statistical data you can do bootstrapping (random sampling with replacement)
Bagging methods help boosting you model accuracy. The pseudocode will be a bit like this.
for estimator in range(number of estimators):
    Sampling some data
    Fitting a model
    Predicting
mean(predictions)

This way you are able to train different models with the same sample of data and helps in boosting accuracy. Well known as Random Forest use this technique.
There is other way to create synthetic data and they are the oversampling/undersampling techniques that you can have a look in the imblearn documentation. Techniques like SMOTE (Synthetic Minority Over-sampling Technique) are sometimes used, but they generate synthetic data.
With statistical data you can not rotate, add brightness...


Activation function vs If else statement

Activation functions in general cannot be implemented with if-else statements.

ReLU is a particularly simple activation function, and can thus easily be implemented with an if-else block. For example (in Python):

def relu(x):
    return x if x > 0 else 0

But this only works because ReLU is linear for $ x \geq 0$.

What about sigmoidal activation functions?

There's no if/else implementation for this function that's analogous to that of ReLU.  In fact, this is a one-liner with no if/else involved:

def approximate_logistic(x):
    return 1 / (1 + 2.71828**-x)

Same goes for lots of other activation functions like Softplus, CELU, Tanh.

If/else blocks simply aren't powerful enough to capture the wide variety of activation functions that we might want to use.


Feature Selection, Machine learning and Time Series analysis, for large financial timeseries

Your model should be based on what your data represents.  If you want meaningful results, you should get some idea of what your data is representing before trying to answer this question.

For instance, you could use your ind# columns to predict the values of your ind#${_j}$ rows, then use the ind#${_j}$s to predict the val${_j}$.  But this would only make sense if there is a predictive relationship between the ind#${_j}$s.  And you would definite want to have some idea of what that relationship might be in order to choose an appropriate algorithm for each particular column.

Furthermore, the same would apply for $j_{th}$ row val${_j}$ prediction.  At the very least you would want to  make an educated guess about the relationship between your ind#${_j}$s and your val${_j}$ column so that an appropriate algorithm could be chosen.

As another example, you may be able to use the previous row as a predictor for the next value in the time series. i.e. ind#${_{j-1}}$s  => val${_j}$.  But again, this only makes sense if you are able to make reasonable assumptions about the relationship of the data, because you ultimately need to choose the algorithm you use to make the predictions.


How can I use mean normalization. Should I use it for numerical columns or categorical columns as well?

Normalization and standardization are transformations that can only be applied on metric variables. You cannot normalize categorical variables.

Either you label-encode your data or you one-hot-encode it, depending on the algorithm you use. That's all you do with it.


Dealing with foreign film titles on IMDb

"Meghe Dhaka Tara" is a noun phrase. It makes sense for an API looking for movies to not translate the phrase as that is not its purpose. I suspect, that at most it will have some basic spell checks in place (for instance "did you mean `Megh Dhaka Tara'")

If "The Cloud-Capped Star" is indeed an english dub for the movie, then it suggests that the service provider has not linked movies to its translations.


Can I install Tensorflow in Anaconda without using Keras?

Absolutely.

  If I can what is the difference between using Keras with Tensorflow
  and only Tensorflow?

While Tensorflow  is a super powerful numerical computation and optimization library with lots of features for building neural networks, YET it is a bit tedious and nontrivial especially for beginners to use.

Here Keras comes very handy. In short, Keras is Tensorflow abstraction. It allows to quickly and easily define a neural net even complex ones with a few lines of codes. Also note that Keras not only runs on top of TensorFlow but also on top of Microsoft Cognitive Toolkit, Theano, or MXNet. Look at at this post where the blogger compares defining a typical neural network via directly in Tensorflow and Keras. You can see that with Tensorflow it is minimum 17 lines of code, whereas Keras reduce it to 10 lines of code.


Faced problem while applying OneHotEncoder

Consider the following demo:

Source DF:

In [53]: df
Out[53]:
   A          B                   C
0  1  Bachelors  Married-civ-spouse
1  2    HS-grad            Divorsed
2  3       11th  Married-civ-spouse

Option 1:

In [54]: for c in df.columns[df.dtypes.eq('object')]:
    ...:     df = df.join(df[c].str.get_dummies())
    ...:

Result:

In [55]: df
Out[55]:
   A          B                   C  11th  Bachelors  HS-grad  Divorsed  Married-civ-spouse
0  1  Bachelors  Married-civ-spouse     0          1        0         0                   1
1  2    HS-grad            Divorsed     0          0        1         1                   0
2  3       11th  Married-civ-spouse     1          0        0         0                   1

Option 2:

In [58]: df = df.join(pd.get_dummies(df.select_dtypes(['object'])))

In [59]: df
Out[59]:
   A          B                   C  B_11th  B_Bachelors  B_HS-grad  C_Divorsed  C_Married-civ-spouse
0  1  Bachelors  Married-civ-spouse       0            1          0           0                     1
1  2    HS-grad            Divorsed       0            0          1           1                     0
2  3       11th  Married-civ-spouse       1            0          0           0                     1


How should the bias be initialized and regularized?

From the Stanford CS231N Notes:

Initializing the biases. It is possible and common to initialize the
biases to be zero, since the asymmetry breaking is provided by the
small random numbers in the weights. For ReLU non-linearities, some
people like to use small constant value such as 0.01 for all biases
because this ensures that all ReLU units fire in the beginning and
therefore obtain and propagate some gradient. However, it is not clear
if this provides a consistent improvement (in fact some results seem
to indicate that this performs worse) and it is more common to simply
use 0 bias initialization.

In LSTMs it's common to initialize the biases to 1 - see for example.


You can multiply with (X==)10/100/1000/... to size of precision you want. After performing multiplication to both numbers, Do your operation(-, /, ...) You will get the result.


Preferred approaches for imbalanced data

I think it mostly depends on your dataset type! are you dealing with text? or image? or... and your features will tell which option is the best fit for your case....but according to my experience in most of the cases, option 1 and 2 besides they depend on your dataset and power of your features they need to be judge based on your model high bias or variance and they should inform you they are good or no! you need to do some experiment to figure out them or know your dataset well to find out adding or reducing dataset will affect your model performance or not!

and what I like to tell is try to use upsampling and downsampling methods same time to make your dataset balanced in a fair way(kinda)!....in this case (87% class 0 and 13% class 1)....upsample class 1 and downsample class 0! how much you need to upsample or how much downsample it is all your choice and definition of fairness in your dataset! and this definition could differ!


How do I resolve this error: "expected string or bytes-like object" please?

You are applying that function to a column in a panda's dataframe. Double check the datatype of the column:
lemmatised_df['label'].dtype


Try using predict_proba instead of predict as below. It should give you the same number.

roc_auc_score(Y, clf_best_xgb.predict_proba(X)[:,1])

When we compute AUC, most of time people will use the probability instead of the actual classs.


Conditional on you having data, yes, you can. Check out Generative Adversarial Networks and/or Reinforcement Learning for text generation. This paper is a good starting point: https://openreview.net/forum?id=rJedV3R5tm.
Also, here's a tool that might help you. What you can do is generat these institution names without differentiating by type, and then build another model to classify them.


Can i use deep learning in my agriculture PHD?

Please note that the following is my personal opinion; I still hope that you find it useful.

If you cannot properly explain a concept on paper, then it should not be part of your PhD thesis. It is possible to implement Deep Learning algorithms without any conceptual knowledge. While this may be (somewhat) acceptable in the industry, it will get you into trouble in academia.

I have no doubt that your algorithms are well-programmed and solve many issues that are challenging from a programmer's perspective. But that is not what the reviewers of your thesis are interested in. They are also not interested in the prediction accuracy of your algorithm.
As scientists, they want to know why something happens; and to my knowledge, Deep Learning algorithms provide next to nothing on this question.

And this is where 'middle age' linear regression has its strengths (by the way: a quick way to make an enemy is to tell him that the method he has been using all his life is too old to be useful). Yes, linear regression is unexciting, and it has many shortcomings. But it is extremely well understood. Most importantly, there are tons of books and papers about the connection between assumptions and valid interpretations: we know under which assumptions and in which sense the linear regression estimator is optimal; we know which assumptions lead to which kind of confidence intervals for the estimator; we know about robustness with respect to model misspecification; etc.

As long as you cannot address these issues convincingly and on paper (That is: In what sense is my result optimal? How confident can I be about my result? How robust is my model?), you should not use Deep Learning in research as an alternative to classical statistical methods.


I know this doesn't answer you 100%, but maybe it helps. Feel free to disregard.

I just implemented this in keras and didn't have any issues

model = Sequential([
    Dense(1,activation='linear',input_shape=(1,))
])
sgd = optimizers.Adam(lr=0.001)
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['mae'])
model.fit(test_df.x, test_df.y, validation_split = .2,epochs=500, batch_size=1)

plt.scatter(test_df.x,test_df.y,s=30,c='r',marker='x')
x = np.linspace(4,25,100)
y = model.get_weights()[0][0][0]*x+model.get_weights()[1][0]
plt.plot(x, y)
plt.ylim([-5,50])
plt.xlim([4.5,23])

Is it possible you read your data in incorrectly maybe? Or maybe you are referencing it incorrectly when you set your Xt and yt?


How to create a language translator from scratch?

Google translate itself uses Deep learning to translate sentences which can be seen here.

You can translate sentences across languages for which you need two things :

A large dataset which has pairs of translations ( like English-French ). You can find such a dataset from here.
A sequence-to-sequence RNN model. They have Encoder-Decoder architecture which encodes the source sentence into a thought vector and then decode it to form the translation. This image may be helpful.

The picture shows the Neural Machine Translation adopted by Google.

Also, TensorFlow and Keras help in creating such deep learning models. You can refer to this blog from the author of Keras. These models ( seq2seq ) have gained popularity in text summarization, NMT, and other sequence-to-sequence tasks.

If you wish to create a full-scale production ready language translator then you will need a GPU as these models are heavy to train as well as a very large corpus of text.


Here is some code that deals with the second part of your question regarding matching names. (BTW including sample data always helps)

require(dplyr)  ## for join

# create two example dataframes
d1 <- data.frame(player_first_last=c("Joe Smith", "Jack Johnson", "A Jones", "Slow Mo"), d1_value=c(1,2,3,4))
d2 <- data.frame(player_last_first2=c("SmithJo","JohnsJa", "MoSl"), d2_value=c(11,12,13))

# first two letters of first name using substr, and trimming whitespace
d1$player_first2 <- trimws(substr(d1$player_first, 1, 2 ))

# last name - regex capturing everything after the first whitespace
# this is a "regular expressions"
d1$player_last <- gsub(".+\\s(.+)", "\\1", d1$player_first_last)

# concatenate last 5 and first 2 using paste0
d1$player_last_first2 <- paste0(substr(d1$player_last,1,5), d1$player_first2)

# join on constructed name -- full_join in case there are differences
d3 <- full_join(d1, d2, by="player_last_first2")

Results in  d3

  player_first_last d1_value player_first2 player_last player_last_first2 d2_value

1         Joe Smith        1            Jo       Smith            SmithJo        1
2      Jack Johnson        2            Ja     Johnson            JohnsJa        2
3           A Jones        3             A       Jones             JonesA       NA
4           Slow Mo        4            Sl          Mo               MoSl        3


Find differences of two protein sequence files

You can do this very easily using python sets as follows:
with open("file_a.txt", "r") as f:
    data_a = f.read().splitlines()

with open("file_b.txt", "r") as f:
    data_b = f.read().splitlines()

set(data_a) - set(data_b)

# {'----------------------MDVFKKGFSIAREGVVGAVEKTKQGVTEAAEKTKEGVMY',
#  '>Q63544|72',
#  '>Q63544|9',
#  'TKTKGERGTSVTSVAEKTKEQANAVSEAVVSSVNTVATKTVEEAENIVVTTGVVRKEDLEP'}


Use of Correlation Map in Machine Learning

It depends. A high correlation between two features suggests that they represents almost the same information. For some problems like clustering, it is always useful to remove redundant features while some algorithm like Gradient Boosting in xgboost is not affected at all by such features. So, it all depends on what you want to do with your data set.

As per my opinion, if your dataset has too many features, then I would suggest to check the correlation between those features and apply PCA to reduce the dimensionality of your dataset especially if you are doing tasks like clustering or regression.


How should I read the following heatmap?

Since the variables isPWKinText and H1_2_Len have the same value for all examples in your dataset they have zero variance and contain no information. There is simply just no inference you can make based on them. That is why they are not relevant and shown blank.

Please note, that this depends on your dataset, of course. The two variable might just be constant for the examples you are currently looking at. For example, if this is an analysis of your training data maybe your datasplit is just really bad and the validation or test set contains examples for which these variable do show other values. That is something I suggest to double check.


Find the optimal n_estimator by looping the model accuracy indicator in random forest algorithm - python

The number of trees in a random forest doesn't really need to be tuned, at least not in the same way as other hyperparameters.  Adding more trees just stabilizes the results (you're averaging more samples from a distribution of trees); you want enough trees to get stable results, and adding more won't hurt except for computational resources.

More directly (question 1), you could instead train the RF with, say, 1000 n_estimators, then just grab the predictions from each tree and average only the first 100 of them, the first 200, etc.  Since the trees are built independently, this is essentially the same as training repeatedly on 100, then 200, etc. trees from scratch (but of course faster).

More generally (question 2), yes, this is one-dimensional grid search.  Hyperparameters may be interdependent though, so doing independent 1D searches will probably be suboptimal.  Look into full grid search, random search, and maybe more advanced hyperparameter optimization methods.


How to determine if problem is solvable by neural network?

Neural network will pick up on any patterns as long as your environment is not fully stochastic.

Here is what authors of Using Recurrent Neural Networks to Frecasting of Forex used to determine the amount of stochasticity:

page 7, bottom. "Hurst Exponent".


Going to your first question,

  What does "cleaning" the data consists of? How do you know if your
  data needs cleaning anyway?

Cleaning refers to the various processes to transform the data so that we can utilize it to the fullest. Removing unwanted features, incomplete entries, NaN or null values ( if the dataset is numeric ) consist of "cleaning" the data. This process is important because directly feeding the unclean data to the model may result in its stunted performance or runtime errors.

Once you have a large dataset, you need to transform it according to the problem which you are solving. If you are training a model to classify movie reviews as positive or negative then you can easily remove columns like "user_id", "category" etc. as these do not contribute to the polarity of the review.

  What features of the data determines which algorithm I should use? Or
  is it mostly trial and error?

Well, the algorithm you choose will mostly depend on your problem. Decision trees are good for smaller datasets and Deep Neural Networks ( DNN ) would be used to complex classification and regression problems.

Text classifier systems use embedding layers, TF-IDF vectorization, n-grams model. We basically choose a model on these factors :

Size of the dataset.
The complexity of the problem.
Computational resources ( in some cases ).

We can always play around with the hyperparameters and also modify the model so that it better fulfils our need.

  Are there any additional things that I should be doing other than
  trying different algorithms and testing how well they fit the data?

We choose a model based on the problem. CNNs have been prevalent in image-related problems. Word embeddings are useful in text classification. LSTMs are used in time-series-related problems.

Tip: You can try to implement various algorithms from scratch ( without using scikit-learn or ML frameworks ). This helps you in developing an intuition regarding how the model learns from the data and makes predictions.


Random Forest Stacking Experiment for Imbalanced Data-set Problem

The underlying idea is fine, but you've fallen into a common data leakage trap.  By recombining the data and then resplitting, your second model's test set includes some of the first model's training set.  The first model knows the labels on those datapoints and, especially if overfit, passes along that information in its predictions.  So the score you see for the ensemble is probably optimistically biased.

The most common approach to fixing this is to use k-fold cross-validation to produce out-of-fold predictions on the entire training dataset for the second model.

Note that sklearn now has such stacked ensembles builtin:
https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html


How to make a classification problem into a regression problem?

So, the direct answer here is clearly NO.

The answer comes from the definitions of classification and regression. In a classification task what a model predicts is the probability of an instance to belong to a class (e.g. 'image with clouds' vs 'image without clouds' ), in regression you are trying to predict continuous values (e.g. the level of 'cloudiness' of an image).

Sometimes you can turn a regression problem into a classification one. For example if I have a dataset of images with labels of their cloudiness level from 0 to 5 I could define a threshold, i.e. 2.5 and use it to turn the continuous values into discrete one, and use those discrete values as classes (cloudiness level < 2.5 equal image without clouds) but the opposite is definitely not possible.

Here's also a link to a similar question Could I turn a classification problem into regression problem by encoding the classes?

To solve the problem of imbalanced classes there are many ways, not just oversampling, you can generate artificial data, add class weights in the loss function, use active learning to gather new data, use models that returns uncertainty score for each prediction (like bayesian networks), I'm sure here is plenty of answers and strategy you can try.


The two modules where you can really harness data mining and big data techniques are probably Leads and Opportunities. The reason is that, as you've written yourself, both contain 'potential' information that you can harness (through predictive algorithms) to get more customers. Taking Leads as an example, you can use a variety of machine learning algorithms to assign a probability to each account, based on that account's potential for becoming your customer in the near future. Since you already have an Accounts module which gives you information about your current customers, you can use this information to train your machine learning algorithms. This is all at a very high level but hopefully, you're getting the gist of what I'm saying.


Comparing Dataset - Should I use the same Test dataset?

By testing a model on the same dataset (sharing same characteristics), you will have information on how pertinent you hyperparameters are for this dataset.
Then you can test on another dataset that has other characteristics. It will give you information on how good is a model to generalize.
I would not create a mixed dataset to test the model for interpretation as it could hide some interesting information. For instance, case 1 you predict 100% of the samples from test dataset A and only 50% from test dataset B. Case 2 you predict 75% from both. Interpretation will not be the same. But If you want to compare the two models created from 2 different training datasets, then yes, compare them with a same test dataset, otherwise the test would be biased.


Unable to understand which features to choose

Create a correlation plot and remove all highly correlated columns. Remove columns that are irrelevant to the prediction, such as home/away

If you're interested in feature selection you can use the methods in mlxtend library. It's likely that selecting the right features won't improve your results drastically. There can be other things that can be done to do so.

Here are some suggestions to improve your results:

Make sure you're treating the columns with continuous and categorical columns in appropriate ways - encoding only the categorical columns
Normalize all columns that have continuous variables, perhaps using StandardScaler
Experiment with using feature extraction techniques like PCA
Try using gradient boosting (LightGBM or CatBoost)
There seems to be missing data. Make sure to deal with them correctly. If they are very less you can drop those rows.


Classification Model based on Ordered Features

You could try a recurrent neural network classifier.

Alternatively, and perhaps more easily, augment your dataset to contain images of cards with changed ordering of the card images, and train on the resulting dataset.


Best practice for developing using Spark

What you want to do is develop your code in IntelliJ, and then package your code and dependencies into an executable jar file using SBT or Maven.

When you have your jar stored locally, you can use spark-submit to transfer the jar to your cluster (along with some other parameters) for execution.

You might also want to take a sample of your data and store it locally so you can run spark locally and test/debug your code in IntelliJ. This can speed up development considerably, and having access to a debugger is a huge help.


NLP: What are some popular packages for multi-word tokenization?

The multiword tokenizer 'nltk.tokenize.mwe' basically merges a string already divided into tokens, based on a lexicon, from what I understood from the API documentation.

One thing you can do is tokenize and tag all words with it's associated part-of-speech (PoS) tag, and then define regular expressions based on the PoS-tags to extract interesting key-phrases.

For instance, an example adapted from the NLTK Book Chapter 7 and  this blog post:

def extract_phrases(my_tree, phrase):
   my_phrases = []
   if my_tree.label() == phrase:
      my_phrases.append(my_tree.copy(True))

   for child in my_tree:
       if type(child) is nltk.Tree:
            list_of_phrases = extract_phrases(child, phrase)
            if len(list_of_phrases) > 0:
                my_phrases.extend(list_of_phrases)

    return my_phrases

def main():
    sentences = ["The little yellow dog barked at the cat",
                 "He studies Information Technology"]

    grammar = "NP: {<DT>?<JJ>*<NN>|<NNP>*}"
    cp = nltk.RegexpParser(grammar)

    for x in sentences:
        sentence = pos_tag(tokenize.word_tokenize(x))
        tree = cp.parse(sentence)
        print "\nNoun phrases:"
        list_of_noun_phrases = extract_phrases(tree, 'NP')
        for phrase in list_of_noun_phrases:
            print phrase, "_".join([x[0] for x in phrase.leaves()])

You defined a grammar based on regex over PoS-tags:

grammar = "NP: {<DT>?<JJ>*<NN>|<NNP>*}"
cp = nltk.RegexpParser(grammar)

Then you applied it to a tokenized and tagged sentence, generating a Tree:

sentence = pos_tag(tokenize.word_tokenize(x))
tree = cp.parse(sentence)

Then you use extract_phrases(my_tree, phrase) to recursively parse the Tree and extract sub-trees labeled as NP. The example above would extract the following noun-phrases:

Noun phrases:
(NP The/DT little/JJ yellow/JJ dog/NN) The_little_yellow_dog
(NP the/DT cat/NN) the_cat

Noun phrases:
(NP Information/NNP Technology/NNP) Information_Technology

There is a great blog post by Burton DeWilde about many more ways to extract interesting keyphrases: Intro to Automatic Keyphrase Extraction


How can conclusions be drawn from recommendation systems evaluation?

"Good", I think, is based on the state of the art at the moment.  So I would look at respected models from industry leaders and use their  reported accuracies as a base line for what is "good": since it comes down to what is possible.


Accuracy decrease in production after adding additional input datas

Obviously, without knowing the exact data, everything I will say is mere speculations. Here are some of my guesses why this might happen:

Your production data has a different distribution from your training data

This can happen, data analysis would be required to double check how your production data might deviate. If this is the case, I would investigate what might cause this, is it something you can adapt to? Or is it due to an external factor you can measure?

Your network is too large and overfitted artifacts in training

If your network is large, you might have overfitted on certain artifacts that are present in your train and test set and therefore, it performs badly on your production data. I would train smaller networks or on many smaller snapshots to double check that your model can learn appropriately.

Your evaluation method is not appropriate

When training time series, make sure to always evaluate on future data and not past (that's cheating!)

Bugs

Obviously, bugs may always be the cause of some issues. Maybe preprocessing is doing something funky.


can we have different features for different hyperplanes in SVM?

It is possible.

All of these hyperplanes live in the space of

$$\text{(age, trip duration, income, cost, purpose of trip, distace)}$$

even though some of the coefficients are $0$.

Remark: For the last hyperplane $w_3\cdot 0 = 0$.


What undergraduate degree should I take to get into data science?

If you can manage it, I suggest focusing more on Statistics. Statistics can be a demanding field, and takes a lot of effort to attain a degree. Of course, a Data Scientist also needs several skills besides Statistics, such as Software, Data Engineering and Visualization, but Statistics knowledge/skills are the most rare to find, and therefore can be the most valuable.
I worked the last 8 years in a data science department and found that when looking to hire a data scientist, it was often difficult to find someone with a solid statistical background. Most candidates had software, engineering, data and visualization skills. However, someone with a stronger statistics background would be hired over the other applicants, and usually with a higher salary.
Internships are very valuable, they will help you get your foot in the door before you graduate. Some companies may even offer you a job during your internship if they like your work. Some of the bigger companies will pay for your continuing education as they will get a better employee. It is a good investment for both you and them.
A Master's degree will also be beneficial, but may limit your chances to get a lower paying entry level job. Larger companies will usually prefer a Master's over a Bachelor's degree.
Response to your followup question:

I strongly suggest doing the internships, they can dramatically
improve your chances to get a job when you graduate, especially with
the companies you did the internship with.
The choice between Masters with little experience vs Bachelors with
several internships would be much more difficult to make.  Overall,
the better choice would be to get the Masters degree with a few
internships along the way.
Also, I suggest going with Statistics all the way, Bachelors and
Masters. Several of the topics in Data Science will evolve over the
6+ years it takes to get the Masters, but the underlying Statistics
will change very little.
Maybe get a minor degree in Data Science or a Statistics degree with
an emphasis in Data Science.
You can also consider looking for a job in between the Bachelors
degree and Masters, and if you find a good job, you can always get
your Masters degree while you are working and earning money.


How to use K.function with two inputs and a concatenate layer?


CUDA compatibility of GTX 1650ti versus 1650

Both can run CUDA to accelerate deep learning. Cards that end in "ti" are slightly better versions than their non-ti counterparts.
NVIDIA, the corporation that makes the 1650 and 1650ti cards, also develops CUDA. Only (and all) NVIDIA graphics cards currently support CUDA because it is proprietary.
However, because deep learning models are often limited by memory capacity, I recommend buying a card such as the GTX 1070, which is comparably priced but has 8GB of RAM.


Is there any data tidying tool for python/pandas similar to R tidyr tool?

I'd start with the melt() function in pandas.  I wrote an article about it:
https://www.ibm.com/developerworks/community/blogs/jfp/entry/Tidy_Data_In_Python?lang=en


Which output of a BiLSTM layer should be used for classification

Nice question! I'm looking at the PyTorch documentation: https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html
If i get that right, lstm_out gives you the output features of the LSTM's last layer, for all the tokens in the sequence. This might mean that if your LSTM has two layers and 10 words, assuming batch size of 1, you'll get an output tensor of (10,1, h) assuming uni-directionality and sequence-first orientation (also see the docs).
For the same LSTM, the size of the ht will be (2*1,1,h) because this output only relates to the features of the last word in the sentence, and is collated from all the layers in the LSTM, unlike the lstm_out which only look at the top layer.
In a nutshell, i think you want to use lstm_out for sequence classification tasks because you have features for all the tokens. Then you want to use ht for tasks that apply at sentence level like classification, as i think it is assumed that the features of the last token are the most rich in an LSTM and representative of the sentence because it contains LSTM-like linkages to all the other tokens (assuming uni-directionality, i think for bi-directionality PyTorch will slice the backward states across the layers for the last sequence too)..
if you want ht to be (1,m,n) instead of (2,m,n) , then you need only 1 layer and 1 direction. This is closely linked to the design of your architecture though, you would need to think through whether you really want to get rid of bidirectionality and keep a shallow LSTM.


How is data science applied in investment banking?

In investment banks, this role is called "Quant" (Quantitative analyst). A quantitative analyst is a person who specializes in the application of mathematical and statistical methods to financial and risk management problems. The occupation is similar to those in industrial mathematics in other industries.

https://www.phaidoninternational.com/industryinsights/the-growth-of-quants-in-investment-banks-71503010311421

https://www.investopedia.com/articles/active-trading/111214/quants-what-they-do-and-how-theyve-evolved.asp

https://www.investopedia.com/articles/financialcareers/08/quants-quantitative-analyst.asp

https://en.wikipedia.org/wiki/Quantitative_analyst


Convergence in Hartigan-Wong k-means method and other algorithms

H-W's algorithm, from the 1979 paper, takes as input initial clusters. However, the authors suggest a method for obtaining them in their last section. They write that it is guaranteed that no cluster will be empty after the initial assignment in the subroutine. It goes as follows:

Compute the overall mean $\bar{x}$.
Order the observations with respect to their distance to $\bar{x}$, that is $||x_i - \bar{x}||_2$ (in ascending order I guess?).
Take the points in position $\{ 1 + (L-1) [M/K] \}$, where $L=1, \dots, K$, as initial centroids. ($[\ \cdot\ ]$ most probably refer to the floor function, hence the $1$ at the beginning.)

As for the main algorithm, it is described in a paper called Hartigan's K-Means Versus Lloyd's K-Means-Is It Time for a Change? by N Slonim, E Aharoni, K Crammer, published in 2013 by AJCAI. Note that this version simply uses a random initial partition. It goes as follows.

For the vectors $x \in \mathcal{X}$ and a target number of clusters $K$,

Set $\mathcal{C}$ to be a random partition of $\mathcal{X}$ into $K$ clusters and compute the centroid vector associated to each $C \in \mathcal{C}$, denote them $v_C$.
Scan $\mathcal{X}$ in a random order, and for all $x \in \mathcal{X}$

2.1 Set the stopping indicator to $s = 1$

2.1. Tentatively remove $x$ from its cluster $C$, letting $C^{-} = C \setminus \{ x \}$.

2.2 Find
\begin{align*}
C^+ = \Big\{ \mathrm{argmin}_{C^* \in (\mathcal{C} \setminus C) \cup C^{-}}\ \frac{1}{n} d(x,v_C^*) + \frac{1}{n} \sum_{y \in C^*} [d(y,v_{C^* \cup x}) - d(y,v_{C^*})] \Big\} \cup \{ x \}
\end{align*}

2.3 If $C^{+} \neq C$, then reset $C \leftarrow C^{-}$ and $C^* \leftarrow C^{+}$ and update their respective vectors $v_{C}$ and $v_{C^*}$. Also reset $s \leftarrow 0$.
If $s=0$, go back to 2.

Here I make a slight abuse of notation letting $C^*$ be the solution of the $\mathrm{argmin}$ in 2.2. This step, 2.2, simply computes the change in the loss obtained if $x$ is added to $C^*$, as opposed to it staying alone in its own cluster (once removed). $d$ stands for distance. Finally, the particular ways to compute the updated vectors $v_C$, $v_{C^* \cup \{x \}}$ and so on is provided in the paper cited above. They can be done in linear time.

I think the answers to all your questions are implicit in the above algorithm... However, I still have to make sure this implementation of the algorithm is standard. In particular if it is the one implemented in R. Any comments/edit are welcomed.


Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work?

This is a quite common mistake, you transformed the test and training data separately which messes with factor levels.

There are multiple solutions to this:

1) Create a common transformer

If you there is no "new" data but simply a test set (e.g. like in kaggle competitions or similar problems) you should create a function that transforms / tidies ALL data at once and only split the data after it already has the proper form for the modelling (including OHE, etc.).

2) Save your factor levels!

For all other case you should save your factor levels and create a transformer accordingly. Here is a code example:

text_levels <- levels(train$text)

levels(test$text) <- text_levels

Realize also, that any word or factor level not present in the data you train your model on cannot be considered in the model and therefore is useless! All above solutions simply add the appropriate factor levels from the training model if they are missing in the test data.

If you have new words in the test data you will simply need to drop them before transforming the data as they are not used in the prediction model.


How to evaluate clustering success in a completely unsupervised system?

First, PCA is not a clustering method. It is a dimensionality reduction scheme.  You can assess the performance of PCA through analysis of the percent of variance in the dataset that is retained as you decrease the number of dimensions.  Retaining 99%,95%, or 90% is usually ideal, depending on your problem.

With regards to clustering, you probably want to start with the Silhouette Coefficient.  This combines assessments of both the cohesion (how tight a cluster is) and separation (how well separated each cluster is from other clusters).

Calculate a = average distance of i to the points in its cluster
Calculate b = min (average distance of i to points in another cluster)
The silhouette coefficient for a point is then given by:
s =1 – a/b, if a < b

It is typically between 0 and 1 with larger numbers being "better".  You can average the coefficients over a cluster or the entire region to get an assessment of the cluster or the entire clustering procedure for your data.

More generally, try googling "assessing clustering" or "cluster validity" to read about all of the other ways that you can score your clustering algorithm.  Here is a very complete treatment of the topic.

Hope this helps!


Should the type of Boolean categorical features be numerical or categorical after encoding?

In mathematics and mathematical logic, Boolean algebra is the branch of algebra in which the values of the variables are the truth values true and false, usually denoted 1 and 0 respectively. Generally, gender can not  be considered  - true or false . Classification is based on sex - Male or female  . Categories differ and can be assigned particular  numerical values (dummy variable) 1, 2 or 2, 3 etc. Binary variable is different. If you want to find correlation  between  the scores obtained by Male students and female students,  you will have to have matched pairs. Categories measure the levels for grouping. Binary classification can be undertaken for a dependent  variable in regression analysis or point biserial correlation. The standard softwares for statistical analysis do not invoke 0,1 as variable. The probability based solutions may use this classification for grouping.


Choice of objective function for log transformed data for feedforward NN

I am assuming that when you say log transforming the data, it is log transformation of outcome variable. In such case, you may want to check root means squared logarithmic error.


how to use (read) google pre-trained word2vec model file?

I have searched about it and fixed the error through these steps:
you should load the "GoogleNews-vectors-negative300.bin.gz" file at first then extract it by this command in Ubuntu: gunzip -k GoogleNews-vectors-negative300.bin.gz. [ manually extracting is never recommended]. Secondly, you should apply these commands in python 3:

import gensim
model = gensim.models.Word2Vec.load_word2vec_format('./model/GoogleNews-vectors-negative300.bin', binary=True)

I hope it will be useful.


Dealing with NA values while using Rotation Forest in R

Are your variables categorical? If they are, then you do not need to remove them. You could just make new columns like NA_Variable1, which will indicate whether a NA value exists.


Apache Spark can do it, using the new MLLib library. Here's a presentation, and here are some benchmarks. Bindings are available for python, scala, and java.


That is an empirical question. The answer will change for different models and different datasets.
The best approach would use cross validation to see which ensembling technique has the best score on the evaluation metric for the given data.


PELU activation: how does it work, and how to implement?

I have not used PELU activation function by myself, so dont know much about its performance benefits but can say

a, b, and c looks to be hyperparameters only.
and it can be directly implemented in tensorflow 2.0 as

tf.cond(h, lambda:c*h, lambda:a(tf.exp(h/b)-1))


Is linear regression suitable for these data?

You can employ the linear regression algorithm even for categorical data. The point is that whether your data is learnable or not. For instance, take a look at your data, and see whether an expert can really find the output by taking a look at the input vector. If it's possible, your task can be learnt using linear regression method.

About linearity, the point is that linear regression can also learn nonlinear mappings. You just have to provide enough higher order polynomials of the current feature space you have which is not an easy task. For instance, you can expand your current feature space by adding the square of each feature to the current feature space. You will observe that it may have better performance than the usual case if your mapping is not linear, but you may still have error. Consequently, you have to supply more polynomial features, but you do not know which to use.

An alternative to linear regression which does not need to add extra features is multi layer neural networks (MLP). You can simply use them which can learn nonlinear mappings. You can take a look at the official page of SKlearn for applying them. Furthermore, you can take a look at here for applying them.


From a quick search, the function you provided doesn't seem to have a particular name.

Are you sure the function belongs to Sigmoidal curves? The function does not remind too much of an S-shape.


Develop a marketing look-alike profile for prospective customers

The problem is that you only have positive instances (businesses who have become a customer) but no negative instances (businesses who haven't become a customer).  This prevents use of supervised learning.  You could try unsupervised learning -- e.g., a one-class classifier -- but I don't expect it to be particularly effective.

If you had data on businesses who you tried approaching, but ultimately did not become a customer of yours, then you could try applying supervised learning.


how to assign back categorical variables to train and test data after training and testing using inverse_transform?

If you are using ordinal data
Use this to encode

from sklearn.preprocessing import OrdinalEncoder
oe=OrdinalEncoder()
social['Gender']=oe.fit_transform(social[['Gender']])

Use this to decode
social['Gender']=oe.inverse_transform(social[['Gender']])

If you are using nominal variables
Use this to encode
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder()

temp_var= ohe.fit_transform(social[['Gender']])

Use this to decode
temp_var2=ohe.inverse_transform(temp_var)

Hope it helps


The idea you have in mind is called "feature selection" or "attribute selection". The fact that you have a categorical dependent variable and continuous independent variables is mostly irrelevant because you're expected to use an algorithm or statistical method that is suitable for your requirements.

As for feature selection methods, there are several options:

Find the subset of features that achieves better performance (usually in cross validation)
Find the subset of features that correlates highly with the target variable and low with each other (although other criteria can be used)
Use an algorithm that includes a built-in feature selection mechanism (e.g. decision trees, hierarchical bayesian methods)

Furthermore, there are several methods aimed at obtaining a good compromise between a thorough search and a reasonable time execution (e.g. best first, steepest ascent search, etc)

This question in particular provides very good suggestions for R packages.


Microsoft announced a couple of weeks ago virtual machines in Azure with GPUs. They use K-80 NVIDIA cards. The biggest machine has 4 GPUs and 224 GB of ram.

Good to play with deep learning :-)