What does this formula in Glorot & Bengio mean?

1. Meaning of the formula
$$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$
This formula expresses the variance of the values of the activation for each neuron $k$ of the layer $i \rightarrow z^i_k$. This value, under the assumptions that are going to be mentioned along this post, is the same for all the neurons in the layer $i$. This is why the authors express it as $Var(z^i)$ instead of a particular $Var(z^i_k)$. Just to simplify the notation
Note that this value is scalar, as well as the other variances involved in the equation: $Var(x)$ and $Var(W^i)$. So, just as a summary, these variances have the next meanings:

$Var(W^i)\rightarrow$ The variance of the random initialization of the weights.
$Var(x)\rightarrow$ The variance of each feature (which is asummed to be the same for every feature as we are going to see in the next lines).
$Var(z^i)\rightarrow$ As said, this is the variance of the activation of each neuron of the layer $i$.

So now let's see how to reach this formula.
2. Reaching the formula
2.1 Analysing some parts of the paper
Let's analyse some parts of that page of the article that answer the other questions and that are going to be also useful to understand the assumptions that are being made.

For a dense artificial neural network using symmetric activation function $f$ with unit derivative at $0$ (i.e. $f'(0) = 1$), if we write $z^i$ for the activation vector of layer $i$, and $s^i$ the argument vector of the activation function at layer $i$, we have:
$$ s^i = z^iW^i + b^i\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,z^{i+1}=f(s^i)$$

We can draw some conclusions given that paragraph:

The authors are considering an activation function subject to $f'(0) = 1$ in that part of the article. The fact that $f'(0) = 1$ means that locally the function $f(s^i)$ will behave linearly at the neighborhood of $s^i_k=0\rightarrow f'(s^i_k) \approx 1$, where $k$ is the index of the neuron.
As you said $\rightarrow z^i$ represents the activation output of one layer

The variances will be expressed with respect to the input,
outpout and weight initialization randomness

This means that the variances that they are going to use are going to be $Var(x)$, $Var(z^i)$ and $Var(W^i)$ respectively, and during the initialization (the weights are only spread randomly at the initialization).
The variance of the biases is not being considered as they are assumed to be initialized to the same value at the initialization $ \Rightarrow Var(b^i)=0$.

Consider the hypothesis that we are in a linear regime at the initialization, that the weights are initialized independently and that the inputs features variances are the same ($=Var(x))$.

Given this, we know that:

Linear regime $\Rightarrow f'(s^i_k) \approx 1$ as we saw above.
Weights are initialized independently $\Rightarrow Var(W^iW^{i+1})=Var(W^i)Var(W^{i+1})$
Inputs features variances are the same $\Rightarrow Var(x_k) = Var(x_{k+1}) \,\,\forall\,\, k$ where $k$ is the index of the input

2.2 Example
So now, let's compute for example the Variance of the activations of the first hidden layer ($i=1$) i.e. $Var(z^1_k)$:
$$ \begin{align}
Var(z^1_k) &= Var(f(s^0_k)) && (1)\\\\
&\approx (f'(\mathbb{E}(s^0_k))^2\,\,Var(s^0_k) && (2)\\\\
&= (f'(0))^2\,\,Var(s^0_k) && (3)\\\\
&= Var(s^0_k) && (4) \\\\
&= Var(W^0_k x + b_k^0) && (5)\\\\
&= Var(W^0_k x) = Var(w^0_{k1} x_1 + w^0_{k2} x_2 +... )  && (6)\\\\
&= Var(w^0_{k1} x_1) + Var( w^0_{k2} x_2) +...  && (7)\\\\
&= n_0Var(w^0_{kj})Var(x_j)  && (8)\\\\
&= n_0Var(W^0)Var(x) && (9)
\end{align}$$
Note that we would end up with the same expression of $Var(z^i_k)$ for every neuron $k$ of the layer $i \rightarrow $ Now we understand why $Var(z^i)$   represents the variance of the activation for each neuron in the layer $i$ i.e. $Var(z^i) = Var(z^i_k)$.
2.3 Justifications
The justifications for each step are:

$z^{i+1}=f(s^i) \Rightarrow z^{1}=f(s^0)$
The justification to this can be found in the Wikipedia post "Approximating the variance of a function"
$\mathbb{E}(s^0)= \mathbb{E}(W^0x) = \mathbb{E}(W^0)\mathbb{E}(x)=0$. This is because $x$ and $W^i$ are independent and $W^i$ is assumed to be randomly initialized with mean $0$.
Linear regime $\Rightarrow f'(s^i_k) \approx 1$.
$s^i = W^ix + b^i$
Bias values are initialized independently and $Var(b^i)=0$ as we mentioned above.
This is true because $Cov(w^0_{k1} x_1, w^0_{k2} x_2)=0$. Something that is proved at the end of the post.
(and 9) This is because the weights and inputs are independent, and because of the assumptions made by the authors, which are:
$$Var(x_j)=Var(x)\,\,\forall\,k \,\,\,\,\,\,\,\text{ and }\,\,\,\,\,\,\, Var(w_{jk})=Var(W)\,\,\forall\,\,k,j$$

3. Conclusion
Extending this reasoning to all the layers we end up with the equation given by the authors:
$$Var(z^i)=Var(x)\prod_{i'=0}^{i-1}n_{i'}Var(W^{i'})$$

Edit regarding the comments
The reasoning holds even if the features are not independent. This is because, we can proof that the covariance of the $z^{i+1}_k$ terms in the same layer $i$ is $0$. This also serves as an explantion to the step $(7)$ of the example above.
To see this let's compute $Cov(w_k^iz^i,w_j^iz^i)$ where $w_k^i$ and $w_j^i$ represent the vector of weights related to the neurons $k$ and $j$ of the layer $i$, with $k\neq j$:
$$\begin{align}
Cov(w_k^iz^i,\,\,w_j^iz^i) &= \mathbb{E}\left[(w_k^iz^i-\mathbb{E}(w_k^iz^i))(w_j^iz^i-\mathbb{E}(w_j^iz^i))\right]\\
&=\mathbb{E}\left[(w_k^iz^i)(w_j^iz^i)\right]\\
&=\mathbb{E}\left[w_k^iz^i\,w_j^iz^i\right]\\
&=\mathbb{E}\left[w_k^i\right]\mathbb{E}\left[z^i\,w_j^iz^i\right]\\
&=0^T\,\mathbb{E}\left[z^i\,w_j^iz^i\right]=0
\end{align}$$
Note that we can do $\mathbb{E}\left[w_k^iz^i\,w_j^iz^i\right]=\mathbb{E}\left[w_k^i\right]\mathbb{E}\left[z^i\,w_j^iz^i\right]$ because $w_k^i$ is independent of $z^i\,w_j^iz^i$
Then by extending this to the other neurons of the layer, we can confirm that: $Var(W^iz^i) = n_iVar(W^i)Var(z^i)\rightarrow$ We can reach the equation given by the authors.


How should ethics be applied in data science

I think ethics in Data Science is important. There is a fundamental difference in using user data to better their experience and show relevant ads and using user data to trick people into clicking on ads for the sake of monetary profit. Personally I like ads that give me relevant information like deals on things I would buy anyway. However, showing me weight loss ads because I got dumped is creepy and unethical. As my friend Peter always says, "don't be creepy with data".


what approach to use for find best customer out of data?

Your intuitions are good: in general you need your data to contain all the possible indications (features) which might help find the answer.

And your thinking is also correct about creating this "other table": the reason why you need it is because for this problem you need each instance to correspond to a customer. Your original data is not organized by customer, it's organized by meeting. So it makes sense to organize your data with features by customer such as these:

days_since_last_visit   number_meeting_scheduled   number_meetings_attended   ...
15                      4                          2
189                     3                          1
24                      2                          2
...

In general it's not recommended to assign scores yourself, because you probably don't know what is the optimal value. For example is a customer better when they schedule 10 meetings and attend 3 or schedule 4 and attend 2? It's usually better to give all the raw values you have to the ML algorithm and let it calculate the best way to use it.

An important point to define is: what is a good customer exactly? If you think your scoring is really accurate to define a good customer, you can calculate the score and rank the customers, then you're done: this is a heuristic, because you calculate the answer directly based on your knowledge of the problem.

Now assuming you are not so sure and want to use ML:

If you can have a sample with labels which say whether a customer is "good" or not, then you have a supervised classification problem: the goal will be to train a model able to predict the class (category) for any customer based on the features.
If you have a sample with numeric values indicating "how good" a customer is, then you have a supervised regression problem. Again the goal is to train a model which predicts the value for any customer based on the features.

For both cases above I'd suggest you start with simple methods such as decision trees or SVM (both can do classification or regression). The former has the neat advantage that you can manually observe the tree and understand how the classifier works.

If you don't have any labeled data, then your only option is unsupervised learning, which usually means some form of clustering: in this case there's no training stage, the algorithm is provided only with the features and it tries to group together the instances which are close to each other. A standard approach for that would be K-means, for example. Note that you're not sure to obtain the groups that you expect in this case.


Probability that ensemble model is correct based on accuracies of its classifiers

It's not the actual data, it's the probabilities. So you should consider all the scenarios of voting.
For the Ensemble to be correct,
Either any two or all the three should be correct
=$[m_1*m_2*(1- m_3) + m_1*(1-m_2)*m_3) + (1-m_1)*m_2*m_3] + [m_1*m_2*m_3]$
= [0.6*0.55*(1-0.45) + 0.6*(1-0.55)*0.45 + (1-0.6)*0.55*0.45] + 0.6*0.55*0.45
= 0.5505


How do I learn experimental methodology? When is it relevant?

On the metrics. A common technique is to make simulated data with known labels. Then see if your algorithm can reproduce the clustering; if it can't there's a problem, if it can there may still be problems but you're off the starting blocks. Typical measures would include ROC and AUC (essentially how many times does the algo correctly place object i and j in the same class, balanced against false positives).

Another tactic is to use well known real datasets to benchmark against. E.g. the classic iris dataset. See how your algo performs in terms of speed and accuracy versus other algos.

As to what people are looking for this could mean a multitude of things. You might do well to consider where your algo is suited, find an application in that area, and find out whether it performs for that purpose.


Choosing the right corpus to build a TF IDF Vectorizer used for comparing the similarity of two strings

Word Mover’s Distance (WMD) is an algorithm for finding the distance between pairs of strings. It is based on word embeddings (e.g., word2vec) which encoded the semantic meaning of words into dense vectors.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper


Reading the article they state that

  The two vertical lines represent the $L^2$ norm of the error, or what is known as the sum-of-squares error (SSE)

Otherwise known as the Euclidean norm, Euclidean length, $L^2$ distance, $ℓ^2$ distance, $L^2$ norm, $ℓ^2$ norm, $2$-norm, or square norm.

Also speaking from my experience, most of the time when a norm is specified without a subscript then you can generally assume that it is the Euclidean norm.


How to build recommendation model based on resume and job description?

If you want a DL approach, I recommend substituting the tf-idf by some kind of word embeddings.
For instance, you can take a pre-trained word embedding model, like glove, and average its outputs both in resume and job description, and then compute cosine similarity. However, I recommend to use a contextual word embedding (BERT-like), as the terms in resumes might be very dependent on the context.
The following article also introduces sentence-bert, which I think is very suited for your problem.


Company secrets - Google does not disclose what they actually use. Any answer to this is pure speculation.

It probably is not even just "one" algorithm, but many. But it may be as simple as finding overlapping dates and locations, assuming that people cannot be in two places at once. You could test this: book a flight, and a hotel on a different destination during the same time interval. I wouldn't be surprised if they nevertheless show up as one "trip" (and it doesn't really degrade user experience much).

It will very likely be a supervised approach. Which does not mean "interactive". On the contrary, unsupervised approaches will likely need interactions to choose parameters. Using some training data (e.g., from employees, or user interactions) to improve matching quality looks promising, though.


does smaller training set always lead to better training accuracy?

I don't think there a connection between size of the training set and model's performance during training. A dataset can be very noisy and unpredictable, independently from its size.

To my knowldge, it is not possible to derive any general rule about this.


Multiclass Classification with Decision Trees: Why do we calculate a score and apply softmax?

Depending on the parameters you used for your model, it may not be calibrated in probabilities. That is, your model output a score, that is helpfull to give a relative order between your instance, but the score may not reflect the real % chance of the output happening.

Softmax, will at least garanty that your output are between 0 and 1 and sum to one. This will give you an output that is nearer to a probability (but that may not be enough to be calibrated to 'historical' probabilities).

PS: I don't think there is a link with xgboost, unless you have very specific options.


Counting the number of layers in a neural network

This is a 2-layer network because it has a single hidden layer and an output layer. We don't count the first layer. When we say 3 layers, we actually mean 2 hidden layers and an output layer. Perhaps this helps you?

EDIT: We don't count the input layer because there's no parameter (bias + weights). In actual implementation, it's not implemented. Netural network framework simply "connects" the input features to the first adjustable layer (eg: hidden layer).


How can I do a train test split for an unbalanced panel data set in Python?

Not sure what do you mean by panel dimension, I think would be best to clarify it more. Nonetheless, I can tell you for your 2nd question Stratified Sampling takes into account the imbalance. And if you want to have one group t on the training and t+1 on the test you can use group sampling. Sklearn has both implementations [1] [2].


Handling near duplicate observations in a regression / Bayesian model

Some models are fine with duplicate data. For example, this is how oversampling works in unbalanced datasets, Also Naive Bayes should handle this.
However there are many other models will end up giving high prediction rates because of this. Duplicate rows can cause problems due to the same data being in the train and test datasets. Those rows will give a high accuracy to the model, since the training dataset already knows the answer.
Exact duplicates can be found and removed using

df.duplicated()
df.drop_duplicates(inplace=True)

Finding similar data will be a bit more difficult. You can use these same 2 functions on a partial set of columns if you think that will find more matches. Or you can run the data through a cluster algorithm (k-means) and sample from the different clusters, or add a weight column based on the clustering output, maybe divide by the number of points in each cluster. You can then pass in this weight vector into the model.


Yes, it has to use the same featues. If not, the training algorithm does not make any sense.


How to make a region of interest proposal from convolutional feature maps?

To implement region proposal you need two major parts:

The region proposal network that generates a set of candidate bounding boxes. It can be implemented simply as two convolutional layers to 1) predict the object presense and 2) predict offsets for the default (anchor bounding boxes)
The ROI pooling layer that provides a fixed-size feature vector for an arbitrary sized proposal.

Here is an implementation of Faster R-CNN in Keras, and here is a detailed explanation of the model and the code.

Here is implementations of the RPN, and here is implementation of the ROI pooling.


Can Micro-Average Roc Auc Score be larger than Class Roc Auc Scores

In a binary problem there's no reason to average the ROC (or any other metric). Normally micro and macro performances are used to obtain a single performance value based on individual binary classification measures in the case of multiclass classification.
So here what happens is this:

The ROC curves for the two classes are the mirror of each other (using the top-left bottom-right diagonal as symmetry axis), because they represent exactly the same points but with the positive and negative classes swapped.
The macro-average curve is the average of both curves, which doesn't make a lot of sense since both already have the same shape. This is why all the AUC values are identical for macro, class 0 and class 1.
The micro-average ROC is the weighted average, so it's made mostly of the majority class (around 75%): since most points in the majority class are correctly predicted as the majority class, the performance looks much better. This is related to the fact that the micro-average F1-score is equal to accuracy, although in the ROC I'm not sure how the points are weighted exactly.

In my opinion, this is a good illustration of why ROC curves should be used very carefully (or not at all) in a multiclass setting. ROC curves are meant for binary (soft) classification, they are useful and interpretable in this context but not necessarily in another context.
In general it also illustrates that it's not because something can be calculated that the resulting value makes sense ;)


How to handle "unknown" category in machine learning classification problems?

I think this is one of those topics with the most frustrating answer - it depends.

To your questions:

How can we handle these data which fall into "unknown" category?

There are many ways of doing this. Some are very simple and some are more complex, but they all depend on you understanding your data and what exactly causes the missingness - e.g. is the data missing at random or is there a specific cause driving it?

Some techniques to treat missing values (in increasing order of complexity):

Exclude all missing values. This may be fine if you have a large amount of data and few missing values (not always the case - you allude to this by mentioning the remaining data set may be imbalanced).
Replace / group the missing values with an appropriate value - e.g. replace missing values with the mean of the variable / group missings with the most populous level.
Impute missing values using models / equations - e.g. Multivariate Imputation by Chained Equations (MICE).

How do we train the model to deal with "unknown" data?

This depends on the model or technique you are using. Some techniques deal with missing values well (e.g. xgboost), while others do not (e.g. R's ranger implementation of random forests). You should take into account the model you are using when deciding on how to treat your missing values.

Or do we ignore it?

Ignore missing values at your peril!

Hope that helps!


Matrix multiplication is just a simplified notation for a particular set of addition and multiplication operations. You can absolutely represent a neural network without invoking matrix notation, it'll just be really tedious (and it will run slower).

Start with a single perceptron and build your way up from there.


Apply LSTM to each matrix element with Keras

SOLUTION

Replace:

# Apply if statement
y = keras.backend.switch(cond,
                          lambda: keras.layers.GRU(units,return_sequences=True)(seq),
                          lambda: keras.backend.zeros(shape=(1,time_steps,units)))

y = keras.layers.Lambda(convert_to_layer)(y)

with:

y = keras.layers.Lambda(lambda x: keras.backend.switch(cond,
                          lambda: keras.layers.GRU(units,return_sequences=True)(x),
                          lambda: keras.backend.zeros(shape=(1,time_steps,units))))(seq)


Since the information on all variables is already captured by the cluster variable, using it again in the Lasso model with some other variables, does it become redundant/biased?

It does not become biased, at least not in the statistical sense. Depending on the clustering mechanism, the information may be redundant.

If you're using regression with the RMSE as the loss function and the squared-error as your loss metric for your clustering algorithm, then the information will work out to be redundant.

Regardless, it's harmless to throw into the model and test this. The LASSO should learn that this information is redundant.

Here's a simulation in R.

library(glmnet)
library(Matrix)

n <- 1e5
nclusters <- 5
set.seed(420)
rmse <- function(y, yhat){
  return (sqrt( sum( (y-yhat)**2 )))
}
ls <- data.frame(sample(letters, n, replace=TRUE))
xs <- sparse.model.matrix(~.-1,data=ls)

print(head(xs))

# Now let's run k-means
out <- kmeans(xs, centers=nclusters)
bs <- runif(dim(xs)[2])

# Let's run k-means on the different categories
clusterpred <- data.frame(out[[1]])
ys <- xs %*% bs + rnorm(n)
print(table(clusterpred))

# Now let's use a clustered data set to predict some outcome
cxs <- sparse.model.matrix(~.-1, data = data.frame(cluster = factor(clusterpred[,1])))
# Concatenating the original features and the assigned clusters
totalxs <- cbind2(xs, cxs)

head(cxs)

# Setting alpha = 1 implies LASSO for the GLMNET Package
model <- cv.glmnet(y=ys, x=xs, alpha=1)
cmodel <- cv.glmnet(y=ys, x=cxs, alpha=1)

# Running on the clusters and the original features
totalmodel <- cv.glmnet(y=ys, x=totalxs, alpha=1)

# Predictions
yhat <- predict(model, xs)
yhatc <- predict(cmodel, cxs)
yhatt <- predict(totalmodel, totalxs)

# Looking at the difference RMSEs
print(rmse(ys, yhat))
print(rmse(ys, yhatc))
print(rmse(ys, yhatt))

# It seems to select most of the features and *one* cluster
print(coef(totalmodel, s='lambda.min'))

In this simulation, we see that the performance of the model slightly worse, but fairly similar, from including the clusters.


What needs to be done to make n_jobs work properly on sklearn? in particular on ElasticNetCV?

When I ran your script, I got the same impression, that n_jobs was hurting you performance. However, you have to consider that parallelizing the cross-validation would only benefit if you have more data samples. With few data, the communication overhead indeed is more expensive than the processing cost involved on the task.

I tried your script with more samples SAMPLE_N = 100000 and got the following results. Setup: macos i5 8gb.

n_jobs = None, perf_counter = 21.605680438
n_jobs = 1, perf_counter = 22.555127251
n_jobs = 2, perf_counter = 15.129894496000006
n_jobs = 3, perf_counter = 11.280528144999998
n_jobs = 4, perf_counter = 13.050180971000003
n_jobs = -1, perf_counter = 20.031103849000004


The initial value of the CNN kernels can be seen from the documentation found here.

The default Conv2D layer looks like

keras.layers.Conv2D(32, (5, 5), activation='relu', kernel_initializer='glorot_uniform', bias_initializer='zeros')

So we see that the kernel weights are initialized by Glorot uniform method an dthe bias is initialized as all zeros.

Glorot Uniform

Uniform sampling from $[-limit, limit]$ where

$limit = \sqrt{\frac{6}{fan\_in + fan\_out}}$

where $fan\_in$ is the number of input neurons to the weight tensor and $fan\_out$ is the number of output neurons of the weight tensor.


How to obtain a count of the classes of a categorical var within a certain time interval for a time-stamped data?

import pandas as pd
data = pd.read_csv("/content/da_time.txt",header=None) # File has your example data
data.columns=['num','date_time','var']     # Named columns
data.date_time = pd.to_datetime(data.date_time) # To Pandas datetime
data['date_time_copy'] = data.date_time   # Created a copy of datetme column

# make second as multiple of 15 and millisecs=0
data['date_time_copy'] = data.date_time_copy.apply(lambda x:x.replace(minute=x.minute//15*15,second=0,microsecond=0))

data.groupby(by=['date_time_copy','var']).count()['num']

Output


How to identify potential customers who are ready to convert in to paid?

Two items come to mind:

You will gain the most benefit from this site when you propose a model and let other people comment on and modify the model. As it stands, you are merely proposing what your data looks like. Do you have a model that you've tried? What data science steps have you already taken with your project? If you post that, this site will be of much better help to you
You should be aware that what you posted makes for a good initial dataset but that's all it is, your initial set. If I was in your position, I would be working towards developing a meta dataset that contains a much larger number of factors. When did the person join? What was the time in between actions? Which country is their IP address from? How many times have they logged into the tool? Etc, etc, etc. When you start to think of your data in this manner, and give yourself a much larger number of factors to start with, then you are taking steps that are much more likely to result in a reliable predictive model.


How to show progress of sklearn.multioutput.MultiOutputRegressor and XGBRegressor?

You can consider modifying the code of MultiOutputEstimator or XGBModel to introduce some debugging output. These seem to be the classes that implement the fitting logic of the two libraries. The corresponding source files should also be provided in the installations on your local hard drive.

For example, you can print information of when the separate threads are starting and stopping in MultiOutputEstimator.fit (inherited and thus reused in MultiOutputRegressor), lines 167-169. Also, consider the tip in the documentation of the n_jobs parameter:

If 1 is given, no parallel computing code is used at all, which is useful for debugging.

You can use a similar approach to expand the debugging in XGBModel.fit (the basis of XGBRegressor).


Apply cross validation on regression algorithms

How to use cross-validation on regression (assuming 10-fold for example purposes): separate your dataset in 10% and 90%, train on 90%, test your metric (squared error or anything you're modeling) on the remaining 10%. Do that 10 times using different 10% groups. Now you have 10 of your metrics, you can analyze its mean and range to see if the model is robust against over-fitting.

How do you pick a model: you train on the whole dataset. Cross-validation is for testing/validating purposes, you don't use the models generated by cross-validation.


What is valued more in the data science job market, statistical analysis or data processing?

Given your current situation, a data analyst or scientist position makes more sense, but with some practice you can acquire the software skills to become a data- or machine learning engineer. There is a glut of people from academia similar to you, and software companies want people who can actually build software, so it is not surprising you see demand for engineers. But that does not mean you have to become one. Do you want to build software products?

My recommendation would be to make sure you are familiar enough with common applications of statistics in industry (e.g., A/B testing, and time series forecasting) to get a job in analysis, then learn computer science on the side. There is a wide range of salaries, depending on seniority, company and location. You can find specific figures on sites like LinkedIn, Glassdoor, Paysa, Payscale.


It seems there are a couple issues at work here.

First, apparently the PCA computation wasn't stalling out because of the absurd dimensions being reported - updating to tensorflow 1.12 and Keras 2.2.4 has eliminated that problem.

Second, it seems that Keras bases the generation of embedding data for tensorboard on the layer inputs rather than the actual layer weights. Maybe this is important for some other usage, but it makes things very complicated in a standard word/token embedding scenario. To work around this, I ended up skipping embeddings during training, exporting the embedding weights at the end, and then training a new throwaway model with a much simpler topology to generate the projections I'm interested in. Given an object model trained from the samples above:

w <- model %>% get_layer(index = 1) %>% get_weights()

emodel <- model <- keras_model_sequential() %>%
  layer_embedding(input_dim = max_features, output_dim = 128, input_length = 1) %>%
  layer_global_max_pooling_1d() %>%
  layer_dense(units = 1)

emodel %>% get_layer(index = 1) %>% set_weights(w) %>% freeze_weights()

emodel %>% compile(
  optimizer = "rmsprop",
  loss = "binary_crossentropy",
  metrics = c("acc")
)

callbacks = list(
  callback_tensorboard(
    log_dir = "my_log_dir",
    embeddings_freq = 1,
    embeddings_data = c(1:max_features)
  )
)

history <- emodel %>% fit(
  1:max_features, 1:max_features,
  epochs = 1,
  batch_size = 256,
  callbacks = callbacks
)

tensorboard("my_log_dir")

Ideally there would be some simpler/cleaner way to achieve this, but this works for now.

edit

Another user asked in a now-deleted question about adding token labels to Projector. For posterity, that is as simple as adding a value for embeddings_metadata in the callback_tensorboard. This should be a path (relative to your log_dir) to a tab delimited file with the desired labels. Details are here: https://www.tensorflow.org/guide/embedding#metadata


What is the role of $W_{ax}, W_{aa}, W_{ay}$ in forward propagation in RNN? Are they hyperparameters? Why are they needed?

Don't get hung up on the word "govern" here. $W_{ax}$, $W_{ay}$ and $W_{aa}$ are simply the weights and they play in principle the same role weights play in feed forward network (except that feedforward networks do not have $W_{aa}$):

$W_{ax}$ are the weights from your input layer to the first hidden layer (just as they are in feedforward networks)
$W_{ay}$ are the weights from your last hidden layer to the output layer (just as they are in feedforward networks)
$W_{aa}$ are the weights applied to the hidden state when it is fed from $t$ to $t+1$ (this is what you do not have in feedforward networks since they do not propagate through time)

You can also read more about this in the respective chapter of the deep learning book which, I think, provides a good explanation of RNNs.

EDIT (based on your comment):

In a feedforward network you would have $\hat{y} = g(W_{ay}\text{ }a+b_y)$ while your RNN has $\hat{y}^{<i>} = g(W_{ay}\text{ }a^{<i>}+b_y)$ (with $i$ being an index for the time step). There is really nothing new here with regards to the output layer in an RNN. In the below image you see to which connections the weights are applied in a feedforward net and in an RNN (I omitted the bias for simplicity):


What to do if training loss decreases but validation loss does not decrease?

Welcome to DataScience. This looks like a typical of scenario of overfitting: in this case your RNN is memorizing the correct answers, instead of understanding the semantics and the logic to choose the correct answers.
A typical trick to verify that is to manually mutate some labels. For instance, you can generate a fake dataset by using the same documents (or explanations you your word) and questions, but for half of the questions, label a wrong answer as correct. If you re-train your RNN on this fake dataset and achieve similar performance as on the real dataset, then we can say that your RNN is memorizing.
Note that it is not uncommon that when training a RNN, reducing model complexity (by hidden_size, number of layers or word embedding dimension) does not improve overfitting.
If it is indeed memorizing, the best practice is to collect a larger dataset. I understand that it might not be feasible, but very often data size is the key to success. If you haven't done so, you may consider to work with some benchmark dataset like SQuAD
or bAbI.


Is numpy.corrcoef() enough to find correlation?

On a side note, I don't think correlation is the correct measure of relation for you to be using, since Survived is technically a binary categorical variable.

"Correlation" measures used should depend on the type of variables being investigated:

continuous variable v continuous variable: use "traditional" correlation - e.g. Spearman's rank correlation or Pearson's linear correlation.
continuous variable v categorical variable: use an ANOVA F-test / difference of means
categorical variable v categorical variable: use Chi-square / Cramer's V


Labeling data is not always an easy task. There are occasion that the data in hand does not have label and you need to make a model using them. You have to find the similarities and differences in your input data. Clustering approaches try to find these similarities and differences to find similar data. Also they are used as a pre-processing before doing supervised classification. In cases that the input data does not have any label, employing clustering approaches can be a way to label the data and use them for training supervised models.


F1 on test set using Naive Bayes classifier decreases as training set size increases. Why?

Overfitted model is something that shows very less error on your training set and then when you test it on a test set, it fails badly. This might be due to class imbalance and you might be giving something that it has not seen or due to using more number of features than you were supposed to, making the model to fit in every datum. Wiki page is nice overview read for this.

Answering your next question, F1 score as might already be knowing has two components in it, Precision and Recall. Check whether there are reasonable number of TP,TN when compared to FN, FP and if you find those numbers to be fair enough, then F1-score is just a metric for you. There is no benchmark score for an awesome F1 score. The decrease might be just because your model has just more data to predict now.

So overfitting can lead to less F1-score but reduced F1-score does not mean overfitting always.


Why do we want the variance of the layers to remain the same throughout a deep network?

What variance is actually being taken here? Variance with respect to what?

$Var(W^i)\rightarrow$ The variance of the random initialization of the weights
$Var(x)\rightarrow$ The variance of each feature
$Var(z^i)\rightarrow$ This is the variance of the values of $z^i_k$ for each neuron $k$ of the layer $i$ which is the same for all the neurons in the layer $i$

What exactly is helped by the variance being stable in this way?

Following the notation of the article, let's compute the gradient of the cost function w.r.t. the parameters in two consecutive layers (that we are going to call: layer $i$ and layer $i+1$). In this layers, the quantity used to update their respective weight matrices, is given by:
$$\text{Layer 1}\rightarrow \frac{\partial Cost}{\partial W^i}= \frac{\partial Cost}{\partial s^i}\frac{\partial s^i}{\partial W^i}=\frac{\partial Cost}{\partial s^i} (z^{i-1})^T$$
$$\text{Layer 2}\rightarrow \frac{\partial Cost}{\partial W^{i+1}}= \frac{\partial Cost}{\partial s^{i+1}}\frac{\partial s^{i+1}}{\partial W^{i+1}}=\frac{\partial Cost}{\partial s^{i+1}} (z^{i})^T$$
There we can see that if we consider:
$$ Var\left(\frac{\partial Cost}{\partial s^i}\right) = Var\left(\frac{\partial Cost}{\partial s^{i+1}}\right) \,\,\,\,\,\leftrightarrow\,\,\,\,\, Var(z^i)=Var(z^{i-1})$$
Then we would have:
$$Var\left(\frac{\partial Cost}{\partial W^i} \right) = Var\left(\frac{\partial Cost}{\partial W^{i+1}} \right)$$
This is a good thing because having the same variance in the updates of both layers means that the updates are globally spread in the same way, so assuming that the mean value of $\partial Cost/\partial W$ in both layers is the same, then this would mean that globally these layers are learning at the same rythm.

Whatever the answer to (2) is, what is the proof or evidence that that's the case?

A good advantage of the previous reasoning is that if we could achieve this happening in the whole neural network, then all the layers in the NN would be learning at the same rythm! So problems like vanishing or exploding gradient would be avoided.


So in the example above if many of the positive reviews have the word 'positive' in them then this word will automatically have modified (and in general lower) TF-IDF scores simply because the majority of the documents in the dataset are positive (600 documents).

You're thinking of the word positive as example because you are a human and you know what to select. The words which have the highest frequency in English are stop words: the, a, is... These are useless and would add a lot of noise in your model. You can remove them using a predefined list of stop words, but what about words which are not stop words but are frequent enough, for instance food, place, chef... This is where IDF helps.


What is the difference between affinity matrix eigenvectors and graph Laplacian eigenvectors in the context of spectral clustering?

The concept is the same but you are getting confused by the type of data. Spectral Clustering as Ng et al. explain is about clustering standard data while the Laplacian matrix is a graph derived matrix used in algebraic graph theory.

So the point is that whenever you encode the similarity of your objects into a matrix, this matrix could be used for spectral clustering.

If you have standard data i.e. a sample-feature matrix, you can find the proximity or affinity or whatever you want to call it as a matrix and apply spectral clustering.

If you have a graph, this affinity would be anything like adjacency matrix, distance matrix or Laplacialn matrix and solving the eigenfunction for such a matrix gives you the corresponding result.

The point about using Laplacian rather than adjacency is to keep the so-called affinity matrix positive semi-definite (and normalized Laplacian matrix is a better choice as it gives you normalized eigenvalues between 0 and 2 and reveals the structure of graph much better).

So the long story short is that as long as you have a matrix containing the affinity of your data you can use spectral clustering in general. The difference is in details (i.g. the property of normalized Laplacian I just mentioned)


Path prediction using the historic path taken by person

I think we've covered this question before but I can't find my response... A lot of work has been done in this area. I don't know of any packages, but the basic idea is to discretize the search space. One way is to just to use the cellular grid, which makes sense if you have that information. Another is to first extract the "locations" from the location trace; i.e., places the users revisits and spends significant time in. Then you can apply well-known techniques such as HMMs to estimate the next discrete location from the recent past. Most of the papers I read on this subject are about a decade old, so they came out before the deep learning flurry. If they were revisiting the subject I'm sure I'd see papers using recurrent neural networks.

Here are some papers for inspiration:

Learning and Inferring Transportation Routines
A personal route prediction system based on trajectory data mining
A Comparison of First- and Second-Order HMMs in the Task of Predicting the Next Locations of Mobile Individuals
Using GPS to Learn Significant Locations and Predict Movement Across Multiple Users


Recommendation system with active learning

One way to approach recommendation systems that can change over time based on user input is to model them with Reinforcement Learning. There is an inherent notion of time and active choices with Reinforcement Learning.
Contextual-Bandit is a popular approach that models information about the user (context) to influence which options (arms of the bandit) are shown to a user.


Unable to understand the usage of labels argument in sklearn.metrics.f1_score

The F1 measure is a type of class-balanced accuracy measure - when there are only two classes, it's very straightforward, as there's only one possible way to compute it. With 3 classes, however, you could compute the F1 measure for classes A and B, or B and C, or C and A, or between all three of A, B and C.

It seems that the "labels" parameter is telling the method which classes to compute your measure over. Since F1 is already class-balanced, you probably want to include all three labels for your measure. This parameter is likely more important for imbalance-insensitive measures like raw accuracy, as it will allow you to compute accuracy in a subset of the data - in the documentation example, they use it to exclude a majority class, allowing the user to evaluate the accuracy in the minority only. If you have a huge imbalance, say 99% of your data is of one class, your accuracy measure will be completely dominated by accuracy within that class - for this reason, it might be more informative to see how well the classifier does in the 1% only.


I'm assuming the displayed time series shows number of jobs submitted per 15 minute interval.

Categorical features

Divide the time series per category. If the jobs can be divided into type1, type2, type3 then make a time series for each type and predict each series individually. So type1-time series has number of type1-jobs per 15 minute interval.

Continuous features

For continues features e.g time-to-do-job you can divide the jobs into categories of time00,time10, time20, time30 for jobs that take 0-9 minutes, 10-19 minutes, 20-29 minutes etc respectively. As before generate a time-series per division.

Depending on how much data you have and how it is distributed you can make more groups or space them differently.


There are general rules of thumb for different models, both in terms of the number of samples required as well as the number of features. I will try to just put some concrete numbers on things, with the caveat that is is indeed problem specific, and you usually have to make compromises.

If you are looking at something like a simple linear regression e.g. with 5 features, then for the basic tests to be considered statistically relevant, you need at least 40 samples.

In your case with 500,000 samples, 50 features is definitely acceptable - you could even have more. There are, however, two things I would be wary of:

Firstly, you make life hard for your model is many of the features are really describing the same thing. The technical term is multicollinearity, which, if your data has it, means that your features themselves are highly (linearly) correlated. It is hard for the model to know which feature to rely on, and your results may be improved by simply removing one. A simple example: if I predict your shoe size based on your height, and I have two features: your height in inches and your height in centimetres, you can see that I only really need one of these, as they are perfectly correlated measures.

Secondly, The more the better is theoretically true, as long as you have a model which is able to decide when to include features and when to ignore them - and that your data is good enough to facilitate this. There are ideas like Occam's razor, saying that a simple model should be selected over a complex model with the same performance; i.e. you should also consider interpretability. There are measures defined that quantify this, such as the Akaike Information criterion. It is a model selection tool - so you could fit the model using 10, 20, 30, and 50 of your features, compare the results of this creiterion and pick the model which scores best. Here is a recent question which touches on that topic.

There are functions that will remove features, based on a correlation threshold with other features in the dataset. It is explained well in the Caret package, for R, but the same thing can be done in Python fairly easily (not sure if there is a standard implementation).
[EDIT] - the Python equivalent looks to be the VarianceThreshold class.


Document classification using convolutional neural network

You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. Doc2vec is one way to do this (each sentence would be a "document").

If you don't want to use Doc2vec, one way to create the sentence vectors would be to average the word vectors for each sentence, giving you a single vector of the same width for each sentence. This may not be as precise as some methods available through Doc2Vec but I have used it with considerable success for topic modeling.

Either way once you have your sentence vectors, line them up in sequence for each document like you're already doing for your word vectors and run then through your model. Because the sequence length for each document is shorter, your model should train more quickly than with word vectors.

By the way, this method could work when scaled up or down to meet your accuracy and speed needs. (e.g. if your CNN still trains too slowly with sentence vectors, you can create paragraph vectors instead).

One way to handle documents of different length is through padding. Your document sequences should all be equal in length to your longest document. So if your longest document is 400 sentences then all document sequences will be 400 vectors in length. Documents shorter than the max length would be padded with vectors filled with zeros.


How do I determine the best statistical way for data transformation for standardization (like log, sq root) to remove bias between different datasets?

Its unclear what the OP is asking (so this response is somewhat general), but the table below illustrates common contexts and the transformations that are typical:

sales, revenue, income, price --> log(x)

distance --> 1/x, 1/x^2, log(x)

market share, preference share --> (e^x)/(1+e^x)

right-tailed dist --> sqrt(x), log(x) caution log(x<=0)

left-tailed dist --> x^2

You can also use John Tukey's three-point method as discussed in this post.  When specific transformations don't work, use Box-Cox transformation. Use package car to lambda <- coef(powerTransform()) to compute lambda and then call bcPower() to transform. Consider Box-Cox transformations on all variables with skewed distributions before computing correlations or creating scatterplots.


Is an artificial neural network a kind of metaheuristic?

I suppose one could possibly argue it either way, but at the end of the day, it doesn't really matter whether we classify an ANN as a metaheuristic.  It sounds like you have one reader who found it off-putting or confusing to have you compare ANN vs metaheuristic because they would have characterized an ANN as a metaheuristic.  So, I suggest asking yourself three questions: (1) How many other readers will have a similar reaction?  (2) What changes can I make to the paper that would avoid that reaction?  (3) Is there any downside to those changes?  Are they worth it?

In this case, it seems easy enough to rephrase your description of the comparison, so you don't trigger this reaction.  You could be more specific about what comparison you are doing: you compare an ANN to a genetic algorithm.  That might avoid the problem, with no downside.  Being specific is often a good thing.


Very low probability in naive Bayes classifier 1

Each column of binary (Y) is a feature. The Bernoulli naive Bayes classifier could identify the class (X) where the number of features (Y) was less than 17. The real data had more features than that. I found that another method could classify it accurately. That was:

Trainining:

(1) Count which features (Y) are in each class (X) in the training data

Testing:

(2) Give each row a score (Z) with a starting value of 0.5

(3) For each row:

If each feature (Y) is in the class (X) in the training data then add 1 to the score (Z).
If each feature (Y) is not in the class (X) in the training data then subtract 1 from the score (Z).
If the class (X) is not in the training data then don't do anything

The score (Z) was a good classifier for my data.


Sparse matrix in R based on the data frame

Your data looks like this at the moment:

data <- data.frame(User_Id = c(276725, 276726, 276725, 276726, 276725),
                   ISBN = c("A", "B", "C", "A", "B"),
                   Book_Rating = c(0, 5, 7, 5, 3))

> data

  User_Id ISBN Book_Rating
1  276725    A           0
2  276726    B           5
3  276725    C           7
4  276726    A           5
5  276725    B           3

With the following commands you can create a dgCMatrix, required as input for a realRatingMatrix object.

library(Matrix)
data_sparse = sparseMatrix(as.integer(data$User_Id), as.integer(data$ISBN), x = data$Book_Rating)
colnames(data_sparse) = levels(data$ISBN)
rownames(data_sparse) = levels(data$User_Id)

> data_sparse

276726 x 3 sparse Matrix of class "dgCMatrix"
     A B C
[1,] . . .
[2,] . . .
[3,] . . .

..............................
........suppressing rows in show(); maybe adjust 'options(max.print= *, width = *)'
..............................

[276722,] . . .
[276723,] . . .
[276724,] . . .
[276725,] 0 3 7
[276726,] 5 5 .

After that you can call the command

new("realRatingMatrix", data = data_sparse)

Bear in mind that this matrix starts at one and it's only populated in two rows (276725 and 276726) but the rest of the columns from 1 to 276725 exist. If you don't want to use User_Id as indices you will have to create new indices and use those instead and have one User_Id correspond to a new index. That means if User_Id starts at, for instance, [11, 18...] then it'd become User_Id 11 = New_Index 1, User_Id 18 = New_Index = 2. Use New_Index instead of User_Id and you're done.

If that's the case then data should end up looking like this:

data <- data.frame(User_Id = as.factor(c(276725, 276726, 276725, 276726, 276725)),
                   ISBN = c("A", "B", "C", "A", "B"),
                   Book_Rating = c(0, 5, 7, 5, 3),
                   New_Index = c(1, 2, 1, 2, 1))

> data

  User_Id ISBN Book_Rating New_Index
1  276725    A           0         1
2  276726    B           5         2
3  276725    C           7         1
4  276726    A           5         2
5  276725    B           3         1

library(Matrix)
data_sparse = sparseMatrix(as.integer(data$New_Index), as.integer(data$ISBN), x = data$Book_Rating)
colnames(data_sparse) = levels(data$ISBN)
rownames(data_sparse) = levels(data$User_Id)

> data_sparse

2 x 3 sparse Matrix of class "dgCMatrix"
       A B C
276725 0 3 7
276726 5 5 .


How to match a word from column and compare with other column in pandas dataframe

If you know the rules, then you can code them up as you would usually do.
For e.g.
df['Keywords'] == "roll-on" & (df['Type'] == "ball" | df['Type'] == "others") => df['Result'] = True
If you have too many rules and associations to manually code, you could consider using a look-up table + merge procedure. Here you would store all the rules and associations as a table (i.e. this would include Keywords, Type, and Result), and merge Result back on to your original dataframe (by Keywords and Type).


For what condition boosting work better than bagging in Ensemble Learner?

Bagging (and features sampling) aim to reduce variance by providing low-correlated trees. Estimators can then be aggregated together to reduce variance. Reason is simple decision trees tend to quickly overfit in the bottom nodes. Bagging and features sampling only adress high-variance (overfitting) problems.

On the other hand, boosting is a meta-algorithm that aims to reduce both bias and variance by training estimators in a sequential way. Each predictor tries to reduce residuals from previous estimators.

In my opinion, the two meta-algorithms can't be easily compared that way. They will give you different results depending of data distributions, potential outliers, usecase... Potentially, boosting is more sensitive to outliers than bagging for example. I think trying the 2 methods is your best chance and you don't have to restrict to only one model.


Which classifier should I use for sparse boolean features?

Mapping your inputs into a sparse matrix and using logistic regression with an $\ell_1$ penalty should work well. In R this is very straight forward with the glmnet and matrix libraries.

Additionally, if you have boolean features with a low frequency, you can filter some of them by taking a subset with at least some number of observations.

Here's a quick simulation in R of a lasso logistic regression with sparse features.

library(glmnet)
library(Matrix)

n <- 1e5
set.seed(420)

# These are all converted to boolean variables since they're arbitrary categories
ls <- data.frame(cat1=sample(letters, n, replace=TRUE),
                 cat2=sample(letters, n, replace=TRUE),
                 cat3=sample(letters, n, replace=TRUE))
ls$cat1cat2 <- factor(paste(ls$cat1, ls$cat2, sep="-"))
xs <- sparse.model.matrix(~.-1,data=ls)
betas<- rnorm(dim(xs)[2])
error <- runif(n)

ytrue <- xs %*%betas
ylogit <- 1/(1 + exp(- (ytrue + error))
ylabel <- ifelse(ylogit>0.5,1,0)

trainfilter <- runif(n)<=0.3
sparsemodel <- cv.glmnet(x=xs[trainfilter,],
                         y=ylabel[trainfilter],
                         alpha=1,
                         nfolds=3,
                         family='binomial')

preds <- predict(sparsemodel, xs, s='lambda.min', type='class')
print(table(ylabel, preds))

The classifier doesn't matter as much in this case as the underlying algorithm's ability to handle sparse matrices.

It's also worth noting that you can handle sparse boolean features by mapping them to arbitrary numeric indices and running them through a tree based algorithm (e.g., Random Forest or Gradient Boosting). In practice, this seems to work well.


R Rpart taking too much time on data set

Have you tried using randomForest instead of rpart? For example, let's assume you have two data.frames:

train_data and test_data

In my example, the last column is the class (and is a factor variable), and all other variables (all other columns) are numeric. For example, one training example might look like:

4 5 10 12 1 0

where 0 is the class label. You can do the following to run a random forest with 50 trees:

library(randomForest)
model_pl_1 <- randomForest(train_data[,-ncol(train_data)], train_data[,ncol(train_data)], test_data[,-ncol(test_data)], test_data[,ncol(test_data)], ntree=50)


How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

Thanks for the clarification. You can configure the parameters once or twice at a time by re-instantiating the RSCV object each time, passing different parameter dictionaries each time. For example:

SEED = 4
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
    }

model = lightgbm.LGBMClassifier()

RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

    parameters = {
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
    }

RSCV = RandomizedSearchCV(RSCV.best_estimator_,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)

By passing the RSCV.best_estimator_ instead of model to the second time, it will automatically use the best values for num_leaves and min_child_weight that it identified in the first run and effectively "freeze" those as they are, finding the best combination of feature_fraction, bagging_fraction and min_data_in_leaf under those constraints.

My understanding, however, is that the best approach is to do as you are currently doing and simply include all the parameters in one search. I generally set n_estimators to some lowish value (200 or so) and learning_rate to some reasonably high value whilst doing the tuning, and then beef them up later. This will generally drastically reduce the time taken to perform the tuning. It does necessitate a refit afterwards to take account of the new values.

model = lightgbm.LGMBClassifier(n_estimators=100, learning_rate=0.2

params = {
    'num_leaves':np.arange(100, 500 ,100)
    ...
}

RSCV.fit(train, label)

model = RSCV.best_estimator_
model.set_params(n_estimators=1000, learning_rate=0.001)
model.fit(train, label)

The literature on Randomised Search is usually quoted as being this paper, which I recommend reading: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf. One key takeaway is the conclusion that roughly 60 trials are needed to find the optimal parameter space with a reasonably high probability, so I'd recommend increasing the n_iter parameter of RSCV to 60.

Finally, the use of np.arange() results in each of your lists being uniformly distributed. For example, the tuning algorithm will select values for feature_fraction between 0.1 and 0.4 with equal likelihood. You can also use Scipy's distribution functions to define other distributions; normal and log-normal for example that targets the search on a particular area (for example, you could make the optimiser more likely to try values closer to 0 than values close to 1 for a the feature_fraction parameter.

Hope all that helps.


Creating trial number ascending list, for separate subjects in long format data frame, R?

If I understand your question correctly, and assuming that your initial data frame looks something like trial_df below, you can use the following code and the dplyr package.

library(dplyr)

# create data frame
trial_df <- tibble('subject' = c(rep('101', 3), rep('102', 3)),
                  'result' = c(32, 33, 64, 12, 18, 14))

# group by subject and add a row number
# assumes your data frame is ordered by trial number for each subject
new_df <- trial_df %>%
  group_by(subject) %>%
  mutate(trial_number = row_number())


Finding a vector that minimize the MSE of its linear combination

If inputs are the same and you expect multiple outputs, I would recommend you to look at multioutput models which can be of two types :

native multioutput algorithms
multiple single output regressors wrapped together. Sklearn’s multioutput does that.


Validation Loss is not decreasing - Regression model

the network architecture above is a very strange choice. When you have only 6 input features, it is weird to have so much Dense layers stacked.
if network is overfitting, WHERE IS DROPOUT? Why not trying some regularizers, if the latter does not help?
+1 for David Waterworth - correlation/causal analysis is not everything yet. Does linear regression provide better R-square values?
what is output(target) variable range? Maybe it should be mapped/scaled to something reasonable? (I judge from loss values).
activation function and initializers are important too. Try using different values, rather than relu/linear and 'normal' initializer.

EDIT: yes, this should be enough data, if your data has only 6 inputs. However, you can try augmenting data too, if it makes sense and you can make reasonable assumptions in your case - sometimes it gives difference in the long run, even if in the beginning you think it does not work.

EDIT2: with specific datasets, neural network can get into local plateau (not minima however), where it does not escape. To test this hypothesis,  you can set learning rate to small value and all initializers to generate small values too - then network may not go to this plateau suddenly, but goes to global minima instead.

EDIT3: increasing batch size leads to faster but poorer convergence on certain datasets. On a smaller network, batch size = 1 sometimes makes wonders.


I will outline some points about the libraries and point you to some good comparisons that I have read. The GitHub star counts are just another reference point to help compare popularity. While I don't condone simply following the masses, those stars do help you see what a lot of other people think about the frameworks.

Tensorflow

is very well documented
scales up into production, being able to use many GPUs or Google TPUs
allows flexible creation of DL architectures, using basic building blocks
is backed by Google
has a very large following - just look at how many stars it has received over on GitHub!

For reference, the awesome NumPy only has 7059 stars!

CNTK

is gaining popularity (they started a bit later than Google)
has very efficient implementations for specific use cases, such as the usage of CNNs and LSTMs in the text domain
is nicely linked to other Micosoft products, like their Azure based toolkits
scales well in production
is backed by Microsoft
has a good following on Github:

Theano

While still open source (and so able to be further developed by the community), the team behind Theano announced that they will no longer actively develop it. This means it will now likely fall even further behind other leading frameworks, and new functionality coming from ongoing research are not likely to make it into the library.

We can see that many people admire Theano, but given that it is basically the oldest DL framework, the star count tells a tale:

Additional points

Have a look for some overviews and comparisons of the deep learning frameworks in general to get a better understanding of when you might use one over another. Here are some I found useful:

Comparing top deep learning frameworks
Battle of the deep learning frameworks
Choosing a machine learning framework   - has a good conclusion

I would suggest reading about the difference between static and dynamic computational graphs. Tensorflow e.g. builds and compiles a static graph of your model and then pushes data through. PyTorch on the other hand allows you to create your model dynamically, giving more freedom during the development of new architectures.

A common workflow would be to develop and do research with PyTorch, then try to write the final production code in Tensorflow for deployment.

One last point - you should be aware of the library Keras. This is a wrapper around the base libraries, such as Tensorflow, Theano and CNTK - maybe more in the future). It is highger-level and easier to use that the others, but behind the curtains it is really just using one of the libraries you are asking about. It is just change on flag in a config file to swap between them!

Just for completeness, the Keras GitHub star banner:

[All gitHub star counts as of April 2018]

I realise some of these points may become outdated and untrue over time. In which case, feel free to leave a comment and I can revise my post.


Your histogram is valid, but it has too many bins to be useful.

If you want a number of equally spaced bins, you can simply pass that number through the bins argument of plt.hist, e.g.:

plt.hist(data, bins=10)

If you want your bins to have specific edges, you can pass these as a list to bins:

plt.hist(data, bins=[0, 5, 10, 15, 20, 25, 30, 35, 40, 60, 100])

Finally, you can also specify a method to calculate the bin edges automatically, such as auto (available methods are specified in the documentation of numpy.histogram_bin_edges):

plt.hist(data, bins='auto')

Complete code sample

import matplotlib.pyplot as plt
import numpy as np

# fix the random state for reproducibility
np.random.seed(19680801);

# sum of 2 normal distributions
n = 500;
data = 10 * np.random.randn(n) + 20 * np.random.randn(n) + 20;

# plot histograms with various bins
fig, axs = plt.subplots(1, 3, sharey=True, tight_layout=True, figsize=(9,3));
axs[0].hist(data, bins=10);
axs[1].hist(data, bins=[0, 5, 10, 15, 20, 25, 30, 35, 40, 60, 100]);
axs[2].hist(data, bins='auto');


Reason for Huge Jump in Loss For One Epoch Only?

I cannot say for sure, but it seems that it might be an error in the printout. The validation loss doesn't seem to spike, so it seems that the loss may not actually be what is printed.

I can only advise to run training multiple times and see if this happens again. If yes, try toying with the learning rate. There is a small chance that the learning rate could be too high making the loss diverge for one epoch.

Based on your code, it seems that your optimizer clips gradient with an absolute value larger than 1. In a scenario where most of your gradients are larger, this could lead to your model being optimized in the "wrong direction". This could lead to the loss being so large for some epochs.


I will start with a toy example for the convergence part. Suppose that the loss function is $f(x) = x^4$ and we want to minimize it using the gradient descent. Clearly, the minimum is attained at zero and, in general, we would like the magnitude of the current approximation to decrease.
The update rule of gradient descent is
$$ x_{k+1} = x_k - \lambda \nabla f = x_k - \lambda \cdot 4x_k^3.$$
Simplifying the expression, we get
$$x^{k+1} = x^k(1 - 4\lambda x_k^2).$$
And now the combo initialization + learning rate start to appear. If $|1-4\lambda x_0^2| < 1$, then $|x_0| > |x_1| > |x_2| > ...$. The sequence will go to zero eventually. If $|1-4\lambda x_0^2| > 1$, then $|x_0| < |x_1|$. In this case, $|x_1| < |x_2|$ and so on -- the sequence will grow. Therefore, if the learning rate $\lambda$ if fixed, then the initial values $x^0$ determine if the grad descent converges or not.
When the gradient descent converge?
The math says that the gradient descent converge when the learning rate $\lambda$ and the gradient $\nabla f$ satisfy
$$\|\lambda \nabla f\| < 1$$
along the optimization path. Note that this condition does not need to be valid for all values of $x$, but at every $x_0, x_1, x_2, ...$
For many "good" functions, it suffices to require $\|\lambda \nabla f\| < 1$ only at $x_0$. The reason is that after the first iteration, we are closer to the local minimum, and for many "good" functions, it means that the gradient will be smaller.
What about weight choice in [-0.5,0.5] and [-1, 1]?
I think of it as follows: Suppose that we selected weights in $[-0.5, 0.5]$ (model 1) and then multiplied all weights by 2 to get uniform distribution in $[-1, 1]$ (model 2). Suppose that the learning rate is identical in both cases, and let's check how the SGD performs. For simplicity of the argument, I replace it with the gradient descent.
How does it transfer to the NN?
Note that a linear map (say, for the dense layer) has the following property. Suppose that all weights W are multiplied by 2, then
$$\|2W\| = 2\|W\|.$$
ReLU is quazi-linearly-scalable: for every $a > 0$,
$$a\cdot\text{ReLU}(x)=\text{ReLu}(ax).$$
Note that if your NN has $d$ layers, then, multiplying all weights by 2, your output increases in $2^d$ times (factor of 2 for each layer).
It is hard to compute the gradient for the NN, but using the product rule, I expect it to increase in approximately $2^{d}$ times. If model 1 satisfies the conditions for the gradient descent convergence, then we need to decrease the learning rate about $2^{d}$ times to guarantee the gradient descent convergence condition.


Is providing class weight to neural network enough for imbalanced binary classification?

Referring to an answer to a similar question, you don't have any reason to handle unbalance from the beginning. An imbalance of 95:5 isn't that big, I'd start with the regular training and if that doesn't work try more sophisticated things.


Converting a regression problem into a classification problem

I think what you should be looking for is ordinal regression/classification in order to utilize the ordering of the classes.

Example python/matlab implementations, with some extra resources.

For fast integration with your system, you could create a new ordinal regression model using the outputs of the already trained network as input to the new ordinal model and the true ordinal value as the output.

For another quick/dirty solution you could try simple Mean Absolute Error or Mean Squared Error, according to this study, between the predicted and target classes of your system, as it is right now, instead of the categorical cross-entropy error function.


Performace of Fischer projection as dimension reduction compared to other LDA methods

I didn't get what you meant by "other LDA Methods". To the best of my knowledge, Fisher method is just one.

Sklearn has the implementation and you can find it here.

Is it good or bad? When it comes to dimensionality reduction you don't know as dimensionality reduction methods are usually unsupervised. So if PCA works better? No one knows. At least in the context of two-class classification, Fisher method is still brilliant.

Hope it helped!


Detecting spammers with artificially generated target class

Machine learning is the process of automatically discovering (inductively) the formula/rule that models a certain random variable. If you can discover the rule by yourself, you don't need the machine to aid you.

You (or your boss), as the domain expert, has written a rule to model the target class. As long as this rule is deemed good enough for the business, there's no need to spend resources to generate another model.


Why is count encoding effective in improving accuracy?

Its a good question,
I would just like to add my points
Lets assume you have dataset with features (patient: id,execercise_duration: int, fav_products: category) target(diabetes: Binary)
Label encoding will just give numbers to every unique category.
Lets assume Category A is ice-cream and Category B is juice and Category C is chocolates.
Now if Category A is encoded 1 and Category B is encoded 2  and Category C is 3 but you keep the encoded feature as numerical series then it would simply mean Category C > Category B > Category A (since 3> 2 > 1). But is it the right information to send to model ?
I guess not.
Intution says people with fav_products as ice-cream and chocoloate will be diabetic.
Category A and B and C just represent three different things nothing is large or small in them.
But if you send frequency or count then lets say more observation in data are of ice-cream, chocoloates and less are of juice. Becuase usually icecream and chocolate are more desirable food than juice. Frequency or count of ice-cream and chocolate will be more than juice.
So keeping frequency or count encoded feature as numerical can give information to model that when this encoded feature value is high outcome is diabetes and when it is low outcome is non-diabetic.
Note: A more complex model like decison tree may be able to give good accuracy even with label encoding atleast for this simple example.


The point of a test set is to give you a final, unbiased performance measure of your entire model building process. This includes all modelling decisions in your pipeline, so any preprocessing, algorithm selection, feature engineering, feature selection, hyper parameter tuning and how you trained your model in general (5 fold? Bootstrapping? etc.). All of these decisions can lead to overfitting; for instance, selecting a set of hyperparameters that are coincidentally optimal for a particular validation set but not for the general population. If we have no test set you would not be able to identify this and would potentially be reporting highly optimistic scores.

Also, because the above modelling pipeline can get very complex, the possibility of leaking data and overfitting becomes very high. If you tune to your validation set, how will you know if your entire modelling process is not leaking data (and therefore overfitting?)

You bring up a good point; of course if we see that the test set score is poor then we will probably go back and tweak again. Thus, this just demotes the test set into a validation one if you use it too many times as you now run into the possibility of overfitting the test set (see almost every Kaggle competition). However, through repeated test set evaluation (train the model, then test it, then repeat with a different partioning) you will at least get a gauge on how variable your model is to help mitigate this problem. The amount of times you repeat will depend on how much the test set scores vary and how much uncertainty you are willing to accept (also time constraints).

In my opinion, in the business setting you should always make time to properly test your model. The dangers of overfitting are way too high and even worse; you would not even know it. If the test set scores end up being "trash" then at least you know the model is trash and you don't use it and/or you change your approach. This is way better than thinking the model is fantastic based off non rigorous validation and then having the model fail in production. The scientific method is there for a reason right?


how can i interpret kernel density plots from classification?

In both graph it show that the model will not perform very well on the classification task as the probability distribution of the model overlaps significantly. A good model will have almost seperated curve for each class. Adding more feature will help the model differentiate between curves.


What you seem to want is to identify meaningful discrepancies between your blog posts in order to identify what category they fit in.

I think it's important for you to consider the classification problem in terms of human thinking:

How would humans decide which blog gets which tags?
Can we distil this decision-making into meaningful features? In other words, does this decision-making depend on (linguistic) phenomena that can be found in the text? (it probably does)
(How) can we extract these features from the text?

Also, it might be the case that either the tag set you propose or the data set you have are too similar. This means that the tags might be too similar and that their occurrence depends on many similar feature values which makes it difficult for your training algorithm to extract discriminative features. The same thing happens when the data expresses too little variety.

On a final note: This might well be the first attempt to solving this particular problem (on this data set and this set of labels), so the number you get is not necessarily bad or sub-par, because it might simply be a difficult task.


Confusion regarding the Working mechanism of Activation function

I think following picture by Rubens Zimbres can express how this process will be done except loss function part:

  Is there a binary step function in the output layer which squashes the values of the linear combination of weights and and inputs to 0 or 1?

Answer : Yes if you consider following example shows how Sigmoid squishes values to be between 0 and 1:

  Does classification means always applying a thresholding function on top of a linear or non-linear function which is in the hidden layer?

Answer: It does.

  Say the predicted output value is 0.75 and actual Y is 0. Then, how is 0.75 converted to 1? The loss function would calculate the error as actual - predicted = 0-0.75 = -0.75

Answer: Loss function and calculation of error for classification-based NN is not just by computation of  $Y_Actual - Y_prediction$

but It is used normally by combination of Softmax activation function and cross-entropy as follow:

See the whole training process Multinomial Logistic Classification $D(S(wx+b),L)$


Normalizing dependent feature by one of the independent ones

I think it's complete alright. In fact, the second model mathematical expression is given by
y=x3f(x1, x2, x3),
which is just like the first model but with some specific feature engineering. I don't see any possibility for data leakage.


How to know when to treat a problem as a classification task or a regression task?

In effect it is ordinal regression/classification. I suggest you to go for Mean Absolute Error to take into account the missclassification that play a bigger role when far from the ground truth :

$MAE = \frac{\sum_{i=1}^n|h(x_i) - y_i|}{n}$, creating your own scoring function in python.

Given the number of different outputs you have, I recommend you to go for regression. Anyway, rounding any hypothesis output would only play on the score, it's up to you to decide if it makes more sense to see wine quality scores as classes or not.


It seems that Conda can only install one version of a package at a time,thus whatever installing command you use it will replace whatever other build you might have installed. So I reinstalled it using conda create -n FERRET -c conda-forge pyferret ferret_datasets --yes. After that I was able to activate the environment with conda activate FERRET If the terminal complains and asks you to initialize a shell, conda init bash should suffice to allow you to activate the environment.


With sklearn's OneHotEncoder, the categories are baked in after fitting. You can apply the encoding to new data with region_ohe.transform(x_new).  (And, as you might guess,  fit_transform just calls fit then transform.)


What algorithm should I use to find which sub-component is associated with a attribute?

The type of clustering depends on the measures scale of the data. You stated you have 5 attribute values. Are those values nominal, ordinal, interval, or ratio?
If they are nominal, correspondence analysis is a common option.
If they are ordinal, k-medoids clustering could work.
If they are interval or ratio, k-means clustering is a very popular choice.


Applying an algorithm on it's own training data for descriptive purposes?

can you apply a trained algorithm / model on its own training data if my interest is not in predicting forward but evaluating success vs likelihood of success (and still yield useful information)?

Yes, absolutely. There are quite a few cases where applying a model on the training data is useful. The most common is probably to detect overfitting: a high difference in performance between the training and test set is a sign of overfitting. In general it can also be useful to know how the model performs on the training set in order to obtain an upper baseline for the performance.
Everybody says "don't predict on the training set" simply because it's a simple rule to remember and such an easy mistake to make for beginners. But as long as one understands what they are doing and knows that obviously the predictions obtained on the training set are biased, there's no problem.
The task described in the question makes sense to me, you have my permission to predict on the training set ;)
Just one more remark: there might be a bias in the training data itself, in the sense that if success for an event is very unlikely then the event will fail more often than suceed even when the condition of its success are satisfied. If the data contains this kind of cases, it means that the model is trained to predict "fail" for potentially successful features. Whether or not this impacts the model depends on whether there are enough similar but successful events in the data, I think.


Can pytorch / keras support dataloader object of Image and Text?

I can only guess what you are trying to achieve, but I think this might be helpful even though it is a different approach. The idea is to create two separate models for text and image and merge them into third.


word2vec - log in the objective softmax function

No, the logartihm doesn't disappear. From the equation
 ,
When you want to calculate
 ,
it essentially means calculating ,
Now ,
So ,

as .


Convolutional layer dropout layer in keras

Let's clarify few things about dropout. And Neil Slater is to credit for this answer since his comments helped formulate a more clear explanation.

First of all, dropout is a regularization method, it is usually only applied during training (although it can be used in prediction as an approximation to a Bayesian Neural Network as is explained by Yarin Gal's paper). As you might have understood, its goal is to limit overfitting. That is, to train a model that will generalize better to newly unseen data samples. So it has nothing to do with testing but everything to do with training.

Second, the reason why you might have seen the output multiplied by $p$ at prediction time is a trick used with the very basic implementation of dropout referred to as vanilla dropout. At prediction time (or testing time if you prefer this wording) no need to drop anymore, but it is needed to scale the outputs by $p$. The reason is that because at training time, dropout was perform with a probability $p$, then the output need to be scaled to adjust for $p$ at prediction time.

Third, inverted dropout (which is the dropout implementation used in all serious DL libraries) does not need to scale the output at prediction time because scaling by $p$ is actually already performed at training time (by dividing by $p$). Therefore, no need for any trick at prediction time!

Finally, the concept is nicely explained in these videos from the Udacity Deep Learning course or this course from Stanford.

I hope this is clear enough. :)


Is there any work done on reconfigurable convolutional neural networks?

It's a great question and one I have thought about a lot too.

Are you are talking about a network structure that evolves/mutates during training, or one that simply chooses whether or not to use all available resources? While I would never give a straight-up No to such a questions (especially the former), I can't say I have really seen it in publications. The cloest thing that comes to mind is a recent paper from DeepMind, which utilises a combination of "model-free and model-based aspects"; however, this more high-level than network architectures specifically, as you refer to them.

Perhaps you are considering a model, which is forced to choose between one resource and another? An example of the latter would be whether or not to use a skip connection or a dense layer (assuming shapes all match). Yann LeCun says (the video linked below) that such paths in a network not only provide more flexibility/complexity, but additionally ease the optimisation problem and provide a source of regularisation.

In any case, I am sure people are doing that, making use of dynamic networks created using PyTorch, the control flow operators in Tensorflow, the new tensorflow eager library (which aims to replicate dynamic networks like PyTorch produces).

One could also argue that providing a monolithic network to a model affords it the flexibility to use the resources (i.e. the weights) that it chooses. It doesn't have to use them all. Just as batch normslisation layers allow the model to parametrically adjust the extent to which they are used.

There are great discussions on what is required in the long run, heading toward Artificial General Intelligence (a recent overview in the context of RL part1 & part2. If we should be training models end-to-end, how much information should we give an algorithm learning to play a game and

The tools that would allow people to do that, like you mention, are coming along, but I haven't yet seen a paper that explicitly outlines how that is done. It would be great is someone else gives you a more satisfying answer! ;-)


Adjusting weights in an convolutional neural network

For learning kernel/filter matrix in convolution layer, we find partial derivative of loss w.r.t. filter matrix and use gradient descent method to update filters.
$$ W = W - \alpha\frac{\partial L}{\partial W} $$

Convolutional Neural Networks also use back-propagation algorithm to find partial derivatives of loss w.r.t. filter matrix.


What's the proper Word2vec model to get pre-trained word embedding for a classification task?

I think this article gives general idea of pros / cons between CBOW and Skip Gram,

Link

  According to Mikolov:

  Skip-gram: works well with small amount of the training data,
  represents well even rare words or phrases.

  CBOW: several times faster
  to train than the skip-gram, slightly better accuracy for the frequent
  words.

So without looking at data, assuming that you have large amount of data, CBOW might be my bet. However, if you have labels for "deceptive" and "truthful", it will be best to train both models, and use whichever outperforms via cross-validation.


How to put the variable names of pandas data frame on Diagonal of Seaborn PairGrid?

With minimal changes, I got this to work (python3.7.3, SeaBorn 0.9.0):

import matplotlib.pyplot as plt
import seaborn as sns
iris = sns.load_dataset('iris')

# Add this before your call to map_diag
next_iris_label = iter(iris).__next__

def diagfunc(x, **kws):
  ax = plt.gca()
  #  replace x.name with `next_iris_label` call
  ax.annotate(next_iris_label(), xy=(0.05, 0.9), xycoords=ax.transAxes)

sns.PairGrid(iris).map_diag(diagfunc)

Output:

I dug into it and understood that the reason this fails is because once the IRIS dataset has been passed to SeaBorn.PairGrid, the original Names of observation columns are not retained. map_diag ends up calling diagfunc with diagonal components of the PairGrid, which do not have name property, let alone name anywhere else in the arguments passed.
Here is what is passed to the first time diagfunc is called:

{
    #  this is what you passed as positional parameter
    #  As you'll notice `x` is just an array, and
    #   does not have any additional properties like a label
    'x': array([5.1, 4.9, , ... 5.9]),
    #  these get passed in as **kws
    'label': '_nolegend_',
    'color': (0.12156862745098039, 0.4666666666666667, 0.7058823529411765)
}

Now, if we could somehow iterate over all labels in IRIS dataset, we could put each one against the diagnoals of the PairGrid. Since diagfunc is called each time and has a new scope, I found that having an iterator that returned a new label each time it was called solves that.

next_iris_label defined as

next_iris_label = iter(iris).__next__

solves that problem. This only has one constraint that I dont like, you will need to reset this by creating a new instance of the iterator by the binding statement next_iris_label = iter(iris).__next__ when you want to do such labelling again(otherwise it would throw a StopIteration error.)


Scikit-learn OneHotEncoder effect on feature selection

Yes it would be possible that it happens. It means that this event happening has no importance for the target.

Imagine a categorical feature with a lot of categories(high cardinality). Maybe only one of them does not have any influence in the target so if you do feature selection this feature might have a high chance of getting dropped. So this is normal thing, just make sure that it doesnt contain any information at any point of the dataset.


Reduce drug spend using medical claims analytics

Things depend upon the dataset we have on hand here. See what observations you can grab from them using combination of different features.

If i were to approach this problem, first i would find what kind of patients go to their physicians (Age, area of living, insurance plan, previous diseases, previous admissions, income zone etc). Then would analyse which outpatients facilities serves in which area, their average charges & other features you can find.

After the above analysis i would go with prediction algorithm so to make a theory that an particular set of groups will visit that facility. And using that deduction along with clustering (what i have in mind is 2 clusters 1 going to physicians & other with outpatient facilities) i will select the overlapping clients (I mean the ones who have features similar to lower cost going site but prefer higher cost sites) of insurance company to move towards lower cost site.

Reason behind using predictive model is because it will help you in predicting the future customers, just normal clustering will just work with the current data. So to make it effective for future i would go with predictive model -> clustering model.


To answer the last question, suppose that you have a binary classification problem. It is customary to label the class as positive if the output of the Sigmoid is more than 0.5 and negative if it's less than 0.5. For increasing recall rate you can change this threshold to a value less than 0.5, e.g. 0.2. For tasks which you may want a better precision you can increase the threshold to bigger value than 0.5.

About the first part of your question, it highly depends on your data and its feature space. There are problems which the data is linearly separable in higher dimensions which means you can easily employ just a single neuron for classifying the data by a single hyper-plane. If it has happened that you have such a good accuracy you can not say anything unless you try to find the value of cross validation error. By interpreting the difference between the value of training data, and cross-validation or maybe test data, you can figure out whether your classifier performs well or not.


Logistic Regression expects labelled y_train, you do not need to do OneHotEncoding.

  y : array-like, shape (n_samples,)

Above is from sklearn LogisticRegression.fit.

I think as long as it is label encoded(using LabelEncoder), basically this tool will automatically generate label mapping from your categorical values to unique integers between 0 to n-1, where n represent count of distinct value of the category. LogisticRegresion will work fine with label encoded target value.

Protip : Please use labelencoder or OneHotEncoder to make mapping of categorical values. The problem with get_dummies is that it relies on , suppose you are doing transformation on different separated dataset(e.g. test set contained in another dataframe), suppose that on trainset you have 3 unique categorical values but on test set there are only 2 unique values. Then pd.get_dummies will only give you 2 columns (other than 3) when you are perform it on the test set.


Understand clearly the figure: Illustration of a Convolutional Neural Network (CNN) architecture for sentence classification

Answering this in terms of NLP examples is quite hard, remember "All models are wrong, some models are useful." First think of this in an image classification problem context, you want to use a large number of filters to collect a large number of features out of the image, one could detect edges, the other could detect densely coloured areas, one might turn a region to b&w. Extend  a similar logic to text, by using a lot of filters, in this case 128, you are trying to capture a lot of features. For an example like , " I like movies very much", a certain filter might detect that like is a positive word and not a similarity comparison, a certain filter of size 2 might detect very much and detect that it is an expression of degeree. You can go on like that, it will be hard to come up with 128 features but the idea is to get enough features. If you think the number is unreasonable and might lead to overfitting, you can reduce the number and compare your results.
No, 1- maxpool means that you take the maximum value of the output vector after applying a filter to the input. So it has nothing to do with the longest word but rather choose an element from the output that express the extracted feature to the highest amount.


So there are few possible answers to your question.
First of all you are mentioning Deep Convolutional GAN's which refers to Architecture of a Network. So you can either change Architectures or change Loss functions which are used in GANs which both can have an positive or negativ effect.
Also you mention not learning the right data distribution, it also cann occur that your data is the problem, i.e. too low number of samples, not representative samples etc.

There is no definite answer to your question, you have to try which works best for your dataset.

But a few hints for losses you can try: Wassersteindistance, Improved WassersteinGAN, Boundary Equilibrium GAN. In all of their publications also the architectures are published.


Duplicating training data for classes with fewer samples can actually improve your accuracy and that's what we call Oversampling.
But if you have balanced data and trying to duplicate then it doesn't differ from doing one more training iteration/epoch.


Suggestion for a better way to organize data to generate frequent item-sets?

If the size is reasonable (i.e. not too many documents and not too many words in a document), you could try to build a map for each possible itemset, for instance like this:

// Assuming data is an array of size N containing all the documents
// clusters is a map associating each itemset with a set of documents
for i=0 to N-1
  for j=i+1 to N-1
    group = overlap(data[i], data[j])
    add data[i] to the set clusters[group]
    add data[j] to the set clusters[group]

An alternative version if the number of different values and size of the sets allow it and/or if it's possible to precompute the itemsets of interest:

for i=0 to N-1
  for every subset S of data[i]
    add data[i] to the set clusters[S]

(adapted from https://datascience.stackexchange.com/a/60609/64377)


How do I compare traditional classifiers performance with my proposed method?

What about doing cross validation on your training set? Once you have the different train/test splits I would start by printing the accuracy (number of correct predictions / total predictions) and the confussion matrix for each method.

If you are using python sklearn.metrics offers a wide variety of useful functions.


How google get the 'use over time for' data from the before internet years?

I am not sure if you are familiar with this, but in linguistics there is a term "corpus". It is a large collection of texts used for statisical analysis and other linguistic research, both historical and contemporary.

There are corpora of digitalized texts, both literature and other domains such as newspapers, bulletins, etc etc. I recon Google uses data from such corpora.

For example: https://www.sketchengine.eu for English.

Here's a somewhat robust list of corpora: https://en.wikipedia.org/wiki/List_of_text_corpora


Powers of 2 for batch_size in model fit in deep learning

It's not just on GPU, you want to have a small power of two as submultiple for CPU as well.

The reason is to allocate the processors on the GPU (or the SIMD registers) as best as possibles. This is directly linked to the warp sizes and what the GPU vendors offer for maximum block sizes,

See also this thread on SO which what the warp size is and its relation to best occupancy.


How to choose the order in which to split a decision tree?

One option is to use a simple approach, like just choosing random attributes, or taking one at a time.

Another option is to use a metric to decide, at each step, the attribute that best splits the data. Different tree algorithms apply different metrics, like for instance Gini impurity (CART algorithm), or information gain (ID3 and C4.5 algorithms).

You can read a brief description of these methods in the "Decision tree learning" entry on the Wikipedia.


Gibbs sampling is a probabilistic framework which assumes all parameters to be learned are random variables. Here, you assume prior distributions for the parameters and find the full conditionals based on the relations specific to the problem, e.g. if you have three parameters, $\alpha,\beta,\theta$, you derive $p(\alpha|,\beta,\theta)$ ,$p(\beta|\alpha,\theta)$ and $p(\theta|\alpha,\beta)$. After deriving the full conditionals you sample from them and execute many iterations. It's like asking for $\alpha$ if we know $\beta$ and $\theta$, asking for $\beta$ when we know $\alpha$ and $\theta$ etc. After sufficient number of iterations you'll converge to a solution. In GMM, these parameters are mixture proportions, which can have a Dirichlet prior for example, and means and deviations of each gaussian component (you can choose suitable prior distributions for these). An example design would include full conditionals $p(\mu_i|\boldsymbol{\sigma},\boldsymbol{\pi})$, $p(\sigma_i|\boldsymbol{\mu},\boldsymbol{\sigma})$ and $p(\boldsymbol{\pi}|\boldsymbol{\mu},\boldsymbol{\sigma})$. Here choice of your priors are important because when you get formulas for these full conditionals it's better to have common distributions for easy sampling, e.g. $p(\mu_i|..)$ turns out to be gamma distributed etc. If not, you have to design samplers for these probability distributions, which can sometimes cause some pain. It is also an iterative algorithm as EM, but EM is purely an optimization procedure while Gibbs sampling is a Bayesian approach.


What is the Most Efficient Tool in Python for row-wise manipulation of data?

It depends on whether you can separate or know in advanced what rows are affected per operation.
In this instance, if indeed they involve collections then Dask Bags. For more non-structured data (as you suggest) but may be easy to run in parallel then Dask Delayed has worked for me.


The packrat package is what you're looking for.

It bypasses R's native packages and allows you to build and deploy a bundle of packages and dependencies.

However, it does not do Maven-style dynamic dependency resolution; you need to specify a list of packages that you believe will work together, then the bundle is installed/deployed as a single unit.