The reason that sigmoid functions are being replaced by rectified linear units, is because of the properties of their derivatives.

Let's take a quick look at the sigmoid function $\sigma$ which is defined as $\frac{1}{1+e^{-x}}$. The derivative of the sigmoid function is $$\sigma '(x) = \sigma(x)*(1-\sigma(x))$$ The range of the $\sigma$ function is between 0 and 1. The maximum of the $\sigma'$ derivative function is equal to $\frac{1}{4}$. Therefore when we have multiple stacked sigmoid layers, by the backprop derivative rules we get multiple multiplications of $\sigma'$. And as we stack more and more layers the maximum gradient decreases exponentially. This is commonly known as the vanishing gradient problem. The opposite problem is when the gradient is greater than 1, in which case the gradients explode toward infinity (exploding gradient problem).

Now let's check out the ReLU activation function which is defined as: $$R(x) = max(0,x)$$ The graph of which looks like

If you look at the derivatives of the function (slopes on the graph), the gradient is either 1 or 0. In this case we do not have the vanishing gradient problem or the exploding problem. And since the general trend in neural networks has been deeper and deeper architectures ReLU became the choice of activation.

Hope this helps.


macro average and weighted average meaning in classification_report

Your data set is unbalanced since 28432 out of 28481 examples belong to class 0 (that is 99.8%). Therefore, your predictor almost always predicts any given sample as belonging to class 0 and thereby achieves very high scores like precision and recall for class 0 and very low scores for class 1.

In the case of weighted average the performance metrics are weighted accordingly: $$score_{weighted\text{-}avg} = 0.998 \cdot score_{class\text{ }0} + 0.002 \cdot score_{class\text{ }1}$$ Which turns out to be 1 due the class imbalances.

However, macro avg is not weighted and therefore $$score_{macro\text{-}avg} = 0.5 \cdot score_{class\text{ }0} + 0.5 \cdot score_{class\text{ }1}$$

Since your model just guesses to almost always predict class 0 these scores turn out to be poor.

Going forward I suggest start reading about unbalanced classification problems. There are many approaches how to tackle this. One important question here is whether false predictions for the two classes lead to different cost or not (which, for example, typically is the case in medical applications, spam filters or financial transactions). Because if they do not then a predictor always guessing the majority class could even make sense. But that strongly depends on the area and way of application.


I don't think you ought to change much in the model definition.

You should, however, consider the amount of time it takes to train on the complete dataset. If it takes too long and you are still in the testing phase, you want to reduce the number of epochs to obtain results faster and make changes in the model accordingly.

I suggest plotting all the metrics and try to understand if the trend is positive or negative. If it's positive the changes you are doing are correct of course!

Then, once you are happy with the hyperparameters, put like epochs=100 and leave the model to train for a whole night, then plot again the learning curves and decide when to stop earlier, or use early_stopping.


Manipulating multi-indices for a pandas dataframe

Ordering

You can achieve this by changing the datatype to ordered categorical:

df['DofWeek'] = pd.Categorical(
                df['DofWeek'],
                categories=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'],
                ordered=True)

And then sorting the index:

df = df.sort_index()

Error

The issue with dataframe.query('DofWeek == 'Saturday') is bad quote encapsulation. You could solve it with e.g. 'DofWeek == "Saturday"'


customer segmentation with categorical variables

You can use mixture model with variable selection. In this framework, the challenge of variable selection consists in model selection. Thus, detection of the relevant features and estimation of the number of clusters can be done according to information criteria (like BIC or ICL).

To perform this analysis, you can use the R package VarSelLCM. Because you consider categorical variables, your dataset must be a data.frame and each column must be a factor. Here is an example of the script. Your dataset is denoted by "my.data".

## Clustering by considering all the variables as discriminative
# Number of clusters is between 1 and 6
res.all <- VarSelCluster(my.data, 1:6, vbleSelec = FALSE)

# partition
res.all@partitions@zMAP

# shiny application
VarSelShiny(res.all)

## Clustering with variable selection
# Number of clusters is between 1 and 6
res.selec <- VarSelCluster(my.data, 1:6, vbleSelec = TRUE)

# partition
res.selec@partitions@zMAP

# shiny application
VarSelShiny(res.selec)


Training a classifier with text and numerical features - what is the state of the art?

Regarding the first question, to the best of my knowledge, there is no SOTA approach. It depends on the task.

For the second one, did you try to combine the numerical values with the TFIDF vectors, and then feed the final vector(s) to the classifier? Probably not. I usually try this way. But, if your vocabulary is too large, try to reduce it by considering top N-words, because combining small numerical vector with a huge TFIDF vector may not allow the numerical values to affect the results.


Is a BiLSTM layer required if we use BERT?

That layer isn't required indeed as it also encodes the sequence, albeit in a different way than BERT.
What I assume is that in a BERT-BiLSTM-CRF, setup, the BERT layer is either frozen or difficult to fine-tune due to its sheer size. Which is likely why the BiLSTM layer has been added there.


Modeling the Price Movement- What analysis should be used

The question is if you really want to treat this as a time series problem. I say this because there may be no obvious/persistent autocorrelation (meaning that $y$ is contingent on $y_{t-1}$).

My take (not knowing the data) would be that each hotel has an own unobserved "identity" (like location, reputation etc) apart from star rating. Bookings (and thus prices) may also be highly contingent on time. So you could give a fixed-effects model a try. The idea is that you model this in a way like:

$$y=\alpha + \beta X+\gamma Z + \theta t + u.$$

Where $X$ are observed hotel characteristics, $Z$ are hotel fixed-effects (i.e. one indicator/dummy per hotel), $t$ is time (day of year or so), and $u$ is the error term.

In terms of model interpretation, $X$ are important (standard confounders), where $Z$ are not so relevant (and often omitted in FE regression) since $Z$ is only "one intercept per hotel". If you are interested in modeling the time aspect, clever encoding of $t$ is key. It is hard to tell how this could work out without knowing the data. There are two general options, a) dummy encoding (one dummy/indicator per time step) or continuous treatment (calender day?), maybe with a lagged component (price yesterday). In the latter case, you go in the direction of dynamic panel, which can be a little challanging.

You can try OLS first since it is very efficient in terms of computation. In order to get a better fit you may also try boosting, e.g. based on LightGBM. Here is a minimal example for boosting regression (but no fixed-effect model, just a normal one).


help finding research discussion on HTS classification

I would approach such a problem with a method a bit similar to record linkage: trying to match every product description with the most relevant HTS description. The traditional approach would be to use textual similarity measures such as cosine TF-IDF, but many variants can also be considered, e.g. with embeddings or other ways to take semantic similarity into account. At the end the code corresponding to the most similar HTS description is predicted.

Initially this could be done with vectors of words (unigrams), but it's true that comparing vectors of $n$-grams is likely to be more precise. However $n$-grams don't work the way you describe: in a case like your example you would have for instance $n=2$ and extract all the sequences of 2 consecutive words. This "bag of $n$-grams" is what the vector represents. Combining different lengths of $n$-grams is possible but not in the same representation: for example you could measure cosine similarity over 2-grams vectors only, then measure cosine similarity of 3-grams vectors only, and take the mean of the two scores (or even build a regression model using different similarity scores as features).

In case it helps, here is an example of computing a very simple similarity score with bigrams (one could certainly find better examples online).


What is the best way to deal with image sizes in object detection using YOLO v. 4?

I seem to have figured it out now. When using darknet.detect_image, it calls predict_image which in turn is network_predict_image. The latter resizes the image if it is not already the same size as the network's input layer.
If you instead call darknet detector test from the command line, as I have seen examples do as mentioned in my original question, the "chain of command" goes through test_detector which resizes the image if it is not already the size of the network's input layer.
So, in any case it seems that images get resized to fit the network's input layer along the way and I do not have to do anything in particular to handle it.


What is the simplest neural network for the simplest nonlinear function $f(x,y) =xy$

Probably you need to do one or more of:

Decrease learning rate. Diverging loss is often a symptom of learning rate too high.
Increase number of hidden neurons. The output function will be the combination of many "patches" each created by a neuron that has learnt a different bias. The shape and quality of each patch is determined by the activation functions, but almost any nonlinear activation function used in an NN library should work to make a universal function approximator.
Normalise inputs. If you are training with high values for $x_1$ or $x_2$, this could make it harder to train unless you normalise your training data.

For your purposes, it might be an idea to skip the need for normalisation by training with $x_1$ and $x_2$ in range -2.0 to 2.0. It doesn't change your goal much to do this, and removes one potential problem.

You should note that a network trained against an open-ended function like this (where there are no logical bounds on the input params) will not learn to extrapolate. It never "learns" the function itself, but an approximation of it close to the supplied examples. When you supply  a $(x_1, x_2)$ input far from the training examples, the output is likely to completely mismatch your original function.


How to implement HashingVectorizer in multinomial naive bayes algorithim

You need to ensure that the hashing vector doesn't purpose negatives.  The way to do this is via HashingVectorizer(non_negative=True).


Tree based method are robust against low probability feature space zones when using ML general interpretability methods?

Well, are you content with a system that doesn't work on that one day in winter when the temperature actually reaches 40 degrees? And would you care, since you probably now got other things to worry about

An assumption of independence is usually made to deal with sparse situations. Naive Bayes for instance works pretty well for document classification tasks: Each word (token) in a document is taken as an individual observation governed by a probability distribution belonging to the document class

Tree-based classifiers, on the other hand, generate composited rules and are thus geared toward exploiting conditional probabilities. Eg: If it is 12.2 degrees or below, and it is Winter, then the humidity is the discerning factor in bike use.

NOTE: That even though your rule sample is small, it is still quite complex Suppose we eyeball it and make a naive rule: If humidity < 83 roughly add around 55% That would emulate your rules quite well, and make it less complex, unless it happens to be a cold spring or summer day. Is that really a rule? Would that cause faulty predictions? Or did we just see little or no cold days that varied in humidity in that sample (200 data points)? I wouldn't start betting my bottom dollar that people react differently to humidity given the date on the calendar

You pose that the tree rules protect against probability mass leaking to impossible situations. Conversely you could pose that naive probability would help you make predictions about situations that were sparse in the dataset. Whether you'd want that is up to your reasonable judgement. It is machine learning, not machine wisdom


Zero Padding in Convolutional neural network

For padding in CNN's there is a useful answer in this link:https://stats.stackexchange.com/questions/246512/convolutional-layers-to-pad-or-not-to-pad
In Q-learning an action is taken in current state, and a next state is obtained. Then state-action value of the current state-action pair is updated by using best state-action value of obtained next state. If you have discrete state-action space, or a few states you can simply record each state-action pair in a table, and then update it by Q-learning, but if the space is continuous, or there are so many state-action pairs then you should use function approximation methods like neural networks. For example by extracting current state's visual features by CNN, and then updating weights to better approximate value of this perceived state, and taken action pair.


If you expand the terms, you can see that the quadratic terms cancel out.

\begin{align}
a &= \ln \frac{P(C_0)}{P(C_1)} + \frac12(x - \mu_1)^T\Sigma^{-1}(x - \mu_1) - \frac12(x-\mu_0)^T\Sigma^{-1}(x-\mu_0)\\
&=\ln \frac{P(C_0)}{P(C_1)} + \frac12\left[x^T\Sigma^{-1}x-2x^T\Sigma^{-1}\mu_1+\mu_1^T\Sigma^{-1}\mu_1\right]\\& - \frac12\left[x^T\Sigma^{-1}x-2x^T\Sigma^{-1}\mu_0+\mu_0^T\Sigma^{-1}\mu_0\right]\\
&= (\mu_0-\mu_1)^T\Sigma^{-1}x+\ln \frac{P(C_0)}{P(C_1)} +\frac12\left[\mu_1^T\Sigma^{-1}\mu_1-\mu_0^T\Sigma^{-1}\mu_0\right] \\
\end{align}


What is the top-5 error rate in ILSVRC challenge?

The Top-5 error rate is the percentage of test examples for which the correct class was not in the top 5 predicted classes.

So, for example, if a test image is a picture of a Persian cat, and the top 5 predicted classes in order are [Pomeranian (0.4), mongoose (0.25), dingo (0.15), Persian cat (0.1), tabby cat (0.02)], then it is still treated as being 'correct' because the actual class is in the top 5 predicted classes for this test image.

Remember to keep in mind that ImageNet has 1000 classes, so a low top-5 error rate is still not extremely easy to achieve.


building a model that calculate effect of an event

This is a classic example of Causality, where we need to find how much one variable is effected by changing other variable.
You can infer the causal graph from data and then select the variables which best describe your model.
Intervention can be done to represent particular event and then check the effect on target variable.
Refer:
Causalnex ,
DoWhy


Project structure - many projects share same large dataset

A database is the best option to share data across projects.

Another option is version control. Check the csv into version control. It could be git, GitHub, or data specific version control system.


What's SHAP contribution dependency plots from xgboost package in R?

On the y axis. That represents the log odd ratios. The formula is log(p/1-p)

P/1-P  P represents probability of success. 1- P represents probability of failure.

When log odd ratio is 0. It means odd is 1. Probability of success is the same as the probability of failure.

When log odd ratio is below 0, it means that the probability of sucess is less than probability of failure.

When log odd ratio is above 0, it means that the probability of sucess is higher than probability of failure.


How can I extract numerical information conditional on other information?

If your data is always as clean as this, you might be able to solve the problem using simple regular expressions. You could simply look for the first number that occurs before the string of interest.

As to the dependency parsing using spacy, this is the output that your sentence would get

There expl are VERB []
are ROOT are VERB [There, people, now, .]
119 nummod people NOUN []
people attr are VERB [119, in]
in prep people NOUN [gym]
the det gym NOUN []
gym pobj in ADP [the]
right advmod now ADV []
now advmod are VERB [right]
. punct are VERB []
100 nummod people NOUN []
people ROOT people NOUN [100, in, ,, 19]
in prep people NOUN [room]
the det room NOUN []
weight compound room NOUN []
room pobj in ADP [the, weight]
, punct people NOUN []
19 appos people NOUN [on]
on prep 19 NUM [treadmills]
treadmills pobj on ADP []

Meaning that you would need to first locate weight room, and then trace back to 100, which in this case might simply be more complicated than using regular expressions.


How can I calculate AUC from the ROC curve for the classification?

Welcome to the community!

As you know, AUC is just the area under ROC curve. So the question is more about numerical methods as you have a set of points and you would like to calculate the area under it.

Riemannian Sum

Trivial solution. Simply make rectangles from points you have. The area of each rectangle is simply the product of edges. Then sum them up! You probably don't like it do you?!

Trapezoidal Method

After Riemannian, the simplest and most naive algorithm to do this. You simply have a set of points and you just calculate the trapezoidal area between each pair and sum them up like what you see in the figure below. It has the maximum computation error as it simplifies the problem a lot.

Simpson (1/3) Method

Much better when we are talking about curves! Let's keep it simple and to the point. You can model your function in every interval using a quadratic ($y=ax^2+bx+c$) and having 3 data points. Using your three data points, you can calculate $a$, $b$ and $c$. Then the area under curve is not that difficult, but we have a better solution! Trust me or not, the value of this integration is simply

$$\frac{b-a}{6} (f(a)+4\times f(m)+f(b))$$

where $(a,f(a))$ and $(b,f(b))$ are endpoints of interval and $(m,f(m))$ is the midpoint. See the image below from here to compare these methods.

Romberg Methods

Simpson and/or Trapezoidal methods can be recursively applied to achieve a more accurate calculation. It's called Romberg method. Accuracy of these methods were in the length of interval. Smaller intervals give more accurate integration. Romberg uses this fact to iteratively get closer to more accurate answer.

And of course tones of more algorithms to do that.

PS: You certainly have libraries and functions in different languages to calculate it for you. Scipy offers for Python for instance.

Hope it helps! Good Luck!


Should a Cluster Validity Index contain the same measure(s) as the Clustering Algorithm?

It's standard to use, e.g., Silhouette to assess the quality of clusterings, that were obtained with other methods.

I'm not a big fan of this (people always think they would get the "optimal" result this way, but they don't), but it's all over literature. Logically, you are trying to maximizing Silhouette and use the other algorithms as sampling procedure for possible results.


Understanding projection layer for BLSTM

A "projection" is a simple linear/dense layer, that is, a matrix multiplication and a bias vector addition.
It is called projection because you "project" a representation of dimensionality $M$ into a representation space of dimensionality $N$.
Sometimes, especially for sequences or 2D data, these projections are implemented as a convolution of size 1, which is equivalent to the computations I described above.


Classification accuracy of a Random Multi-label Classifier

In the context of a multi-label classifier (that is, having potentially more than one label per data point) we can have a prediction which is either fully correct, partially correct or fully incorrect. To account for this, the following accuracy metric has been proposed:
$$\text{Accuracy} = \frac{1}{n}\sum_{i=1}^n\frac{|T_i \cap P_i|}{|T_i\cup P_i|},$$
where $n$ is the sample size, $|\cdot|$ denotes the number of elements of a set, $T_i$ is the set of true labels and $P_i$ is the set of predicted labels for an item of the sample.

For instance, let's say the $i$-th item has actual labels $A, B, C$ and $D$ (that is $T_i = \{A, B, C, D\}$, but your classifier predicts labels $B, D$ and $E$ (that is $P_i =\{B, D, E\}$), then you'd get
$$\frac{|T_i\cap P_i|}{|T_i\cup P_i|}=\frac{|\{B,D\}|}{|\{A,B,C,D,E\}|}=\frac{2}{5}.$$
To get the accuracy for the entire training set, you calculate this for each instance $i=1,\dots, n$ and you average it out.

You can also have a look at the evaluation metrics section of the article in Wikipedia about multi-label classifiers: https://en.wikipedia.org/wiki/Multi-label_classification#Statistics_and_evaluation_metrics


Unfortunately, there isn't a built-in option to do it. Each time you run K-Means, the labels are assigned randomly. Even if you state the same random seed. However, based on this answer in StackOverFlow, you can create a lookup table and run it after your K-Means.
from sklearn.cluster import KMeans
k = 5
kmeans = KMeans(n_clusters=k, random_state=0).fit(X)

idx = np.argsort(kmeans.cluster_centers_.sum(axis=1))
lut = np.zeros_like(idx)
lut[idx] = np.arange

With this, you will always have the same output as
In [73]: kmeans.labels_
Out[73]: array([1, 4, 1, ..., 0, 1, 0])

In [74]: lut[kmeans.labels_]
Out[74]: array([3, 0, 3, ..., 2, 4, 2], dtype=int64

When lut[kmeans.labels_] = 0, then you always have the smallest cluster and with lut[kmeans.labels_] = 4 you have the largest cluster.


As far as I understood stacking does not add features to the original data set. The point is to train several models on the training data and use their predictions on training data as input features to another model.

First such kind of construction used logistic regression as a final ensemble and and class probabilities from each base learner as input features. Now, what I have described is a technical layout, the intuition behind is the following: considering that there are no models which are good over all joint probability space of features, one can combine their results in order to get the best from each one. In other words we can state the we explore the richness of models (seen as function spaces) to get a combined thing. This strategy does not work always but often it works.

I think you do something wrong. I think is better to use original features only for base learners. Be careful to use scores or probabilities if possible, instead of final classifications from base learners, it gives more space for improvements. Often is better to stack learners from different families, not the same model with different parameters (better to use a gradient boost and random forest than two gradient boosts). All of those advices are not rules which cannot be broken, and even if you take them all there is no guarantee that there will be improvements.


Transform large pd.Series into a DataFrame of n columns

Here is a quick solution that does not do it in-place but takes up extra space:
def transform_series(x, chunk_size):
    df = pd.DataFrame()
    for i in range(chunk_size):
        df[f'column_{i+1}'] = x[i::chunk_size].reset_index(drop=True)
    return df

input_series = pd.Series([10,11,12,13,14,15])
transformed_df = transform_series(input_series, chunk_size=2)

Output:
print(transformed_df)

   column_1  column_2
0        10        11
1        12        13
2        14        15


How can you map the exceedance of a threshold into an activation function of a Neural Network?

This is feasable. This is also called a Binary/Step activation function. You must only use this activation function on the output neurons.

The Step function will round down an answer that is lower than 0.5 to 0, and an answer that is higher than 0.5 to 1. However, please note that you do not need to use a binary activation function to output 1 - I advise you to just use TanH or sigmoid and backpropagate a whole bunch of iterations.

However, in other comments, you mentioned that you know what y1 is. That is not of importance, the network will act as a black box and will figure out a treshold itself. Don't set up your own activation function just to get the right output - that avoids the whole point of backpropagation.


Please read carefully caret::train since S3 method for class formula is slightly different than default S3 method for caret::train (and takes different arguments)

This should do the work:

model <- caret::train(y = traing$TargetBuy, x = traing[, colnames(traing) != 'TargetBuy'], trControl = train_control, method = "nb")

You can also use model.matrix, but then you need to remember that it removes missing values. See this post for solution:

model <- caret::train(y = factor(traing$TargetBuy), x = model.matrix(TargetBuy ~ ., data = traing), trControl = train_control, method = "nb")


Similarity of search results using Jaccard

I realised if I used the format

X = np.array([[5,4,2,3,1,0,0,0,0],[4,0,0,0,0,5,3,2,1]])

and a sparse matrix from the scipy module, I could use efficient matrix calculations. This ran in about 2 seconds for 7k samples and 400 features.

from scipy.sparse import csr_matrix
import numpy as np

def jaccard_sim_matrix(X):
    """X is an integer array of features"""

    sparseX = csr_matrix(X)

    # make a binary version of the matrix
    binX = sparseX
    binX.data[:] = 1

    intersection = ((sparseX * binX.T) + (binX * sparseX.T))

    rowwise_sum = np.sum(sparseX, axis=1)
    union = np.repeat(rowwise_sum, intersection.shape[0], axis=1) + np.repeat(rowwise_sum.T, intersection.shape[0],
                                                                              axis=0)

    return intersection / union

Then of course I got the distance by 1 - similarity


"ValueError: Index contains duplicate entries, cannot reshape" error when I try to use pd.MultiIndex.arrays

Thank you very much to who ever had taken time into looking into this.  I have resolved this issue by adding a new unique col before apply multiIndex

df_s2 = df_s[df_s['Test'].isin(['TOTAL TRIIODOTHYRONINE (T3)','TOTAL THYROXINE (T4)','THYROID STIMULATING HORMONE (TSH)','FREE THYROID 3','FREE THYROID 4','Human Chorionic Gonadotropin (hCG)','BILRUBIN'])]
[df_s2['ID1'] = range(1, len(df_s2.index)+1)

df_s2.set_index(['ID1'])

idx = pd.MultiIndex.from_arrays([df_s2['ID1'],[df_s2['ID'], df_s2['Name'], df_s2['Age'], df_s2['Sex'], df_s2['CT'],df_s2['RT'], df_s2['Units'], df_s2['Test'],  ])

df_s5 = df_s2.set_index(idx).Test_Result.unstack(fill_value='')
df_s5.columns.name = None
df_s6= df_s5.reset_index()
df_s6.head(100)
```


Well, there are ways to do it within model.fit(). Namely, you can define validation_split parameter, (e.g. 0.2 or 0.3) which represents a percentage of the data that will be used as a validation in training (0.2 = 20%, 0.3 = 30%).
The second way, when you have your validation data, is to use validation_data = (x_val, y_val), where you just pass your validation x and y


Is DQN limited to working with only image frames?

DQNs don't only accept image frames as input. For instance, this DQN for the CartPole environment takes in a state with only 4 elements (the position and velocity of the cart and the angle and velocity of the pole.)

Images aren't necessarily more efficient representations than other options.  For instance, here is another DQN for the same CartPole environment which uses an image representation and is much less efficient.  A couple reasons people often use image representations include:

It's the only available representation for the task.
Other models  uses image representations and you want to compare.
You want to pass the network the same information available to humans.

If none of these apply, though, if you have a representation which gives you more insight into the position or trajectory of elements in the screen it's probably best to use that rather than a raw image.


Getting different precisions for same neural network with same dataset and hyperparameters in sklearn mlp classifier

If the initialization of your network is random, and the order in which you feed it the training examples is not fixed, then your training can lead to a different model every time, with different performance. That’s normal, although if you train long enough, you would hope that you often reach a solution of similar quality. So perhaps you are not training long enough. Have you plotted the loss as a function of the training time (in terms of max_iter)? Is your model training until reaching the maximum number of iterations, or is it converging before?

Another possibility is the amount of training data. So repeat the above exercise where you figure out how long you need to train with smaller fractions of your training data. For example, what’s the best you can do with 20% of your training data, with 40%, 60%, and so on? If your best result is still improving significantly when you reach 100%, then maybe you need to get more training data.


What does depth mean in the SqueezeNet architectural dimensions table?

In this paper the depth is defined as the number of layers. The table shows a depth=2 for the fire layers because each is comprised of 2 layers. First is the squeeze layer and then followed by the expand layer.

To calculate the number of parameters of a CNN we can do as follows for a single layer. Assume an input that is 28*28*64. This is the size of the MNIST dataset with 64 channels. Now assuming we want to convolve this input with 32 filters of a 3*3 kernel.

Let us give the following variable names, $l=64$ is the number of channels in the input, $m=n=3$ is the size of the kernel and $k = 32$ is the number of filters. The number of parameters is then calculated as

$((n*m*l)+1)*k = ((3*3*64)+1)*32 = 18,464$.

The $+1$ is used to add the biases.

To calculate the number of parameters in fire2 we first note that this module is comprised of 2 distinct layers.

Layer 1 - The squeeze layer - $s_{1x1}$

$l = 96$, $m = n = 1$ and $k = 16$. Thus giving a total of 1,552.

Layer 2 - The expansion layer

Kernel size 1 - $e_{1x1}$

$l = 16$, $m = n = 1$ and $k = 64$. Thus giving a total of 1,088.

Kernel size 3 - $e_{3x3}$

$l = 16$, $m = n = 3$ and $k = 64$. Thus giving a total of 9,280.

Total

Thus, a total of $1,552 + 1,088 + 9,280 = 11,920$.


Simply put you can create an "index data base" in the following format:
ID | URL | Timestamp | link_to_file | other_metadata

You could even store the actual file, instead of just the link with most databases.
However the simplest way might be the best here. This "index db" would be created automatically in the scraping process and serve as navigation and check to avoid double scraping, etc.


Optimal projection for data visualization

There are a few interesting plots and transformations you could start with, each dependent upon the purpose of your analysis. Below are some first steps I might take.

If simply visually looking for clusters:
If clusters are what you are after, then I would recommend applying a Principal Component Analysis to the dataset and then plotting the dataset with the first 2 principal components as the axes. However, the major downside to PCA is that you will have to go "unpack" the principal components to find the original variables. In other words, you'll be may be able to identify cool clusters, but it will be a little harder to tie the findings back to your 32 variables.
If looking for quick relationships between variables to build on:
With "only" 32 variables you could do a pairwise plot. However, a smarter way may be to first identify the relationships mathematically (e.g. correlation) and then plotting those variables.
If looking for quick relationships in each variable to build on:
Or, look at 32 histograms to start off with. Look out for clear bi-modal (or more) to begin piecing together an understanding of how your variables may contribute to an unsupervised model. If you end up look at 32 unimodal histograms then you can conclude early that no matter how you cluster, you will simply end up with a blob.

Actually, in usual analytics workflows I would go 3 > 2 > 1. But if I was fishing for clusters and just wanted to see if clusters will appear or not, PCA would be a good shortcut.

Also, feel free to sample your dataset. 90k points on your screen will likely do more harm than good.


Difference between OLS(statsmodel) and Scikit Linear Regression

So my question is the both method prints our R^2 result but one is print out 0.98 and the other one is 0.85.

There is a lot of information missing here, but it looks like the first example could be finding the correlation between the predicted and actual values of data from your training set and the second example is finding the correlation between the predicted and actual values of your testing set. This would also explain why the correlation in the second result is lower. The performance of a model on novel (testing) data should nearly always be poorer than on the training dataset that was used to fit the model.

  Is there a way that work with test data set with OLS ?

The statsmodels documentation describes a predict() method that can be used to make predictions on inputs from a design matrix.  You should be able to use this method on your test data once you have fit the model.

  Is the traning data set score gives us any meaning(In OLS we didn't use test data set)? From my past knowledge we have to work with test data.

Evaluation on training data can often be used as a quick check for "best case" generalization performance.  However, this is often only used as a diagnostic to ensure your model has fit properly and should not be taken to be a true indicator of generalization.


CSVs: how to output missing data to make processing easier?

Both pandas and readr allow flexibility on the marker used to indicate missing values, allowing to specify values that should be treated as missing when reading csvs:

readr https://readr.tidyverse.org/reference/read_delim.html
pandas https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

given that, maybe you can handle missing data in your code rather than replacing values in the original csvs.

If you get to choose, maybe NA (no hyphens) is a sensible default as it's handled as missing value by both libs by default.


How to compare the performance of feature selection methods?

This is a hard problem and researchers are making a lot of progress.

If you're looking for supervised feature selection, I'd recommend LASSO and its variants. Evaluation of the algorithm is very straightforward with supervised learning: the performance of whichever metric you choose on test data.

Two major caveats of LASSO are that (1) the selected features will not automatically detect an interaction, so you have to craft all of your features a priori (i.e., before running them through the model) and (2) LASSO will not identify non-linear relationships (e.g., a quadratic relationship).

A way to try and get past these two caveats is to use Gradient Boosted Machines which does feature selection automatically. It's worth noting the statistical properties of GBM are a little more ambiguous than that of the LASSO.

If you're looking for unsupervised feature selection, it seems there's a similar regularization approach used by these researchers, but evaluation in this particular case becomes less obvious. People try a lot of different things like PCA/SVD or K-Means which ultimately will try to find a linear approximation to the data.

In that case, the typical measures of performance are the reconstruction error or the RMSE of the clusters.

In terms of software, R and Python both have GBM, LASSO, K-Means, SVD, and PCA. GLMNET and XGBoost in R and Sklearn for Python are the relevant libraries.


Difference between parameters and weights in ANN

That's partially right.

Apart from the weights (and bias) you also have the so called hyper-parameters. Depending on your architecture, you may have different ones. For a simple feedforward neural network, the hyperparameters are:

number of neurons
number of layers
learning rate eta (η)
regularization penalty lambda (λ)
momentum
number of epochs
batch size
dropout

etc

These hyperparameters are (usually) user-tuned. The weights (and bias) are self-trained by the network, based on optimization. Keep in mind that hyperparameters like the number of neurons or layers are immediately connected with the underlying model of your data, therefore they are an index of your dataset's complexity.


Intuitively, a weak learner is one that is somewhat better than random guessing, but not particularly good. More precisely, a weak learning algorithm is an algorithm to produce some classifier which has a high chance of being better than random guessing, depending on your training data.
There is a more formal definition given in [1]; one of the authors is Leslie Valiant, who developed the PAC learning framework. The definition given is cast in this framework – if you're unfamiliar, a good reference is chapter 2 of Mohri's Foundations of Machine Learning. If the formalism is difficult to follow, hopefully the more intuitive definition will help – in effect, the formal definition says the same thing, but in more precise terms.
It's easiest to think about binary classification. We can certainly produce a classifier that is correct with probability $1/2$, no matter what the distribution is: just always pick the more common class!
The idea of a weak learning algorithm is to ask for a bit more: can we find an algorithm $\mathcal A$ that will give us a classifier that has accuracy at least $1/2 + \varepsilon$ with high probability? Note that the classifier depends on the randomly drawn training data, so we must speak in terms of probabilities here. We might be unlucky and draw training data which makes it hard for the algorithm, for example.
I prefer the presentation of weak learning in [2], so I will paraphrase it here.

A weak learning algorithm $\mathcal A$ is an algorithm where there exist $\varepsilon > 0$, $\delta \in (0, 1)$ and $N \in \mathbb{N}$ such that given a randomly drawn training set $\{(X_i, Y_i)\}_{i = 1}^N$ from any distribution, the corresponding classifier $h_{\mathcal A}$ constructed by the algorithm satisfies
$$ \mathbb{P} \left [ h_\mathcal{A}(X) \neq Y\right ] \leq 1/2 - \varepsilon$$
with probability greater than $1 - \delta$ for randomly drawn $(X, Y)$ from the distribution.

In other words, for a weak learner as defined, given $N$ samples, the algorithm will perform strictly better than random guessing with probability at least $1 - \delta$ on any distribution. But, it might not be much better: $\varepsilon$ might be small, so we are only a little better than random guessing.
Every strong learner is also a weak learner under the formal definitions in [1] and [2], so indeed, if a strong learner exists, a weak learner could actually be a strong learner. It is certainly possible that an algorithm could produce a strong learner on one distribution while satisfying the weak learning algorithm property.
References
[1] Michael Kearns and Leslie Valiant. 1994. Cryptographic limitations on learning Boolean formulae and finite automata. J. ACM 41.1 (Jan. 1994), pp. 67–95.
[2] Leo Breiman. 1998. Arcing classifier (with discussion and a rejoinder by the author). Ann. Statist. 26.3, pp. 801–849. Open Access.


Many classifiers will give the option to get predicted probability. Then you can just put a threshold. Here is how it can be done in with sklearn:

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification

# Make a dataset
X, y = make_classification(n_samples=1000, n_features=4,
                           n_informative=2, n_redundant=0,
                           random_state=0, shuffle=False)

clf = RandomForestClassifier(n_estimators=100, max_depth=2,
                              random_state=0)
clf.fit(X, y)

# 1 if proba is less than 0.8, otherwise 0
predictions = 1 - (clf.predict_proba(X)[:, 0] > 0.80)


Which one of these tasks will benefit the most from SPARK?

I think second job will benefit more from spark than the first one. The reason is machine learning and predictive models often run multiple iterations on data.

As you have mentioned, spark is able to keep data in memory between two iterations while Hadoop MapReduce has to write and read data to file system.

Here is a good comparison of the two frameworks :

https://www.edureka.co/blog/apache-spark-vs-hadoop-mapreduce


Combining outputs of ridge regression models?

If you mean predictors by "variables", You may combine your 7 models by using "rbind or cbind" and save it into a data frame or data table. But, remember to first save your each of the 7 models into a datatable or dataframe.

About ranking, I agree with @Valentin you may want to make the predictions first, then rank it . However , if you want to take all the predictors into account there is no point to apply the ranking of all the sets. You will see their importance once you are done predicting.

Additional note: Glmnet is way more sensitive library/method when performing predictions, make sure all the columns while creating a model and doing a prediction are in the same order and have the same names. This caused me much trouble while predicting.
Hopefully, this might help you.


Building a machine learning model based on a set of timestamped features to predict/classify a label/value?

You can use panda date time function to create various features representing time; Total time, day etc.  Then use these features as your independent variable to train your model


Why validation loss worsens while precision/recall continue to improve?

What I found while digging down:

As neural network continues to train, it gets more confident in predictions. And while it overfits at certain scenarios, it makes log_loss to increase exponentially, since it was more confident while mistaken.

Meanwhile, if I look at mean average error, I would see that it continues to decrease, which indicates that network still learns something. Which at the end make f1_score to increase.

One important thing I noticed: it will not always hold true, because there is a tradeoff between overfitting/learning new, and MAE is weaker metric in binary outcomes.


Can someone explain to me how to use a predictive model to predict something other than the training set

Yes, you can predict who will open a loan with this model. You have to keep several things in mind. The model expects the new data to be from the same distribution as the training data, meaning the variables have the same distribution and targets as well.
You can not train on one set where 3% opened a loan and then predict on a set where you expect that 90% open a loan. This data is then too different for the model to accurately predict on.
Remember that your model only has 90% accuracy, so it is not guaranteed that a specific costumer will open a loan when the model predicts so.


KNN Variance using a high value of K and cross-validation

It's just an example in the article trying to make the point that a small change with regards to the age of $p_{11}$ from $30$ to $32$ has relatively large impact on the prediction since with $k=1$ for

$age_{p_{11}}=30$, the model would predict $heigth_{p_{11}}\leftarrow heigth_{p_{3}}$ and for
$age_{p_{11}}=32$, the model would predict $heigth_{p_{11}}\leftarrow heigth_{p_{9}}$.

Moving $p_{11}$ to $age = 50$ would just not serve as a good example for the case they are making because it's not surprising that a model predicts different height values for $age=30$ and $age=50$.
To phrase it differently, the relation between $p_{11}$ being at $age=30$ or $age=32$ is that it's only a small difference in terms of age.


Summing three lexicon based approach methods for sentiment analysis?

What you're proposing is some very simple form of ensemble learning.
You need to have at least a sample of labelled data in order to evaluate any method. Using this labelled data you can:

evaluate each of the three methods on their own
evaluate your idea of averaging the 3 methods predictions
if you have enough labelled data, you could even train a model which combines their predictions optimally (this would be full-fledged stacked generalization)


Machine learning algorithm that uses the Pearson or Spearman correlation?

I do not think this exists. I suppose an algorithm could use pearson coefficients as starting coefficients, but honestly it seems like a waste of computational resources. Here are some reasons that occur to me as to why it is a bad idea:

Pearson and Spearman correlations become decreasingly meaningful as
the number of dimensions increase. I commonly work with millions of dimensions...Spearman correlations for individual features?
In sparse matrices, these coefficients mean next to nothing as there will be only a very slight correlations between a feature and the target. Usually, it is a multi dimensional relationship that we are trying to find (lots of caveats placed here :P)
Pearson and Spearman correlations assume certain parameters which are not usually true in ML applications ie homoscedasticity, linearity, normality, etc.

For the above and many other reasons imho it doesn't serve any purpose to use these anywhere in ML algorithms.


How to calculate the output shape of conv2d_transpose?

Here is the correct formula for computing the size of the output with tf.layers.conv2d_transpose():

# Padding==Same:
H = H1 * stride

# Padding==Valid
H = (H1-1) * stride + HF

where, H = output size, H1 = input size, HF = height of filter

e.g., if `H1` = 7, Stride = 3, and Kernel size = 4,

With padding=="same", output size = 21,
with padding=="valid", output size = 22

To test this out (verified in tf 1.4.0):

import tensorflow as tf
import numpy as np

x = tf.placeholder(dtype=tf.float32, shape=(None, 7, 7, 32))
dcout = tf.layers.conv2d_transpose(x, 64, 4, 3, padding="valid")

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    xin = np.random.rand(1,7,7,32)
    out = sess.run(dcout, feed_dict={x:xin})
    print(out.shape)


Unbalanced target variable in Orange, how do I balance it?

For unbalanced classes use the Python Script widget, with imblearn add-on, you will need to code!

Link for thread in github


Need help in selecting a model for a classification problem with binary data

Logistic regression models can certainly be used with dichotomous features. It also provides coefficient estimates for each feature so that relationships between features and the target labels can be tested and interpreted. Predictions made from logistic regression models are probabilities rather than binary decisions which can be helpful if you have targets for Type II Error rates, False Omission rates, etc. which is often the case in clinical data.

Distance-based methods can also be used such as the k-nearest neighbor algorithm. With all binary features, it would make sense to use distance measures designed for dichotomous data, such as the Russell Rao distance metric. These models will make predictions based on the class labels of the k-nearest observations in the feature space.


Which model could I use for time-series regression?

You could start using a dynamic regression model, they are used when besides the times series to be forecasted, you have another series where you could use it.
Essentially, this is a usual regression model, also called Autoregressive Distributed Lags (ADL), where you include not only the dependent variable $ y_{t} $ but also explanatory variable $ x_{t} $. Following equation exemplifies that:
$$ y_{t} = c + a_{1} y_{t-1} + b_{0} x_{t} + b_{1} x_{t-1} + \epsilon_{t} $$
Now notice that equation above uses $ x_{t} $, meaning that at time you have variable $ x $ so you could use it to forecast $ y_{t} $. This is not always possible, so take care of data leakage.
You can read more about it in Forecasting: Principles and Practice - Chapter 10 Dynamic regression models
Having said all that, you can also improve your ML approach by computing temporal features. Rolling Averages up to lag t, Rolling sum, max, min, and all lags variables could be example of feature engineering you could apply and still use a machine learning approach. Also, if you use python, there is a fresh package called sktime where you can use Reduced Regression.

Example from their notebook (link above):

from sktime.forecasting.compose import ReducedRegressionForecaster

forecaster = ReducedRegressionForecaster(regressor=regressor, window_length=12, strategy="recursive")

forecaster.fit(y_train)
y_pred = forecaster.predict(fh)
plot_ys(y_train, y_test, y_pred, labels=["y_train", "y_test", "y_pred"]);
smape_loss(y_test, y_pred)


Why don't companies use on-device ML to serve ads to protect privacy?

There are several problems:

The problem with serving ads on server side is that they process information about the users, so it can interfere with their privacy, or even break some privacy laws in some cases.
The suggestion to move the algorithms to client side (e.g. mobile app) solves this issue, but produces the opposite problem: now you need to upload all the data about the adds to client-side app. This may be problematic because this could expose the corporate secrets of your company (who are your customers, what kind of adds they want to show to users, the target group for the ads). Your customers, who paid for the ads may not like it. Your competitors may potentially abuse this to gain advantage over you.

Another problem is that serving ads is about matching users to ads. On client side you could only match ads to user. Imagine that your customer pays you for showing 10K ads to the users, if you assigned ads on user side, you would not have direct control on showing exactly 10K ads, so you could loose money by showing the ads that were not paid for. Sure, there are ways around this, but this is a problem that needs consideration.
In real-life this gets even more complicated, because the ads are usually served in terms of betting (who pays more, gets their ad shown), where different customers bet to make ads and algorithm decides about most optimal share, such that it maximizes the gain from showing the ads. In such case, not only you don't have control over global gain, since you are making only local optimization decisions, but you would need to send the bets to client-side app, so again share sensitive information about your customers (how much do they pay for the ads).

Third thing is that even if the client-side app made a decision, it would probably still need to query ads database to download the user-matching ad. I guess, you could backward-engineer this to learn the characteristics of the user given the ads that were assigned to them by the algorithm, so in the end, this might not protect privacy.
As alternative, you could send all the ads to user and store them in the app database, but then you would need user to store potentially large amounts of your data just that you could show them your ads. I doubt users would like this.

Finally, I'm not sure if ad selling businesses is the most concerned about user privacy, i.e. if they would care.


Which is better for a beginner: R or Python?

It's a perennial debate. Python is more readable and quicker to learn, no question. It's also a bit more general-purpose language than R. R, on the other hand, has special statistical packages for just about anything you could even dream of doing. There are some stats it can do that Python doesn't have a library for (though I suspect people are working on that).

What's more important in the early stages of learning data science is the more fundamental theory: mathematics, linear algebra, calculus, and statistics. A firm grasp of those areas is a considerably larger share of the learning path than a particular language.

Having said that, I do happen to prefer Python because of the readability. The fact is, that very often other people will need to come in after you and read what you wrote. For that matter, you sometimes have to read your own code! Readability is more important than the ability to do fantastic things with one-liners. As for the advanced stats, that comes into play most often in the medical fields; there you definitely find a preponderance of R over Python.


Text Classification with mixed features in Random Forests

Are you using raw term frequencies, or TF-IDF?

Perhaps you could simply combine the terms in the tweet with the terms in the URL-linked pages (if any) into a single bag of words, calculate TF-IDF, and normalize to avoid bias towards longer documents (i.e., those tweets containing URL links).

  if I decide to use the same term frequency representation also for URLs I will have a huge number of features only from URLs

I don't understand what you mean here. Aren't your features the terms in your bag of words? So the number of features will be the size of your vocabulary, which I imagine won't change much whether you include URLs or not.

  Besides I'll have to fill some impossible values (like -1) for the URL features for tweets that do not have URLs, and I will probably worsen the classification for this tweets, as I will have a huge number of uninformative features.

I don't understand this either. Term-document matrices are virtually always a sparse matrix, since most of the terms in your vocabulary won't appear in most of your documents. So, the vast majority of values in your TDM will be 0. I don't know where you're getting -1 from.


If you are performing regression, you would usually have a final layer as linear.

Most likely in your case - although you do not say - your target variable has a range outside of (-1.0, +1.0). Many standard activation functions have restricted output values. For example a sigmoid activation can only output values in range (0.0, 1.0) and a ReLU activation can only output positive values. If the target value is outside of their range, they will never be able to match it and loss values will be high.

The lesson to learn here is that it is important which activation function you use in the output layer. The function must be able to output the full range of values of the target variable.


Understanding word2vec vectors representation

First of all, consider this: Word2Vec generates this representation by looking at the context (neighboring words, here with a window of 1) of each word or n-gram (e.g. pair of words such as Contain Docker).

Your are using a uni-dimensional representation of those pairs of words. Even in a particular training text such as yours which seems to be revolving around a unique subject (i.e. docker), a unique value as representation means a unique dimension in the "semantic space". This yields really poor representation power, and Word2Vec models are usually trained with more than 300 hundreds dimension to fully capture the semantic of words. And usually, more is better (but not always).

I hence suggest that you increase the size of your vectors (try different values, such as 100 and 200).

To see if the new representation makes sense, I suggest to use Multidimensional Scaling, which enables to project N-dimensional data on a plan. See for instance the answers to this question

What if you still want to use a uni-dimensional encoding ?

For a training text about emotions, it could mean a positive vs. negative dimension (example representations: happy = 0.85, sad = -0.94). In your case, I have no idea and suggest to plot a selection of words along this axis, to see if there are any obvious pattern for having high value or low value.


Clarification about Octave data size limit

You have about 4GB of RAM on your machine and Octave is an in memory application.

If you want to work with 515345 instances and 4275 features, assuming that you are using double precision (i.e. 8 bytes), you would need a memory of 515345*4275*8/1000000/1024 bytes ~ 17.6 GB. Even if you were using 4 bytes for each data point, you would require at least 9 GB for the computation to go through.

This issue might not be the Octave memory restriction in this case. See here for further details on Octave's memory usage.


Sampling from a multivariate von Mises-Fisher distribution in Python

It looks like you can sample the von Mises-Fisher distribution with that R package. Have you thought about calling R from within Python using the rpy2 package? I haven't tried this for myself, but could you try the following?

from numpy import *
import scipy as sp
from pandas import *
from rpy2.robjects.packages import importr
import rpy2.robjects as ro
import pandas.rpy.common as com
from rpy2.robjects.packages import importr

# import the movMF R package
movMF = importr('movMF')
# call the rmovMF sampling function from the R package
print(movMF.rmovMF(10, 3 * c(1, -1) / sqrt(2)))


I've managed to write a function that iterates over it for me:

class TaggedDocumentIterator(object):
def __init__(self, doc_list, labels_list):
    self.labels_list = labels_list
    self.doc_list = doc_list
def __iter__(self):
    for idx, doc in enumerate(self.doc_list):
        for idy in enumerate(doc):
            yield TaggedDocument(words=idy[1].split(), tags=[str(self.labels_list[idx]) + '.' + str(doc.index(idy[1]))])

docLabels = list(range(len(texts)))
data = list(texts)
sentences = TaggedDocumentIterator(data, docLabels)


When is it appropriate to split a dataset on a categorical value and generate $n$ models instead?

I've tried 2 several times but it has never proved better than 1.
I think the reason is, the more data you feed to a model, the better. The disadvantage of 2 is that the models that are trained use less data than the model in 1.
In addition, some features might be independent of the group. For instance, when modelling property prices, being in the city centre always increases the price, both for residential and industrial.
Let me discuss two of the main models used for tabular data:

Tree based models will already do the feature engineering that you described in your first point. The model will already do a split residential/industrial if it contributes to the gain and then it will keep doing particular splits for each group.
Linear models: a generalization of general linear models are mixed models, that kind of does what you mentioned on the second point, but keeping some structure that allows it to acknowledge that the city center is more expensive.

That being said, if you have very different categories, it might be worth splitting the dataset, it's just a matter of trying.


Train a deep learning model in chunks/sequentially to avoid memory error

Pandas is not memory efficient for data loading.

It appears you are using Keras / TensorFlow package. It would be better to use that package's data loading for csv which will better manage memory.


Why doesn't class weight resolve the imbalanced classification problem?

Class weights do help with the imbalance problem ("resolve" seems too much), but upsampling has a certain advantage on it.

If you think about it, downsampling/upsampling the number of samples in each class to balance the dataset is almost exactly the same as using class weights.

For example, say you have a dataset containing 3 samples divided into 2 classes:  and you are training with an MSE loss.

You can choose to upsample the number of samples from class B, which will get you the following cost function over a single epoch:

Here is where the small difference comes into effect, if you are training in a batch gradient descend (a single weights update per epoch), the prediction for the 2 identical B1 samples will be the same, so the loss function can be written as:

Which is exactly the same as using a weighted loss function. However if you are using a mini-batch gradient descend (as most model our days), the 2 different samples may appear in different mini-batches, and so the predictions for them at the same epoch won't be the same (because one of them will pass thru a model that was already updated once).

This is a small difference but sometimes it is important. It means that with weighted classes, the effective learning rate varies between mini-batches. In some cases that can make learning unstable. So, when possible, upsampling is the better approach (practically it produces slightly better results).

The problem is that you can't always upsample/downsample without any worry. If we go back to our example but this time we have a dataset of 5 samples, divided into 2 classes:

Downsampling is problematic - which of the 3 A class samples do we ignore?

Upsampling is problematic - which of the 2 B class samples do we duplicate?

It can be solved by randomly choosing which samples to keep/ignore at each epoch, however with very big datasets, this can lead to slower processing... So the weighted classes is still a valid option.


Memory efficient encoding logic for group categories

There is several techniques that could work for you:

Target Encoder: Works well when there is a high cardinality of a categorical feature.
Ordinal/Label Encoding: Tradition label encoding
Weight Of Evidence:  tells the predictive power of an independent variable in relation to the dependent variable


Does Python have R's tidytext equivalent?

Scikit-learn has a great implementation of latent dirichlet allocation, which I would argue is as straightforward to use as the implementation in tidytext. There’s a tutorial here.

Also, Python has SpaCy, which is slicker than anything R has so far in terms of tooling for NLP pipelines,

I do love R, and I feel it’s still a better language for tidying and processing data than Python. Tidytext is currently nicer than anything in Python in terms of getting data in and out of topic models. However, Python is a lot better resources than R for text mining overall.


Why is convolution filter used instead of correlation filter in CNN?

Convolution (as a term) matches with what we are trying to do in a CNN: we are building a (sort of) filter to extract certain feature from a signal. What is the output of the filter when a given image (vector) is given as input? Best thought of as convolution between the input and the filter.

Correlation (as a term) would be preferred when we are dealing with two signals/inputs. Probably bit misleading applied to CNN feature extraction operation.

I think this is more to do with terminology. The magnitude will be same as you suggest. Flipping happens because convolution is generally calculated after flipping the filter vector representation.


How to execute a git project with different input data?

Apparently the program expects a JSON file (probably) like this:

[
    {
    "id": ".......",
    "mailing_list_url": "......",
    "type_of_recipient": "......",
    "email_address": ".......",
    "message_body": "......",
    "is_response_of": "......"
    },
    {
    "id": ".......",
    "mailing_list_url": "......",
    "type_of_recipient": "......",
    "email_address": ".......",
    "message_body": "......",
    "is_response_of": "......"
    },

    ...

    {
    "id": ".......",
    "mailing_list_url": "......",
    "type_of_recipient": "......",
    "email_address": ".......",
    "message_body": "......",
    "is_response_of": "......"
    }
]

So if your data can be somehow converted to this that would be ideal, I guess. Otherwise you might have to play with the code.

Alternatively you could contact the authors by opening an issue.


Drastic shift in feature importance upon adding other features

There's a good chance that it's a sign of overfitting: the fact that the importance of the features is not stable can be considered as an indication that the model itself is not stable, and this typically happens when it doesn't have enough information in the data to be sure how to use the features. As a result minor variations in the features or data cause the model to change a lot, because it uses features which occur by chance. A way to investigate is to reduce the number of features: if the model becomes more stable this way, it confirms overfitting (and the performance on the test set should be the same or better).
[edited] There is also a possibility that the new features are really useful for the model, causing it to use the whole set of features in a very different way since it can take advantage of new combinations of features. For instance, suppose we have a model which predicts a disease based on features which represent the patient's symptoms, then we add features which represent the patient's age and gender. Let's assume that having a particular symptom at a particular age is a strong indicator of the disease, so this would mean that this symptom feature was not very useful on its own, but becomes more important with the additional feature.
In such a case I would probably expect to observe a significant improvement in performance on the test set when the new features are added.


Display a subset of classes in axes in ggplot

I used this kind of wor around, but if you have another solution, please enlighten me :)

df %>%
  ggplot() +
  geom_bar(mapping = aes(x = N%>% as.factor())) +
  xlab(label = 'N') +
  theme(axis.text.x = element_text(angle = 45, size = 8))


The goal of LM is to learn a probability distribution over sequences
  of symbols pertaining to a language.

That is, to learn $P(w_1,...,w_N)$ (resource).

This modeling can be accomplished by

Predicting the next word given the previous words: $P(w_i | w_1,...,w_{i-1})$, or
Predicting the neighbor words given the center word (Skip-gram): $P(w_{i+k}| w_i), k \in \{-2, -1, 1, 2\}$, or
Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_{i-2},w_{i-1},w_{i+1}, w_{i+2})$, or other designs.

  Does the deep model need the encoder? From the ptb code of
  tensor2tensor, I find the deep model do not contains the encoder.

Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.

  Both with-encoder and without-encoder can do the LM task?

LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.


LSTM for prediction of next location step - help with standardization

Say I want to use min-max scaling, do I scale across ALL samples at once or scale per XYZ space? say i have 100 samples from XYZ_A and 50 from XYZ_B, do i take my min/max from 150 point of both space or not?

Even though your samples come from different databases, they represent the same physical quantity, i.e. 3D coordinates. Since the units of measurement are the same for the samples of both databases, you should apply the scaling over all 150 samples, not separately.

  should I also scale my output labels? and if so should I use them in the initial scale or should I only scale on my train inputs and use that scaler on my outputs?

I wouldn't use the word labels, since you are predicting the next XYZ coordinates, therefore it is a regression problem, not classification. However, you should also apply the scaling function on those values too, since they have the same units of measurement as the input.

  When I want to make a prediction after training, should I scale the data according to my training data?

Exactly, usually you fit a scaling function to your training dataset and you apply the same (already fit) function on the testing dataset.


Multiclass classification in a balanced dataset with one high-priority label

This can be tackled in two places:

Data: as you mentioned, this is done by artificially increasing the number of samples from critical class $cc$. This produces the same effect as data-sets that are naturally imbalanced,
Model: this is generally done by over-penalizing the miss-classification of $cc$ compared to other classes. One place for this modification is the loss function. A frequently used loss function in classification is cross entropy. It can be modified for this purpose as follows. Let $y_{ik} = 1$ if $k$ is the true class of data point $i$, otherwise $y_{ik} = 0$, and $y'_{ik} \in (0, 1]$ be the corresponding model estimation. The original cross-entropy can be written as:
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$
which can be changed to
$$H_y(y')=-\sum_{i}\sum_{k=1}^{K}\color{blue}{w_{k}}y_{ik}log(y'_{ik})$$
For example, by setting $w_{cc} = 10$ and $w_{k \neq cc}=1$, you are essentially telling the model that miss-classifying $1$ member from $cc$ is as punishable as miss-classifying $10$ members from other classes. This is roughly equivalent to increasing the ratio of class $cc$ $10$ times in the training set using method (1).


How to predict multiple images from folder in python

You are looping on a folder to predict each image -
for filename in os.listdir(image_path):
    pred_result = model.predict(images)
    images_data.append(pred_result)
    filenames.append(filename)

But the argument of the predict function is not changing. Its a stacked value defined above as -
images = np.vstack(images)

This same prediction is being appended into images_data
Assuming your prediction is not failing, it means every prediction is the prediction on all the images stacked in the images_data.
So, for every iteration for i in range(len(images_data)):
This images_data[i][0] is returning you the 1st prediction only.
Changing to for i in range(len(images_data[0])): and images_data[i] should work


The reason is that by adding more layers, you've added more trainable parameter to your model. You have to train it more. You should consider that MNIST data set is a very easy-to-learn dataset. You can have to layers with much less number of neurons in each layer. Try $10$ neurons for each to facilitate the learning process. You can reach to $100%$ accuracy.


I think the most important problem you are facing is that you are trying to fit a function that somewhat erratically flips between 0 and 1 as if it were a smooth function. My Mathematical intuition says that won't be numerically stable.

To create a more smooth function you could calculate a running average of your y data and plot it against the corresponding time data.
It should descend as you forget more over a longer period of time and should be easy to plot to demonstrate your intuition.

It has hysteresis/history since initial successes will be remembered. Your fit function would be the integral:

$$\int_0^X e^{-x/a} dx = a - a\cdot e^{-X/a}$$

where $a$ will be the estimator of the forgetting rate.

Other problems could be in the time domain (just leave the unit of time at seconds), or you are mixing time scale/sample scale.

Both your expectation and reality plot should cover the complete time range on the x-axis to make them better comparable.
Maybe you should throw away data with exceptionally large time delta values anyway, as they are likely to disturb the calculations and the plots. The programs from your references remove data > 9 months and < 45 minutes if I recall correctly.

Another domain problem that could interfere is that the data mixed words that are easy to remember and words that are difficult to remember, e.g. 'cat' is in many languages easier to remember than 'butterfly'  (though that could also average out over much measurement data).


Script to convert non-numeric elements that are numbers to numeric

Try avoid looping when working pandas. Either create a function and apply it to your dataframe/series or you can use a lambda like I do below. I basically took your if/then logic with x*10**6 and applied it to a new column

import pandas as pd
df = pd.DataFrame({'Number':'10M 20M 10.5M 30M 100M 1000000 5000000'.split()})

    Number
0      10M
1      20M
2    10.5M
3      30M
4     100M
5  1000000
6  5000000

df['Num_Transformed'] = df['Number'].apply(lambda x: float(x[:-1])*10**6 if str(x).upper()[-1]=='M' else float(x))

    Number  Num_Transformed
0      10M       10000000.0
1      20M       20000000.0
2    10.5M       10500000.0
3      30M       30000000.0
4     100M      100000000.0
5  1000000        1000000.0
6  5000000        5000000.0


PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

In FastText Users FB page a certain Maksym Kysylov answered me " It's not a FastText problem. It's a Spark problem :) When you apply function to Dataframe (or RDD) Spark needs to serialize it and send to all executors. It's not really possible to serialize FastText's code, because part of it is native (in C++). Possible solution would be to save model to disk, then for each spark partition load model from disk and apply it to the data. Something like: df.rdd.mapPartitions(func). And func should: 1. load the model; 2. for record in partition: yield ft.predict(record['text'])."   It works for me and I thank him very much !!!


Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?

Monte Carlo methods can be incremental in an episode-by-episode sense, but not in a step-by-step (online) sense. The term Monte Carlo is often used more broadly for any estimation method whose operation involves a significant random component. It is used specically for methods based on averaging complete returns.

An obvious way to estimate it from experience, then, is simply to average the
returns observed after visits to that state. As more returns are observed, the average should converge
to the expected value. This idea underlies all Monte Carlo methods.


What are the sparse and dense vector ? I cant undestand ,can you explain to me?please.Why do we use for?

A sparse matrix

A sparse matrix is a matrix that is comprised of mostly zero values.
e.g

     1, 0, 0, 1, 0, 0
A =  0, 0, 2, 0, 0, 1
     0, 0, 0, 2, 0, 0

A Dense matrix

Dense matrices with mostly non-zero values, which are referred to as dense matrices.

e.g.

     1, 1, 3, 1, 4, 5
A =  0, 1, 2, 0, 1, 1
     1, 0, 0, 2, 1, 1


what is the best approach to detect small objects with similar shape?

If all objects are observed in the same distance and almost same angle, the relative height and width can be helpful features for recognizing objects with similar shape and different size. By this features different methods like GAN algorithms such as CoGAN and BiGAN may help you in this problem.

It should be noticed that for recognizing the size of the objects the features play more important role than the algorithms.


Why does the classic Neural Network perform better than LSTM in Sentiment Analysis

First of all, I have noticed that you have used sigmoid activation function for your LSTM Dense Layer and in your ANN you used relu, maybe, MAYBE, this can be a reason for your lower performance. That is could be happening because sigmoid functions suffer from two problems:
 - Saturation of gradients: sigmoid functions have tail distributions, meaning that they saturate in this 'flat' regions practically diminishing the gradient to zero and affecting backpropagation/training process.
 - Sigmoid outputs are not zero-centered: This is an issue due the gradient calculation during backpropagation. Either having all enters positivo or negative will add a 'zigzag' effect difficulting the training process.

My comments above are taken from this excellent tutorial that you should read: http://cs231n.github.io/neural-networks-1/
 I tried to summarize it but they did a master work and I think you should read it.

Second, we must consider other factors such as: are you analyzing your train/test/val losses? Maybe your LSTM networks just takes longer to train and reach its minimum. You need to work a little more on these parameters before taking any conclusions. Plot a graph showing your training and validation loss so we can assess if your model is underfitting.

Lastly, I have a question for you: Why should your LSTM network perform better than a simple ANN? Although LSTM + Embeddings are powerful techniques that have gained attention in a lot of fields, essentially NLP, that will be not every task that they beat classical approaches. I myself have tried with different data sets, and depending on the application, simple ML algorithms such as SVM would easily beat the more complex ones, including sentiment analysis.

So to conclude, try these things and let us know your results. Also, if anyone disagrees with my answer, I would like to discuss it. I hope it helps.


Training Restricted Boltzmann Machines (RBMs) using gradient descent

What you're trying to do is to minimize the difference between input and its reconstruction. They may be completely different values and RBM may be working fine! It's due to the fact that Restricted Boltzmann Machine models some probabilistic distribution. Learning rule is to find values of weights to model the distribution of the data. You may clamp a data (visible units) and see a reconstruction, but since hidden units are stochastic units, they would have many values. Even when you set your network free and let the units have some values according to stochastic units rule, after some time you can get a sample from RBM, which should be from your data distribution.


How to get columns from unsorted rows in Pandas? (MALLET)

So one way to do it is to use python slice operators to grab every other value in the line and zip them (along with the filename) into 3-tuples, e.g.:

data = []
malletOutput = open('doc-topics','r').readlines()

for line in malletOutput:
    line = line.split('\t')[1:-1] # slicing out useless leading index and trailing \n
    _id = line[0]

    tIndicies = map(int,line[1::2])
    tVals = map(float, line[2::2])
    topics = sorted(zip(tIndicies, tVals))
    topics = [t + tuple([_id]) for t in topics] # add _id
    for t in topics:
        data.append(t)

Now you have a list of 3-tuples which you can add to a dataframe. A simple pivot from there results in the desired shape:

df = pd.DataFrame(data)
df = df.pivot(index=2,columns=0,values=1)


What to do with large number of collinear variables?

First, if you're doing regression on a time series, you have to check for autocorrelation or your p-values and significance tests are going to be highly inaccurate. Also, the R^2 value you will get will be seriously misleading if you don't account for autocorrelation.

Secondly, if you are using lagged variables, it's highly likely that many of them are going to be correlated.

Collinearity affects linear regression models by making it difficult for the model to determine which coefficient is causing the effect on the dependent variable. The result is that between highly collinear variables, you will have inaccurate p-values and very small coefficients or even coefficients with the wrong signs.

Multicollinearity can be ignored when you are using some of those variables as control variables or when you don't care very much about interpretation; furthermore, collinearity doesn't matter when the variables of interest are not collinear.

Cross-validated has many good discussions on regression as well.


Predict ratings for Item Based Collaborative Filtering

You should at first calculate the similarity between co-rated items (items which both active user and another user rated them). then you can predict the rate of active user to the unrated item.
one of the methods for this work is Slope One


Can I fine tune the xgboost model instead of re-training it?

I see that in the current version of python wrapper of xgboost you can specify file name or existing xgboost model (class Booster) in train function.


You could use HuggingFace's BertModel (transformers) as the base layer for your model and just like how you would build a neural network in Pytorch, you can build on top of it. HuggingFace's other BertModels are built in the same way. For reference you can take a look at their TokenClassification code over here. This hasn't been mentioned in the documentation much and I think it should.

For Tensorflow however, you would have convert the Bert Model into a Keras layer. I haven't really gone through it much or tried it, but I think this blog post does a pretty good job of explaining it.


Why decreasing the number of convolutional layers inside a CNN increases the number of parameters?

Generally removing the number of convolutional layers indeed decreases the number of parameters in a network, but given your situation I suspect you are using fully connected layers after the convolutional layers in your network. When removing some convolutional layers it means that the image/tensor size before being passed to your fully connected layers is larger and contains more pixels. Since each neuron in the fully connected layer is connected to all pixels in the layer before it the increase in the number of parameters caused by the larger image size is larger than the decrease in the number of parameters caused by the removal of the convolutional layers (leading to an overall increase in the number of parameters in your network).


Anomaly detection without any knowledge about structure

Graph Embeddings

You try to use graph embeddings. These would offer the ability to map your graph-based data into a single latent space, at based on the vertices, the edges or the graph schema as a whole. This has become somewhat of an exciting research topic within research over the last few years; combining graph analysis with modern embedding techniques.

Here is one survey paper that list approaches to these embeddings, along with the authors' own libraries for both static graphs and dynamic graphs. Looking through the results of that survey paper (they perform some becnhmarks towards the end), it would seem that the most promising model would be:

node2vec - if you want to classify the nodes as cleanly as possible (reference implementation)
Structural Deep Network Embedding - if you want to re-construct the graph (reference implementation)

Anomaly Detection

As to the end goal of anomaly detection, I am not sure which of the embedding methods would be best suited. It might be a case as to whether you want to preserve more local or global structure (as with the trade-off between methods such as t-SNE and PCA).

I think you can imagine what to try out with the embeddings in order to detect anaomalies. There might be more clues in the papers linked above as to which way researchers tend to go for certain problems.

Given the hierarchical nature of your graph data, I would be tempted to start by looking into using Hierarchical Density Based Spatial Clustering with Applications of Noise (i.e. HDBSCAN) - the main library specifically includes a few few methods of outlier detection! Here a citation from the linked paper:

  Based on these definitions, we can devise an algorithm DBSCAN* (similar to DBSCAN) that conceptually finds clusters as the connected components of a graph in
  which the objects of X are vertices and every pair of vertices is adjacent if and only
  if the corresponding objects are ε-reachable w.r.t. user-defined parameters ε and mpts.
  Noncore objects are labeled as noise.


There are a couple of problems and things you might want to add to your existing script.

Below I separate your example data into two NumPy arrays:

input values x
labels y

It is also important to make sure they are of type float32, because Tensorflow will complain if you pass it integers (as they otherwise would be interpreted).

The following works for me, the model trains to completion:

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers

syslog_data = [
    [302014, 0, 0, 63878, 30, 3, 1],
    [302014, 0, 0, 3891, 0, 0, 0],
    [302014, 0, 0, 15928, 0, 0, 2],
    [305013, 5, 0, 123, 99999, 0, 3],
    [302014, 0, 0, 5185, 0, 0, 0],
    [305013, 5, 0, 123, 99999, 0, 3],
    [302014, 0, 0, 56085, 0, 0, 0],
    [110002, 4, 2, 50074, 99999, 0, 4],
]

print(tf.VERSION)
print(tf.keras.__version__)

x = np.array([arr[:-1] for arr in syslog_data], dtype=np.float32)
y = np.array([arr[-1:] for arr in syslog_data], dtype=np.float32)

model = tf.keras.Sequential()
# Adds a densely-connected layer with 64 units to the model:
model.add(layers.Dense(64, activation="relu"))
# Add another:
model.add(layers.Dense(64, activation="relu"))
# Add a softmax layer with 10 output units:
model.add(layers.Dense(10, activation="softmax"))

model.compile(optimizer=tf.train.AdamOptimizer(0.001), loss="categorical_crossentropy", metrics=["accuracy"])

model.fit(x, y, epochs=10, steps_per_epoch=30)


Encoding categorical data with pre-determined dictionary

Assuming you have a pandas DataFrame and one mapping per column, with all mappings stored in a 2-level dict where the keys of the first level correspond to the columns in the dataframe and the keys of the second level correspond to the categories:
{'fruit': {'banana': -1, 'apple': 1}, 'color': {'yellow': -1, 'red': 1}}

Then, you can do the following:
encoded_data = data.apply(lambda col: col.map(mappings[col.name]))

[EDIT] if have columns for which you don't have a mapping, you can do one of the following:
data.update(data[list(mappings)].apply(lambda col: col.map(mappings[col.name])))

or if you want it in a new dataframe (eg to keep the dataframe with the original values):
encoded_data = data.copy()
encoded_data.update(data[list(mappings)].apply(lambda col: col.map(mappings[col.name])))


Are there some research papers about text-to-set generation?

Check out these papers below and google keywords: Multi-label Classification.

X-BERT: eXtreme Multi-label Text Classification with BERT
HAXMLNet: Hierarchical Attention Network for Extreme Multi-Label Text Classification
SGM: Sequence Generation Model for Multi-label Classification
Ranking-Based Autoencoder for Extreme Multi-label Classification
AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification

Also check out the code below:

https://github.com/chenyuntc/PyTorchText


How to find the input variables for a classification problem?

Two approaches come to mind:

Use all of them and perform feature selection to identify indicative features. Algorithms include Information Gain and Mutual Information.
Hand-pick some features you deem indicative intuitively, or delete the ones you deem non-indicative.

Generally, you have some sort of hypothesis that you are testing where you attest the capability of (a subset of) your features (and the algorithm you'll use) to uniquely identify the label/class they belong to. The features you use are one of the most important things in the classification task. So it is good to spend time to intelligently find out which features contribute.

For this task, you do have a lot of possible features so the automatic feature selection approach might be better.


LSTM Model for predicting the minutely seasonal data of the day

When represented at a minute level, the pattern is going to recur after almost 1440 minutes. So your LSTM needs to learn a really long term dependency here. And LSTMs are not great at such long term dependencies.

I'll have a contrary opinion here. Drop neural networks, fit a good linear model to get a benchmark. If the linear model (example outlined below) beats your LSTM error rate currently, invest in building a better (generalised) linear regression model instead. Also, start with a more coarse definition of output (predicting total for next 15 minutes / 1 hour).

Create hand-rolled features for your data. Start with traffic at the same hour of the day yesterday, traffic in same hour of day for past 7 days, traffic on the same hour of the day on same weekday last week.
Add more advanced features by defining better aggregation window than hour. Something like 6 - 9 PM, 12 midnight - 6 am.
Add features related to recent data (ratio of cumulative traffic today compared to cumulative traffic till same time yesterday).
Build a suitable linear model on this data, benchmark the error rate. If the traffic is being measured in number of hits, use poisson regression instead of simple linear regression.

Complex neural net models like LSTM are best used when hand-rolled features are extremely difficult to think and implement. For many time series applications, generating features yourself is intuitive, simple to achieve (most libraries like Python pandas have great date-time functionality) and good first step. It is much better than teaching LSTM to differentiate between morning and afternoon.

François Chollet's book on deep learning has a chapter on similar application. He tackles the problem of predicting weather in particular time window based on past data. He shows that many complex neural net architectures find it hard to beat simple baseline like averaging from same time period in past.


Ways to Encode context for text classification?

A Recursive Neural Network (RNN) is best suited to solve your problem. And if you want to take a step further try Recursive Neural Tensor Network. You can generate compositions for entire sentences using RNN both for your content and context data. This will help you map your sentences to a higher dimensional space and as an output you will get numerical values (or embeddings) both for your content and context. Feeding these values into a simple logistic classifier will do your task. For a brief overview on RNNs take a look at this lecture by Richard Socher.


Based on your screenshot, it's quite clear that the accuracy isn't 0.0 since the first two predictions match the true labels. So something must be wrong with how the accuracy is calculated.

If you go to sklearn's documentation, you'll see that accuracy_score requires 1-d arrays while it seems that you are feeding it 2-d arrays. My guess is that right now, it compares the elements of your arrays and checks if they are identical. Because you feed a 2-d array, it checks whether all predictions match, which unless you are perfectly correct, will always yield you 0.0.

Doing the following should fix your issue:

acc=accuracy_score(y_train[0], y_pred[0])


Perceptron learning rate irrelevant in convergence

Your conclusion is correct.

Note that the classifier is of the form of

$$f(x; \theta) = \operatorname{sign}(\theta^Tx)$$

while the update rule is

$$\theta^{(k+1)}=\theta^{(k)}+\eta y_tx_t$$ which only occurs when there is a misclassification and we only care about the in the classification but

$$\operatorname{sign}(\theta^Tx)=\operatorname{sign}(\eta \theta^Tx)$$

as long as $\eta$ is positive.

It only scales $\theta^Tx$ but our concern is just the sign, hence the learning rate is irrelevant.

Remark: You might like to note the assumptions being used in the proof though. For instance, there is this assumption that $\theta^*$ exists. It is assumed that the data can be perfectly separated by a hyperplane.


How to train a Text Based data for a Machine Learning problem?

Your problem looks more like a ranking problem than a classification problem to me. Have you tried a more naive method, like a 1-NN "classifier" with unigram text representation, Tf-Idf term weighting and a cosine similarity metric? It's far from the state of the art but it tends to give rather good results in retrieval and recommendation tasks.


Change rate of cross validation data, after training

Due to stochastic nature of NN training, the best epoch may vary upon each restart. In other words, at epoch 8, each of (the best, under-fitted, over-fitted) cases may happen. However, if you train multiple times and the best model is consistently found at (or around) 8th epoch, it is safe to say 8th epoch gives the best model away from under- or over-fitting, thus definitely validation set can be added to training set to improve the performance.

A more solid approach would be to plot the effect of training size (10% up to 90%) on the best epoch and the validation error. This means producing two plots (training size, the best epoch) and (training size, validation error), where each point is an average over multiple restarts. This way you can better find (1) the best epoch, and (2) the degree to which the added validation set will going to boost the performance on unseen test data, i.e. extrapolating the validation error for training size 100%.

It is possible that performance goes flat after for example 70% of training set, implying that adding the validation set has no gain.


There are several general approaches to this but almost all of them compare the values to some baseline and make a decision whether the individual value is close enough.

You can compare the similarity of strings using different methods e.g. I often use the Levenshtein distance which basically measures how many characters you would have to change to convert word a into word b. By grouping all words with a low enough Levenshtein distance you already identify all words that should be the same.

It is even easier if you have a dictionary of "correct values" in which case you would compare each value to all entries in the dictionary and assign it the value with the lowest levenshtein distance.

Using some basic text analysis you could also identify the most common spelling mistakes in your data and eliminate them based on replacement rules.


Solving multi label image classification using TimeDistributed dense layer

There is no need to repeat any features. What you want to do is change your activation function from softmax to sigmoid, otherwise leaving everything else as it would normally be done for binary classification.

...
model = Sequential()

model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dense(5)) # whatever the number of classes is
model.add(Activation('sigmoid'))
...

You will then need to simplify your Y tensor from N x 5 x 2 to N x 5, e.g. with Y = Y[:, :, 0].

This may or may not be why you get all zeros, but it is certainly a more standard approach to a multi-label classification than using RepeatVector and TimeDistributed. Something else to look at is how much you are training, 5 epochs with only 1600 examples is not a lot. Similarly, a Dense(512) is very large for your small dataset size. Maybe reduce that down to something like Dense(16)?


Although you can generate text in this way - sampling from a RNN trained to predict next character or word - it will not be meaningful. At best it will be grammatically accurate (in terms of nouns, verbs, adjectives etc) but semantic nonsense. It would not be a summary, except by lucky accident.

To generate summaries using LSTM directly, you would need to train a network with real examples of summary inputs and outputs. This would be similar task to machine translation, but much harder due to variability in size of the inputs. It is unlikely that you will find enough training data to test the idea fully, and it is not clear that such a direct approach can yield acceptable results even with large amounts of training data.

In fact text summarisation is not a solved problem by any means. There are deep learning approaches, such as Google Brain team's effort using TensorFlow, which you could study to get some sample code and a sense of state-of-the-art. This approach uses an attention model to extract apparently informational content (i.e. content that would have low probability of appearing in some assumed generic document, thus is assumed to be interesting due to standing out). It is possible to use a trained LSTM to build such an attention-based model - the intuition is that the parts of the document that the already-trained LSTM is least able to predict are likely to contain noteworthy information.


Is there a way to get gini index values for every node in rpart model?

Gini impurity can be calculated as $1-p_{1}^2-p_{2}^2$ for each node. For example, if node 1 contains 40% '1' and 60% '0', gini = 1 - 0.4^2 - 0.6^2. The information of node size n, number of '0' dev are stored in model$frame. The Gini for each node could be calculated with node size n and number of '0' dev in model$frame:
frame <- model$frame
frame[['gini']] = 1 - (frame[['dev']] / frame[['n']])^2 - (1 - frame[['dev']] / frame[['n']])^2

frame[,c('var','n','dev','gini')]
>      var  n dev      gini
> 1     x3 10   5 0.5000000
> 2 <leaf>  4   1 0.3750000
> 3 <leaf>  6   2 0.4444444

The Gini improvment for each split is calculated by weighted difference between parent and children nodes.
frame[['improve']] = NA
for (i in 1:nrow(frame)) {
  if (frame[i,'var'] == '<leaf>') next

  ind = which(rownames(frame) %in% (as.numeric(rownames(frame)[i])*2+c(0,1)))
  frame[i,'improve'] = frame[i,'n']*frame[i,'gini'] - frame[ind[1],'n']*frame[ind[1],'gini'] - frame[ind[2],'n']*frame[ind[2],'gini']
}

frame[,c('var','n','dev','gini','improve')]
>      var  n dev      gini   improve
> 1     x3 10   5 0.5000000 0.8333333
> 2 <leaf>  4   1 0.3750000        NA
> 3 <leaf>  6   2 0.4444444        NA

#comparing with
model$splits
>    count ncat   improve index  adj
> x3    10    2 0.8333333     1 0.00
> x2    10    2 0.2380952     2 0.00
> x2     0    2 0.7000000     3 0.25


Difference between sklearn’s “log_loss” and “LogisticRegression”?

The metrics module by Scikit-learn implements functions assessing prediction error for specific purposes (Regression, classification, etc.). Model is the algorithm that actually does the classification/regression/clustering (as per need) for you.

Log-loss measures the accuracy of, say a classifier. It is used when the model outputs a probability for each class, rather than just the most likely class.

EDIT: Thanks to @mapto for suggesting documentation reference:

sklearn.metrics.log_loss,
and
sklearn.linear_model.LogisticRegression