Supervised multiclass classification : is ANN a good idea ? or use other classifiers?

In general, if you only have $5$ to $6$ scalar features. I would simply start with easy methods like logistic regression and discriminant analysis. I would guess that you should be able to get good results.

You should also look at the distribution of the scalar features. Maybe you can derive new features that are helpful in separating. A simple visual way to see if it is possible to separate the classes by a linear hyperplane is to use a principal components analysis (short PCA) and extract 2 or 3 factors. Then use these factors to visualize your datapoints (maybe use a random sample from the training data set and repeat this three or more times to see if the trend is there in all random samples that you looked at). You should see if the classes are well sparable.

If you see that your model performance is not good enough I would try out decision trees (these are very interesting as they allow you to see how the decisions of your classifier are made).

Depending on the model performance you could also use neural networks. I personally would rather try it with simpler models first. Neural networks are very powerful function approximators, but you will most likely not be able to extract some useful information about the relationship between the features and the classes of products.


What are available methods for modeling startup survival rates?

If the data you show are the only data that you have, then the Markov Chain is really boring: it is a linear chain, going from Round A to Round B to Round C, and all those states are connected to a base state (which is Death, or something).

You can directly calculate the transition probabilities from the data you have, since the number of companies that reached round N are all companies that could have reached round N (there is no alternative path).  The death probability at the previous stage is (1 - $P_{reaching N}$)

In [1]: raw_data = """
   ...: Company 1: Seed Round, Series A Round
   ...: Company 2: Seed Round, Series A Round, Series B Round
   ...: Company 3: Seed Round, Series A Round, Series B Round
   ...: Company 4: Seed Round, Series A Round, Series B Round, Series C Round
   ...: Company 5: Seed Round
   ...: Company 6: Series A Round, Series B Round
   ...: Company 6: Series A Round
   ...: """
In [2]: data_lines = raw_data.splitlines()[1:]
In [6]: key_vals = {}
In [12]: for line in data_lines:
             key, val = line.split(':')
             key = key.strip()
             vals = [v.strip() for v in val.split(',')]
             key_vals[key] = vals

In [13]: key_vals
Out[13]:
{'Company 1': ['Seed Round', 'Series A Round'],
 'Company 2': ['Seed Round', 'Series A Round', 'Series B Round'],
 'Company 3': ['Seed Round', 'Series A Round', 'Series B Round'],
 'Company 4': ['Seed Round',
  'Series A Round',
  'Series B Round',
  'Series C Round'],
 'Company 5': ['Seed Round'],
 'Company 6': ['Series A Round']}

In [14]: transitions = ['Seed Round', 'Series A Round', 'Series B Round', 'Series C Round']

In [19]: for transition in transitions:
    summed = 0
    for company, rounds in key_vals.iteritems():
        if transition in rounds:
            summed += 1
    prob = float(summed) / float(len(key_vals.keys()))
    death_prob = 1 - prob
    print "From previous to %s: probability %s" % (transition, prob)
    print "Death rate at %s: probability %s" % (transition, death_prob)

From previous to Seed Round: probability 0.833333333333
Death rate at Seed Round: probability 0.166666666667
From previous to Series A Round: probability 0.833333333333
Death rate at Series A Round: probability 0.166666666667
From previous to Series B Round: probability 0.5
Death rate at Series B Round: probability 0.5
From previous to Series C Round: probability 0.166666666667
Death rate at Series C Round: probability 0.833333333333

However, if you had some more features of each company, like the amount of money they received at each stage, or the profit they were making, then you could train a decision tree, e.g. with this implementation in sklearn, that told you, in simple words, "if a company arrived at round X with at least Y dollars raised and at least Z dollars of profit, then they are passing to the next round with 0.XX probability".  Which is, I think, what you are aiming at.


Group neural networks outputs using Keras/Tensorflow

Using the keras functional API will get you what you need.

I'm assuming that you are currently using the standard keras sequential model API, which is simpler but also restricts you to a single pipeline. When using the functional API, you do need to keep track of inputs and outputs, instead of just defining layers.

For the example in your question:

from keras.layers import Input, Dense
from keras.models import Model

# Left side sub-model:
L1 = Input(shape=(2,))
L2 = Dense(2, activation='softmax')(L1)

# Right side sub-model:
R1 = Input(shape=(2,))
R2 = Dense(2, activation='softmax')(R1)

# Combining them together:
merge = concatenate([L2, R2])

# Some additional layers working on the combined layer:
merged_layer_1 = Dense(4, activation='relu')(merge)

# Output Layer:
output = Dense(2, activation='softmax')(hidden1)

# Defining the model:
my_model = Model(inputs=[L1, R1], outputs=output)

The depths of the layers are made up, but you should get the general idea.


How to calculate Precision and Recall using confusion matrix in Matlab?

if yHat are your predictions and yval are your y true then

tp = sum((yHaT == 1) & (yval == 1));
fp = sum((yHaT == 1) & (yval == 0));
fn = sum((yHaT == 0) & (yval == 1));

precision = tp / (tp + fp);
recall = tp / (tp + fn);
F1 = (2 * precision * recall) / (precision + recall);


You got outliers. Standard scaler scales each attribute independently to center at zero and unit variance. Either deal with your outliers or use some more robust scaler. Try instead to plot each attribute with separate box plot.


Text Classification on a very small data set with a lot of classes

Accuracy is not the best measure for imbalanced data. Prefer precision and recall.
Do undersampling/oversampling to get equal samples for each class and try XGBoost.
Or else you can use SVC with class weights, give lower class weight to classes with more samples and vice versa.


weighted cross entropy for imbalanced dataset - multiclass classification

If you are looking for just an alternative loss function:

Focal Loss has been shown on imagenet to help with this problem indeed.

Focal loss adds a modulating factor to cross entropy loss ensuring that the negative/majority class/easy decisions not over whelm the loss due to the minority/hard classes.

I would look into using that is it seems to be really promising.

Link To Focal Loss Paper:  https://arxiv.org/pdf/1708.02002.pdf


What would be the best way to structure and mine this set of data?

How best could I organize this data, in what type of database?

A simple relational database should do, but you could also use a "fancy" graph database if you want. One table for the users, and one for the "interactions". Each interaction would have foreign key columns for the two participants, labeled winner and loser, and the number of the episode the interaction it occurred.

  Also any ideas on the best way to visually represent this data?

A graphical representation for social network analysis suggests itself. Here are some papers and a subreddit for inspiration. In your case, there is a concept of competition with clear winners/losers, so you could make your graph directed. Have the characters be the nodes, and add directed edges from the winning party to the losing party for each interaction. Collapse repeated interactions, etc. This approach would let you quickly identify overall winners and losers, as well as simply who interacts with whom.


How to properly use approximate_predict() with HDBSCAN clusterer for text clustering (NLP)?

Why do you recompute the distance matrix?

Just compute the 1x5000 vector directly for all new points. You can even do this in batches, then feed one row at a time to the predictor.


The cost function of the Logistic Regression derived via Maximum    Likelihood  Estimation:

If y = 1 (positive): i) cost = 0 if prediction is correct (i.e. h=1), ii) cost $\rightarrow \infty $ if $h_{\theta}(x)\rightarrow 0$.
If y = 0 (negative): i) cost = 0 if prediction is correct (i.e. h=0), ii) cost $\rightarrow \infty$ if $(1-h_{\theta}(x))\rightarrow 0$.

The intuition is that larger mistakes should get larger penalties.
Further readings, 1,2,3,4.


Updating One-Hot Encoding to account for new categories

I think you have two options:

Automate your train/test pipeline so that one-hot encoding is part of
it. If new categorical variables are introduced, they can be featured
in the training dataset even if not very prevalent. This would
introduce some bias if the nature of the TV show distribution has
changed over time (e.g. 20 years ago there weren't as many options)
but I don't necessarily think it is a show stopper.
If new possibilities are introduced over time but for whatever reason you can't  retrain, then you should omit using that new value. This has its own disadvantages because in your example, it would be a TV show with no network.


Is plant classes or any crop is present in Imagenet datasets?

Yes, there are many kinds of plants in Imagenet. image-net.org has a search feature. For example, http://image-net.org/search?q=tree


What are Machine learning model characteristics?

I am using neural networks in industrial fault diagnosis and dynamic system modeling, so I would consider:

is it a static or dynamic system? Depending on it you may choose other network structure (for example RNN for dynamic systems)

what is the quality of quantity data? Depending on it you may choose training algorithm (maybe bayesian regularization for noisy data). SVR modeling would be good for high-quality data with a lots of samples. It would be way faster learning than neural networks,

are there any delays in the system? Sometimes performance might be improved by adding delays to model. Cross-correlation might be applied to check optimal delays or diagnose some echos in signals

sometimes correlation analysis could reduce dimimensionality of a problem. Computing correlations might help selecting inputs of the model. Sometimes PCA is a nice idea to get rid of curse of dimensionality.
But depending on field of use those questions could be different.


The entities W , U and V are shared by all steps of the RNN and these are the only parameters in the model described in the figure. Hence number of parameters to be learnt while training = $dim(W)+ dim(V)+ dim(U)$.

Based on data in the question this = $ n^{2}+ kn + nm$.

where,

n - dimension of hidden layer
k - dimension of output layer
m - dimension of input layer


fluctuating values for validation set only

Thanks for updating the post, this level of fluctuation in the validation set is a lot less dramatic than before and appears to be similar to regular fluctuation I have seen in my experience. Kudos that you have also managed to prevent the model from overfitting.


Imbalanced training set vs smaller balanced training set?

Why should I choose one over the other?

You sould prepare a common validation set out of your dataset and try out each and every method on your dataset.
Following are the methods that I know to handle imbalanced datasets. -

Use weighted cross-entropy loss (as you mentioned)

You can assign weights to your loss such that it will penalize more to the smaller classes and the less to larger classes. Many frameworks have a very easy way to do this.
In Scikit-learn you can look out for class_weight parameter. For eg - random forest
Here is how you can use this in Pytorch
Here is how you can use this in Keras

Use focal loss

Originally proposed for object detection, but we can also use this for any other use case. Read more about it here
Here is how you can use this in Pytorch for multi-class classification
Here is how you can use this in Keras

Over Sampling and Under Sampling

There are so many techniques in this, check out imblearn a dedicated library just to deal with imbalanced datasets.

Create a separate model for small classes

If you have some classes that have very small number of instances, you can consider creating a separate classifier for these small classes (called small_classifier for eg). You can group together these small clases under a single class (called small_class for eg) so that your main classifier will classify small_class with all other big classes in the dataset. And if your main classifier encounters any instance of small_class, it will pass it to small_classifier, which will predict the actual class for the small_class instance. This technique can give you accuracy boosts are now main classifier does not need to deal with very small classes, and insted small_classifier will be looking just at these small classes.


How to plot multiple models val and traing acc/loss curve from csv files?

Here are the steps I solve my problem:

Save Statistical records (Training-Validation Accuraciers, Training-Validation Recall, Training-Validation Precisions, Training-Validation F1) from CSV files of this following code tf.keras.callbacks.CSVLogger after training each fold (I trained for the 3 Folds).

And load those csv files in the list:
    import pandas as pd
    l = ["/Records_of_Fold_2.csv" , "/Records_of_Fold_3.csv"]

After that, Plot using matplotlib :
for (k,i) in enumerate(l):
data = pd.read_csv(i)
plt.figure(str(k))
plt.xlabel("x"+str(k))
plt.ylabel("y"+str(k))
plt.plot(data["epoch"],data["f1_score"])
plt.plot(data["epoch"],data["loss"])
print(k)
plt.show()


Simplest way to build a semantic analyzer

Is there any simpler approach?

Very unlikely, semantic similarity is a very complex problem related to Natural Language Understanding (NLU). You could look at the techniques used for textual entailment, Question Answering and summarization.

Simple methods like the baseline system proposed in the github link, but they don't really try to analyze the semantics.


What is the use of applying img_to_array() after cv2.imread()

This is because of the data type you provide to the imshow() function.
Check the documentation:

The function may scale the image, depending on its depth:
If the image is 8-bit unsigned, it is displayed as is. If the image is
16-bit unsigned or 32-bit integer, the pixels are divided by 256. That
is, the value range [0,255*256] is mapped to [0,255]. If the image is
32-bit or 64-bit floating-point, the pixel values are multiplied by
255. That is, the value range [0,1] is mapped to [0,255].

So when you use cv2.imread() it reads data uint8, that is 8-bit unsigned format thus imshow() displays the original data.
But when you use img_to_array() it converts data to the float. Thus imshow() multiplies it with 256. It expects the float values in normalized form, that is all the pixels are divided by 256. That is the reason for such a display.
This is how cv2.imread() data looks like...

and its data type...

This is how img_to_array() data looks like...

and its data type...

If you divide your image by 1.0 to convert it to the float...
import cv2
# from keras.preprocessing.image import img_to_array
image = cv2.imread("car.jpg")
image = image/1.0
cv2.imshow("Divided by 1.0", image)
cv2.waitKey(0)

...this is what you will get:

Same as img_to_array.
But it you divide image with 256.0 as follows:
import cv2
# from keras.preprocessing.image import img_to_array
image = cv2.imread("car.jpg")
image = image/256.0
cv2.imshow("Divided by 256.0", image)
cv2.waitKey(0)

You get the original image since imshow() multiplies the float with 256.

So what you need is to divide your img_to_array() output by 256 or convert it to the uint8.


Math behind L2 Regularization for Logistic Regression

Your question is really about the method of Lagrange multipliers in constrained optimization, not logistic regression per se. The gist of it is that a constrained optimization problem can be recast as an unconstrained optimization problem by adding a term, called the regularizer, and vice versa. The sphere comes from recasting the unconstrained problem into a constrained one; recall that a constant $L_2$ norm defines a hypersphere.


Neural network accuracy in Torch depends on compute power?

Available compute power does not directly affect the accuracy of a neural network. If your different runs of the network have:

identical architecture and meta-params
identical code (including library code)
all training data is identical
all stochastic parts of training use the same random seed and generator
all data types are identical precision (e.g. all vectors and matrices are 32-bit or 64-bit floats)

then the behaviour of neural network training in each run is fully deterministic and repeatable. Having a faster processor will just get you to the result faster*.

The most likely difference between your tests is due to not seeding the random number generators used in the training the process. For you this includes weight initialisation, possibly train/test split and possibly shuffling training data in each epoch. As you did not use any regularisation, then accuracy of the trained network can vary quite a bit due to over-fitting.

To verify this, you can train a second or third time on each CPU. I expect you will see a lot of variation in final accuracy, regardless of which machine you run it on.

* This does mean that having a faster machine can result in you having a more accurate final network in practice when you are tuning the parameters, because you can try more variations of meta-params with multiple training sessions.


Will my validation loss eventually go down?

When training neural network, you need to train many epochs in order for the model to learn enough to generalize. What is happening now the model is changing weights to fit the data that it observes (lower training loss) but does not generalize (as evidenced by high variance on validation data).
There are a couple of options:

Downsampling your data. Take only the most meaningful data to train.
Scale up training. Either use graphics processing units (GPUs) or distributed training.
Run training for longer.


Scraping mixed elements and passing to SQL

One way would be to declare that column as a character type. I think this would be a poor solution as it forces all interpretation and processing of the data back onto the application. Better would be to declare Val2 as datatype sql_variant. At least then the DBMS has a fighting chance of interpreting the value correctly.

Since each of those datatypes represents a different attribute with unique semantics it would be better to separate them into their own columns. SQL Server has the TRY_PARSE function which allows you to discover the datatype of each value and assign it to its proper column.


Colorfulness, specifically managing overly gray output, was touched on in the pix2pix paper, down in Section 4.2. It might be worthwhile to try out their formulation of a conditional GAN, as their results indicate that their objective function was useful in generating a more diverse set of colors.


Where in practice can we see following information propagation model

Assuming the central node is at level 0 (root), this graph becomes a tree. Now it is easy reason about the different types of trees that can be formed and different types of social networks that can be modelled.

In order for the graph to be a tree, communication between nodes has to be restricted, such that the tree structure is not violated (e.g. two nodes on the same level cannot communicate). I can't think of any social network where communication is naturally restricted in this way, but such a social network can be created artificially.

Examples of such artificial social networks would commonly follow a chain-of-command structure. For example in a military setting, the official information flows would have a tree structure.


How can I figure out section headings in a document?

It seems you are lucky and the section headings are XML-tagged. So you can use XSLT transformations or Python lxml.etree to get rid of them or to extract them.


Approximating multi-variable function with neural network in python

One problem that I see is that notice that sine is a function that takes value from $-1$ to $1$ but sigmoid function takes value from $0$ to $1$.

Hence you are being penalized whenever the sine value takes negative value.

You might like to try to change your last layer to a tanh layer or alternatively, rather than predicting sine directly, predict $\frac{\sin(2x_1+x_2)+1}{2}$ first.

I manage to achieve MSE of $0.228686$ using the tanh modification. Of course, you can still try to tune other parameters and try other stuff to improve the model.

import numpy as np
import matplotlib.pyplot as plt
import math

class Layer:
    """
    Represents a layer (hidden or output) in our neural network.
    """

    def __init__(self, n_input, n_neurons, activation=None, weights=None, bias=None):
        """
        :param int n_input: The input size (coming from the input layer or a previous hidden layer)
        :param int n_neurons: The number of neurons in this layer.
        :param str activation: The activation function to use (if any).
        :param weights: The layer's weights.
        :param bias: The layer's bias.
        """

        self.weights = weights if weights is not None else np.random.rand(n_input, n_neurons)
        self.activation = activation
        self.bias = bias if bias is not None else np.random.rand(n_neurons)
        self.last_activation = None
        self.error = None
        self.delta = None

    def activate(self, x):
        """
        Calculates the dot product of this layer.
        :param x: The input.
        :return: The result.
        """

        r = np.dot(x, self.weights) + self.bias
        self.last_activation = self._apply_activation(r)
        return self.last_activation

    def _apply_activation(self, r):
        """
        Applies the chosen activation function (if any).
        :param r: The normal value.
        :return: The "activated" value.
        """

        # In case no activation function was chosen
        if self.activation is None:
            return r

        # tanh
        if self.activation == 'tanh':
            return np.tanh(r)

        # sigmoid
        if self.activation == 'sigmoid':
            return 1 / (1 + np.exp(-r))

        return r

    def apply_activation_derivative(self, r):
        """
        Applies the derivative of the activation function (if any).
        :param r: The normal value.
        :return: The "derived" value.
        """

        # We use 'r' directly here because its already activated, the only values that
        # are used in this function are the last activations that were saved.

        if self.activation is None:
            return r

        if self.activation == 'tanh':
            return 1 - r ** 2

        if self.activation == 'sigmoid':
            return r * (1 - r)

        return r

class NeuralNetwork:
    """
    Represents a neural network.
    """

    def __init__(self):
        self._layers = []

    def add_layer(self, layer):
        """
        Adds a layer to the neural network.
        :param Layer layer: The layer to add.
        """

        self._layers.append(layer)

    def feed_forward(self, X):
        """
        Feed forward the input through the layers.
        :param X: The input values.
        :return: The result.
        """

        for layer in self._layers:
            X = layer.activate(X)

        return X

    def predict(self, X):
        """
        Predicts a class (or classes).
        :param X: The input values.
        :return: The predictions.
        """

        ff = self.feed_forward(X)

        # One row
        if ff.ndim == 1:
            return np.argmax(ff)

        # Multiple rows
        return np.argmax(ff, axis=1)

    def backpropagation(self, X, y, learning_rate):
        """
        Performs the backward propagation algorithm and updates the layers weights.
        :param X: The input values.
        :param y: The target values.
        :param float learning_rate: The learning rate (between 0 and 1).
        """

        # Feed forward for the output
        output = self.feed_forward(X)

        # Loop over the layers backward
        for i in reversed(range(len(self._layers))):
            layer = self._layers[i]

            # If this is the output layer
            if layer == self._layers[-1]:
                layer.error = y - output
                # The output = layer.last_activation in this case
                layer.delta = layer.error * layer.apply_activation_derivative(output)
            else:
                next_layer = self._layers[i + 1]
                layer.error = np.dot(next_layer.weights, next_layer.delta)
                layer.delta = layer.error * layer.apply_activation_derivative(layer.last_activation)

        # Update the weights
        for i in range(len(self._layers)):
            layer = self._layers[i]
            # The input is either the previous layers output or X itself (for the first hidden layer)
            input_to_use = np.atleast_2d(X if i == 0 else self._layers[i - 1].last_activation)
            layer.weights += layer.delta * input_to_use.T * learning_rate

    def train(self, X, y, learning_rate, max_epochs):
        """
        Trains the neural network using backpropagation.
        :param X: The input values.
        :param y: The target values.
        :param float learning_rate: The learning rate (between 0 and 1).
        :param int max_epochs: The maximum number of epochs (cycles).
        :return: The list of calculated MSE errors.
        """

        mses = []

        for i in range(max_epochs):
            for j in range(len(X)):
                self.backpropagation(X[j], y[j], learning_rate)
            if i % 10 == 0:
                mse = np.mean(np.square(y - nn.feed_forward(X)))
                mses.append(mse)
                print('Epoch: #%s, MSE: %f' % (i, float(mse)))

        return mses

    @staticmethod
    def accuracy(y_pred, y_true):
        """
        Calculates the accuracy between the predicted labels and true labels.
        :param y_pred: The predicted labels.
        :param y_true: The true labels.
        :return: The calculated accuracy.
        """

        return (y_pred == y_true).mean()

def my_func(x1, x2):
    return [math.sin(2*x1+x2)]

n = 40
np.random.seed(4)
x1_low, x1_up = -3, 3
x2_low, x2_up = -1, 3
x1s = np.random.uniform(x1_low, x1_up, size=n)
x2s = np.random.uniform(x2_low, x2_up, size=n)

Xs = []
for _x1 in x1s:
    for _x2 in x2s:
        Xs.append([_x1, _x2])

Zs = [my_func(_x1, _x2) for _x1, _x2 in Xs]

# Define test data
x1_pred = np.random.uniform(x1_low, x1_up, size=n)
x2_pred = np.random.uniform(x2_low, x2_up, size=n)
Xs_pred = []
for _x1, _x2 in zip(x1_pred, x2_pred):
    Xs_pred.append([_x1, _x2])
actual_ys = [my_func(_x1, _x2) for _x1, _x2 in Xs_pred]

# Train and test neural network
alpha = 0.001
nn = NeuralNetwork()
nn.add_layer(Layer(2, 5, 'tanh'))
nn.add_layer(Layer(5, 1, 'tanh'))
errors = nn.train(Xs, Zs, alpha, 300)
print('Accuracy: %.2f%%' % (nn.accuracy(nn.predict(Xs_pred), actual_ys) * 100))


Tableau: Clustering based on value-range for map coloring

You can, as an example, create a binned field for the measure. The value range can be specified in the tooltip. I used a single color based continuous palette since (population) total is a continuous field.

[Edited]
Based on your comment, I tested with a fixed dimension based binned field for the continuous measure. At least in my example file, the values make more sense now with the new binned field. I see China at 16M population, India at 14M and so on.

If this solution is relevant to your use case, you will need to be careful with dimensions included/excluded in the fixed calculation.


Is it possible to design a deep CNN model on a small size image dataset

According to your description of the data, it is highly probable that training any neural network of reasonable size will overfit the training data.

One option is to apply any possible regularization approach; these come to mind:

Use L1/L2 weight regularization.
Reduce the number of parameters as much as you can (less layers, less number of filters).
Use dropout.

Also, try any data augmentation trick: rotate images, mirror images, crop and scale images, etc.


Why do we calculate partial derivative of Error w.r.t output of Neural Network during backpropagation?

The partial derivative is used precisely because it separates concerns about how the value is calculated (from all the other parameters and outputs in the network) from how the value affects the output. This is purely by definition so you can do the calculation.

The "normal" derivative of the error function can be expressed as the sum of any "complete" set of independent partial derivatives. There are a few such sets possible - in a simple feed-forward network one for each layer plus one for all the weights combined.

So in essence you are calculating the "normal" derivative w.r.t. the network weights, but due to the nature of the problem this is done by calculating the partial derivatives in multiple steps.

Caveat: I am probably not using those maths terms 100% accurately. For instance, I am ignoring the role of the training data and treating it as a constant.


What should be the ratio of True vs False cases in a binary classifier dataset?

Ideal true vs false ratios don't exist and they should reflect the the reality the best they can, you can always remove negatives if the ratio is too skewed to improve training speed though. Let me explain it with an example. Ads CTR is as old as the internet and it's skewed to less than 1% positives vs. plus 99% negatives. Yet, data scientists prefer to train it on the entire dataset because many negatives will include information that models couldn't find otherwise. They might not provide a lot of information as a positive one but they are still somewhat important. There are approaches where CTR ratios get artificially rebalanced by sampling in case you want a swifter training and it will still work. In your case, positives are 0.4% which resemble CTR on ads so you can: gather more data to increase the number of positives in order to better understand what makes an article interesting. In case that is not possible trying ensembles which often improve prediction performance.

Clustering is an unsupervised approach so you would be losing information by doing so (training labels) besides, sentence embeddings (representations)  of one big cluster of negatives and a tiny cluster of positives do not convey information as well as word embeddings which have already been trained on billions of documents.

In addition, running k-means on categorical variables will yield anomalous clusters because it's meant to be used with continuous variables. You can find more information about the topic on the following links:

Kmeans: Whether to standardise? Can you use categorical variables? Is Cluster 3.0 suitable?

My data set contains a number of numeric attributes and one categorical

Kaggle

Why does K means clustering perform poorly on categorical data The weakness of the K means method is that it is applicable only when the mean is defined one needs to specify K in advance and it is unable to handle noisy data and outliers

Therefore, you should use high dimensional embeddings or representations to cluster meanings together, this has been explored in word meanings but for sentences or articles, a vector representation becomes more complicated to implement. One possible approach is the Word Movers’ Distance but there are many more possible approaches, you should google them. In addition a non-linear clustering algorithm such as t-sne will  probably yield better results than k-means using the embeddings approach.
A better approach is:

to use multiple models and compare their performance on this dataset. I have the impression that there will be certain keywords that make articles interesting, so a bag of words will still be helpful, even as a starter model.

Use feature engineering. Your model might be overloooking important features, such as article length, reading time, number of paragraphs, ratio of complex words  (measured by length), etc. Feature engineering is always important in case you haven't used it yet.

Use pretrained embeddings. CNN and RNN models can use pretrained embeddings such as GloVe, Word2Vec or FastText so you use better representations plus other complex layers later on in the architecture. This is extremely important to increase accuracy.

Use metrics to measure improvement and ranks to check for the best predicted interesting articles.


Welcome to the site!

what you where thinking is right, the new version plot looks a bit different, to get exactly the same plot as above you can use the following code:

import pandas as pd
import matplotlib.pyplot as plt
df3 = pd.read_csv('df3')
#%matplotlib inline (This line is to be used in jupyter notebooks to display plots)

ax = plt.subplot(111)
ax.hist(df3['a'],ec='black')

#this is x axis limiter since your value ranges in between 0 to 1 I gave these values
plt.xlim([0,1])
plt.show()

Outcome:

Let me know if you need anything else.

If this is what you were looking for, then you can accept the answer by clicking on the green tick mark.


Multilabel classification for a learning to rank application

In the multinomial mode, the docs specify that the outputs of coef_ and intercept_ are as you are seeing them: one output for each target class. The underlying model is three logistic regressions, whose outputs are softmax'ed (or with mode ovr, simply normalized).

As to the broader question, since your three output classes are ordered, you might benefit from using that information. Either just perform regression (assumes that the numeric 0,1,2 are meaningful) or use "ordinal regression."


Can we quantify how position within search results is related to click-through probability?

you might want to look at this paper
Predicting Clicks: Estimating the Click-Through Rate for New Ads

  Whenever an ad is displayed on the search results page, it has
  some chance of being viewed by the user. The farther down the
  page an ad is displayed, the less likely it is to be viewed. As a
  simplification, we consider the probability that an ad is clicked on
  to be dependent on two factors: a) the probability that it is viewed,
  and b) the probability that it is clicked on, given that it is viewed:


Does anyone have memory utilization benchmark for random forest and xgboost?

Try to use memory_profiler.
All what you need to do is to decorate your training function:

@profile
def train_xgb():
   gb = xgb.XGBClassifier(learning_rate=0.1, n_estimators=100, subsample=0.8, max_depth=6)
   gb.fit(new_trainX,new_trainY)


Why this model does not converge in keras?

I'm not totally sure exactly what you're doing with your scoring equation, but the first thing you need to look at is your loss function. Categorical Crossentropy is for multilabel classification, and you're trying to predict a float value.

So, you should have your network output be a single value (and don't squash it through a sigmoid unless the range of your function is (0,1)).  You should be using a regression loss function - I'd definitely start with mean squared error. Check out the regression example here (under Regression Predictions) for some sample code.

Edit: Further to discussion below, the network cannot predict the index of the highest-valued node because each node's score has no dependency on the node's position among the 1000 nodes being scored. As the probability of the best node being in each position is the same, there is no optimum prediction to be learned.


The way to do this in PySpark is to use groupBy and pivot. Since you don't want to do any actual aggregation, just the pivot, you can use first here.

from pyspark.sql.functions import first

(df.groupBy("Customer")
  .pivot("product")
  .agg(first("rating"))
  .fillna(0))

pivot will give nulls when there is no value so fillna needs to be used as well to give the wanted result.


Should the minimum value of a cost (loss) function be equal to zero?

Saying that the well-known loss functions, like MSE or Categorical Cross Entropy, has a global minimum value equal to zero is flawed . The idea behind loss function is to measure how near the model predictions are to the actuals(in case of a regression). Now ideally , you would want your model to predict exactly equal to the actuals . In that case only , we get loss equal to zero. Otherwise , loss is non zero almost all the time . If you remember the loss function for a linear regression setting ,

We need to minimise  so that the predictions  can be as close to the actuals as possible . For that the derivative of  should be zero . It doesn't matter if   is zero or non zero . Graphically , for a cost function like this
 , you want to reach the point  where the derivative is zero.


Minimal (numerical) example for simple neural net

Please check the work of Andrew Trask in this book.

Most detailed work on this that I have seen.

grokking Deep Learning


Is autocorrelation of residuals a problem in machine learning?

Yes, autocorrelation in residuals is a problem, but this is essentially because it is a clear illustration that there was more learnable information in the process you are modelling but your model missed it.
In the unlikely event that you have two equally performant models but one shows significant autocorrelation (you can test for this using the Durbin-Watson test as suggested in Noah Weber’s answer), this suggests neither model is working as well as we might hope (the autocorrelated model has failed to predict some predictable patterns and the other model is failing in some other way as its predictive power isn’t any better).
If you have two models that have different residuals but both are beating a naïve baseline, you’ve probably got models that will ensemble well.


Does LSTM without delayed inputs work as a deep net?

LSTMs are RNN with memory cell
It can be difficult to train standard RNNs to solve problems that require learning long-term temporal dependencies.
LSTM units include a 'memory cell' that can maintain information in memory for long periods of time. A set of gates is used to control when information enters the memory, when it's output, and when it's forgotten. This architecture lets them learn longer-term dependencies.
So without the delayed inputs a LSTM is simply a RNN.

As you can see here, RNN has a recurrent connection on the hidden state. This looping constraint ensures that sequential information is captured in the input data.
Check this for more details


Python Script using pandas to plot histograms between the features

cols = list(df.columns.values)
for val in cols:
        df.groupby('Survived')[val].hist(alpha = 0.2)
        plt.show()

just add plt.show() in the end it would create histogram for all columns


I don't think there is need for very complex methods here, I would take a look at several granularity levels and take means for these levels. One important thing is if all the households are in the same area/timezone. After establishing this I would first take the general mean and see per household how they compare to this, then do this for every halfhour segment of the day to get rid of time-of-day seasonality, do this for day of the week, month of the year etc. It really depends on what you want to know what to do with this data.


How would you categorise email subjects to find similar emails?

You'll probably have to experiment a bit with different approaches.  Let me outline two different kinds of approaches you could try.

Topic modelling

You could try applying unsupervised topic modelling to your subject lines.  LDA is probably the most widely used method.

Topic modelling tries to find a limited number of "topics", and assigns each subject line to one or more "topics" based on the words in the subject line.

Clustering

You could try using clustering.  Broadly, you would find some way map each subject line to a feature vector, and then apply some unsupervised clustering method.  There are many options for each of those two steps.  To get feature vectors, you might try any of the existing word embeddings; e.g., you might try word2vec.  For clustering, there are many, many clustering algorithms; e.g., you might try k-means.  I recommend you do a little reading on these topics and then experiment with them a bit.

Caveats

A warning.  Don't set your expectations too high. Subject lines are typically very short, and that will make it hard for these techniques to find clusterings.  In other words, you are operating in a regime that's known to be hard for existing NLP and ML techniques.

If you can find any other features that might assist with clustering similar emails (e.g., the identify of the sender? the mailing list it was sent on, if any?), including that information might improve the quality of the clusters.


Is Overfitting a problem in Unsupervised learning?

Overfitting happens when the model fits the training dataset more than it fits the underlying distribution. In a way, it models the specific sample rather than producing a more general model of the phenomena or underlying process.

It can be presented using Bayesian methods. If I use Naive Bayes then I have a simple model that might not fit either the dataset or the distribution too well but of low complexity.

Now suppose that we use a very large Bayesian network. It might end up not being able to gain more insight and use the complexity to model the dataset (or even just trash).

So, overfitting is possible in unsupervised learning.

In PCA we start with a model in the size of the dataset. We have assumptions about the way the data behaves and use them to reduce the model size by removing parts which don't explain the main factors of variation. Since we reduce the model size one could expect to benefit always.

However, we face problems.
First, a model in the size of the dataset is extremely large (given such a large size you can model any dataset). Compressing it a bit is not enough.

Another problem is that it is possible that our assumptions are not correct. Then we will have a smaller model be it won't be aligned with the distribution. Again, it this case it might overfit or just not fit the model.

Though that, PCA is aimed to reduce the dimensionality, what lead to a smaller model and possibly reduce the chance of overfitting. So, in case that the distribution fits the PCA assumptions, it should help.

To summarize, overfitting is possible in unsupervised learning too. PCA might help with it, on a suitable data.


Images Dataset Augmentation with fixed parameters of Crop & Rotation Angle

If you need to work on images using Python, the preferred library is PIL.  Here I show a function to do the modifications you have delimited.  This code makes no effort to manage multiple files, or how to name the files converted.  But it does show how to modify the images in the ways you have asked.

Code:

from PIL import Image

def modify_image(im, crop_corner=None, crop_percent=5, rotate=0):
    """
    Crop and rotate an image

    :param im: base image
    :param crop_corner: crop corner 0-3 clockwise from top left
    :param crop_percent: crop percentage of size
    :param rotate: rotate degrees (rotate after crop)
    :return: a modified image
    """
    x, y = im.size
    if crop_corner is not None:
        crop_x, crop_y = [int(i * crop_percent / 100.0) for i in (x, y)]
        size_x = x - crop_x
        size_y = y - crop_y
        if crop_corner in (1, 2):
            crop_x = 0
        if crop_corner in (2, 3):
            crop_y = 0
        im = im.crop((crop_x, crop_y, crop_x + size_x-1, crop_y + size_y-1))
        im = im.resize((x, y), Image.BICUBIC)

    im = im.rotate(rotate, expand=True)
    return im

Test Code:

test_image = Image.open('test_image.png')
new_image = modify_image(test_image, rotate=90)
new_image.show()

To save the image see the (Docs)


Can machine learning models treat a vector as a whole feature to learn

To avoid having too much features produced by a OneHot, you can either :

Transform your data before your OneHot, regrouping different classes in intervals you created yourself (for example, reducing your 128 features to 30 classes you created yourself) then applying OneHot
Use CategoryEncoder (Such as TargetEncoder, James-Stein, LeaveOneOut, WeightOfEvidence) : this will make stats and transform the classes into a number, higher if the class is well known (appearing often) and tend to give more True prediction than the average


I guess you have a confusion here. The None part represents the number of samples.
For example if you have a neural network with architecture 100-50-10, it means that you have

(None,100) : input layer shape
(100,50) : shape for weights connecting input to hidden layer
(None,50): shape for hidden layer given by (None,100)*(100,50) matrix multiplication
(None,50): shape after nonlinearity application.
(50,10):  shape for the shape matrix between hidden and output layer
(None,10) : output layer shape (None,50)*(50,10) matrix multiplication

So if youre feeding a single input sample the shapes would be:
(1,100)[Input] => (1,100)(100,50) = (1,50)[Hidden Layer] => (1,50)*(50,10)=(1,10)[Output Layer]


how to create a searchable tree on Persian text?

This is an answer with tire tree:

readindg data:

#readindg stopword data
stopwords = pd.read_csv('STOPWORDS',header=None)

tire tree:

#creating tire tree
class TrieNode:

    # Trie node class
    def __init__(self):
        self.children = [None]*15000

        # isEndOfWord is True if node represent the end of the word
        self.isEndOfWord = False

class Trie:

    # Trie data structure class
    def __init__(self):
        self.root = self.getNode()

    def getNode(self):

        # Returns new trie node (initialized to NULLs)
        return TrieNode()

    def _charToIndex(self,ch):

        # private helper function
        # Converts key current character into index
        # use only 'a' through 'z' and lower case

        return ord(ch)-ord('!')

    def insert(self,key):

        # If not present, inserts key into trie
        # If the key is prefix of trie node,
        # just marks leaf node
        pCrawl = self.root
        length = len(key)
        for level in range(length):
            index = self._charToIndex(key[level])

            # if current character is not present
            if not pCrawl.children[index]:
                pCrawl.children[index] = self.getNode()
            pCrawl = pCrawl.children[index]

        # mark last node as leaf
        pCrawl.isEndOfWord = True

    def search(self, key):

        # Search key in the trie
        # Returns true if key presents
        # in trie, else false
        pCrawl = self.root
        length = len(key)
        for level in range(length):
            index = self._charToIndex(key[level])
            if not pCrawl.children[index]:
                return False
            pCrawl = pCrawl.children[index]

        return pCrawl != None and pCrawl.isEndOfWord

Example of use:

# Input keys (use only 'a' through 'z' and lower case)
keys = list(stopwords.loc[:,0])

output = ["Not present in trie",
        "Present in trie"]

# Trie object
t = Trie()

# Construct trie
for key in keys:
    t.insert(key)

print("{} ---- {}".format("از",output[t.search("از")]))

Output:

از ---- Present in trie


Steps in exploratory methods for mild-sized data with mixed categorical and numerical values?

You can get a reasonably good approximation of steps for exploratory data analysis (EDA) by reviewing the EDA section of the NIST Engineering Statistics Handbook. Additionally, you might find helpful parts of my related answer here on Data Science SE.

Methods, related to EDA, are too diverse that it is not feasible to discuss them in a single answer. I will just mention several approaches. If you are interested in applying classification to your data set, you might find information, mentioned in my other answer helpful. In order to detect structures in a data set, you can try to apply principal component analysis (PCA). If, on the other hand, you are interested in exploring latent structures in data, consider using exploratory factor analysis (EFA).


Plotting two pandas data frame columns against each other

You need to groupby to deal with multiple vote counts:

  df.groupby('timestamp').sum().plot(x='timestamp', y='vote_count')


Regression error increase after shuffing data

You need to shuffle the whole dataset together before separating the features (X) and target variable (y). This is the only reason I can think of for getting this error.


How can we effectively measure the impact of our data decisions

In A/B testing, bias is handled very well by ensuring visitors are randomly assigned to either version A or version B of the site.  This creates independent samples drawn from the same population.  Because the groups are independent and, on average, only differ in the version of the site seen, the test measures the effect of the design decision.

Slight aside: Now you might argue that the A group or B group may differ in some demographic.  That commonly happens by random chance.  To a certain degree this can be taken care of by covariate adjusted randomization.  It can also be taken care of by adding covariates to the model that tests the effect of the design decision.  It should be noted that there is still some discussion about the proper way to do this within the statistics community.  Essentially A/B testing is an application of a Randomized Control Trial to website design.  Some people disagree with adding covariates to the test.  Others, such as Frank Harrel (see Regression Modeling Strategies) argue for the use of covariates in such models.

I would offer the following suggestions:

Design the study in advance so as to take care of as much sources of bias and variation as possible.
Let the data speak for itself.  As you get more data (like about searches for David Beckham), let it dominate your assumptions about how the data should be (as how the posterior dominates the prior in Bayesian analysis when the sample size becomes large).
Make sure your data matches the assumptions of the model.


Naive bayes, all of the elements in predict_proba output matrix are less than 0.5

No, your classifier can label text. It doesn't do it well but it is still almost 2 times better than random (for 7 classes, random will get you ~0.15 accuracy).

Looking at the test set is not enough. You need to create the same confusion matrix for you training set.

If the results you will get for the test set are similar in magnitude than maybe your model is too simple for the task or maybe you haven't trained it long enough.

If the results of the test set are good, than you might have a generalization problem (overfitting), which means that you need to increase the regularization during training. It also might mean that your training set comes from a different distribution than your test set.


How to identify recurring patterns in this set of transactional data

I don't know the model you use, but I would suggest the perhaps old-fashioned feature engineering approach:

In each set of transactions, calculate the duration between two consecutive transactions for every pair of consecutive transactions (i.e. difference between the dates). Then add the standard deviation of these durations as a a feature, so that a very regular transaction set should get a near zero value.
Same idea with the amount of the transaction: the standard deviation will be zero if the amount never changes.
Additional ideas:

number of distinct amounts across all transactions
number of distinct duration values between transactions in days, weeks, months, years.
standard deviation of the sum by day/week/month/year


Two recent papers use a Deep learning architecture called CharWNN to address this problem. CharWNN was first used to get state of the art results (without handcrafted features) on Part of Speech (POS) tagging on an English corpus.

The second paper by the same author uses the same (or similar) architecture for predicting whether a word belongs to 10 Named Entity classes, with apparent state of the art results.


How to convert regression into classification?

To add to the number of methods you can use to convert your regression problem into a classification problem, you can use discretised percentiles to define categories instead of numerical values. For example, from this you can then predict if the price is in the top 10th (20th, 30th, etc.) percentile. These values you can easily find out using Python's numpy.percentile(X[i], percentile) method, for example.
This type of labelling will place thresholds with respect to the distribution of the data.


Training an RNN with examples of different lengths in Keras

This suggests that all the training examples have a fixed sequence length, namely timesteps.

That is not quite correct, since that dimension can be None, i.e. variable length. Within a single batch, you must have the same number of timesteps (this is typically where you see 0-padding and masking). But between batches there is no such restriction. During inference, you can have any length.

Example code that creates random time-length batches of training data.

from keras.models import Sequential
from keras.layers import LSTM, Dense, TimeDistributed
from keras.utils import to_categorical
import numpy as np

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(8, return_sequences=True))
model.add(TimeDistributed(Dense(2, activation='sigmoid')))

print(model.summary(90))

model.compile(loss='categorical_crossentropy',
              optimizer='adam')

def train_generator():
    while True:
        sequence_length = np.random.randint(10, 100)
        x_train = np.random.random((1000, sequence_length, 5))
        # y_train will depend on past 5 timesteps of x
        y_train = x_train[:, :, 0]
        for i in range(1, 5):
            y_train[:, i:] += x_train[:, :-i, i]
        y_train = to_categorical(y_train > 2.5)
        yield x_train, y_train

model.fit_generator(train_generator(), steps_per_epoch=30, epochs=10, verbose=1)

And this is what it prints. Note the output shapes are (None, None, x) indicating variable batch size and variable timestep size.

__________________________________________________________________________________________
Layer (type)                            Output Shape                        Param #
==========================================================================================
lstm_1 (LSTM)                           (None, None, 32)                    4864
__________________________________________________________________________________________
lstm_2 (LSTM)                           (None, None, 8)                     1312
__________________________________________________________________________________________
time_distributed_1 (TimeDistributed)    (None, None, 2)                     18
==========================================================================================
Total params: 6,194
Trainable params: 6,194
Non-trainable params: 0
__________________________________________________________________________________________
Epoch 1/10
30/30 [==============================] - 6s 201ms/step - loss: 0.6913
Epoch 2/10
30/30 [==============================] - 4s 137ms/step - loss: 0.6738
...
Epoch 9/10
30/30 [==============================] - 4s 136ms/step - loss: 0.1643
Epoch 10/10
30/30 [==============================] - 4s 142ms/step - loss: 0.1441


What is the difference between an autoencoder and an encoder-decoder?

According to section 2.3 of this paper:

  Auto-encoders are special cases of encoder-decoder models in which the input and output are the same.

The same section of the paper describes encoder-decoder as follows:

  Encoder-Decoder models are a family of models which learn to map data-points from an input domain to an output domain via a two-stage network: The encoder, represented by an encoding function z = f(x), compresses the input into a latent-space representation; the decoder, y = g(z), aims to predict the output from the latent space representation.


It's analogous to an extremely specific convolution step on a 2D image. That is, it's analogous to an $N \times I$ image with one feature map (e.g. a black & white image, if we're talking about the input), and you choose to use $J$ filters of size $F \times I$ which spans the entire width of the image and only strides along the length to create $J$ feature maps of size $(N - F + 1) \times 1.$ The restrictions here are that:

-The "image" (or current layer you're working on) necessarily only has one feature map

-The filters span the entire width of the image, so they don't stride along that direction, and the resulting feature maps have a width of $1.$

You could then re-interpret the $(N-F+1) \times 1$ resulting feature maps as a single $(N-F+1) \times J$ image, which then creates the exact same restrictions for the next convolutional layer (or more restrictions if you're doing pooling).

So yes, the analogy is there, but it's only analogous to a very restricted class of convolution on 2D image arrays, which I don't think is very useful. The kind of application it would be used for is black and white images for which you don't care about translation invariance along one of the dimensions.

The reason CNNs on images are much more flexible than this analogy allows for is because the input objects (and each respective hidden layer) of an image CNN is a 3D array, not a 2D array as it is in this case. The third dimension is the feature maps (the RGB values for the input image, for example). Then you could allow the filter to be of any size in both dimensions, and stride along both directions. I would prefer to keep the text interpretation when it comes to 2D inputs of a CNN.


How to draw line plot with multiple lines on Tableau?

Use "Measure Values" in your Measures Tab. Pull it in Row, and then under the Measures Marks, pull in all the measures you want to display.


Adding to what Himanshu Rai said, you should be careful not to oversample before the StratifiedKFold, once you risk to put the same sample in both the training and testing folds check this, (where you took your image from) and that doesn't really evaluate your model's capacity to data that it never saw. What i did was to use StratifiedKFold, and then oversample or SMOTE (or whatever you want) all the training folds separately, by only looking at that fold's training data. Then use all those folds to validate your models.


When is a Neural networked considered to be overfitted?

A model is said to have overfitted when it performs awesomely on the training set (error's low), but when tested on test set, it fails badly. This might be due to bad sampling distribution over our training and tes divisions, so that model might not have seen this kind of data or due to using many irrelevant features in the samples than we were supposed to etc.

You cannot really say that the cat is dead until you open the box. So until you test, you would never know.

  I read somewhere that if the number of input parameter equate the number of hyper parameter, that would be an indication of overfitting

This statement can be explained in general, not only for neural nets. Assume you know the relationship is linear for your data, but you got some error and you are not happy with this, then say you add an extra term say $ax^2$ to this equation and the model now has an extra parameter to tune and this makes our line into curve. For a neural net we basically start by random weights from a distribution and then we keep reducing them until we feel like our training error is cool.

Assuming that you are using a multi-layer perceptron (fully connected), pruning is one of the techniques to remove weights from the model which might not be so contributing still making sure your model still behaves well with lesser number of parameters. SVD on this weight matrix can be one of the useful techniques to do this. But how many weights to keep depends on how many singular values you keep for reconstructing the matrix. I heard that early stopping can reduce overfitting, but I had never really done this, so can't really give my opinions about this.

Hope this helps.


Can TF turn a given graph into a recursive one?

I finally managed to find a solution by using tf.scan (see this question on stackexchange)

The trick is to create a model object that calls tf.scan in one of its methods using a step function that operates on the elements of a list of inputs.

The function tf.scan takes care of feeding the output of the step function back into itself for the next iteration, and it is quite flexible so you might be able to do what you need without the need to implement a full RNN from scratch (say, with tf.nn.raw_rnn).

Below, the minimal working example that I had posted over there:

import tensorflow as tf
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2' # to avoid TF suggesting SSE4.2, AVX etc...

class Model():
    def __init__(self):
        self._inputs = tf.placeholder(shape=[None], dtype=tf.float32)
        self._predictions = self._compute_predictions()

    def _step(self, old_state, new_input):
        # ---- In here I will write a much more complex graph ----
        return old_state + new_input

    def _compute_predictions(self):
        return tf.scan(self._step, self._inputs, initializer = tf.Variable(0.0))

    @property
    def predictions(self):
        return self._predictions

    @property
    def inputs(self):
        return self._inputs

def test(sess, model):
    sess.run(tf.global_variables_initializer())
    print(sess.run(model.predictions, {model.inputs: [1.0, 2.0, 3.0, 4.0]}))

test(tf.Session(), Model())


What does the alpha and beta hyperparameters contribute to in Latent Dirichlet allocation?

The Dirichlet distribution is a multivariate distribution. We can denote the parameters of the Dirichlet as a vector of size K of the form ~$\frac{1}{B(a)} \cdot \prod\limits_{i} x_i^{a_{i-1}}$, where $a$ is the vector of size $K$ of the parameters, and $\sum x_i = 1$.

Now the LDA uses some constructs like:

a document can have multiple topics (because of this multiplicity, we need the Dirichlet distribution); and there is a Dirichlet distribution which models this relation
words can also belong to multiple topics, when you consider them outside of a document; so here we need another Dirichlet to model this

The previous two are distributions which you do not really see from data, this is why is called latent, or hidden.

Now, in Bayesian inference you use the Bayes rule to infer the posterior probability. For simplicity, let's say you have data $x$ and you have a model for this data governed by some parameters $\theta$. In order to infer values for this parameters, in full Bayesian inference you will infer the posterior probability of these parameters using Bayes' rule with $$p(\theta|x) = \frac{p(x|\theta)p(\theta|\alpha)}{p(x|\alpha)} \iff \text{posterior probability} = \frac{\text{likelihood}\times \text{prior probability}}{\text{marginal likelihood}}$$ Note that here comes an $\alpha$. This is your initial belief about this distribution, and is the parameter of the prior distribution. Usually this is chosen in such a way that will have a conjugate prior (so the distribution of the posterior is the same with the distribution of the prior) and often to encode some knowledge if you have one or to have maximum entropy if you know nothing.

The parameters of the prior are called hyperparameters. So, in LDA, both topic distributions, over documents and over words have also correspondent priors, which are denoted usually with alpha and beta, and because are the parameters of the prior distributions are called hyperparameters.

Now about choosing priors. If you plot some Dirichlet distributions you will note that if the individual parameters $\alpha_k$ have the same value, the pdf is symmetric in the simplex defined by the $x$ values, which is the minimum or maximum for pdf is at the center.

If all the $\alpha_k$ have values lower than unit the maximum is found at corners

or can if all values $\alpha_k$ are the same and greater than 1 the maximum will be found in center like

It is easy to see that if values for $\alpha_k$ are not equal the symmetry is broken and the maximum will be found near bigger values.

Additional, please note that values for priors parameters produce smooth pdfs of the distribution as the values of the parameters are near 1. So if you have great confidence that something is clearly distributed in a way you know, with a high degree of confidence, than values far from 1 in absolute value are to be used, if you do not have such kind of knowledge than values near 1 would be encode this lack of knowledge. It is easy to see why 1 plays such a role in Dirichlet distribution from the formula of the distribution itself.

Another way to understand this is to see that prior encode prior-knowledge. In the same time you might think that prior encode some prior seen data. This data was not saw by the algorithm itself, it was saw by you, you learned something, and you can model prior according to what you know (learned). So in the prior parameters (hyperparameters) you encode also how big this data set you apriori saw, because the sum of $\alpha_k$ can be that also as the size of this more or less imaginary data set. So the bigger the prior data set, the bigger is the confidence, the bigger the values of $\alpha_k$ you can choose, the sharper the surface near maximum value, which means also less doubts.

Hope it helped.


Dealing with a dataset with a mix of continuous and categorical variables

To clarify, you mean mixed variables in one column? e.g. ABC123

If yes, you create two additional columns: one with categorical and one with numerical values. Afterward, you can encode them (one hot encoding not always necessary).

Detailed explanation in Chapter 11 "Feature Engineering Mixed Variables":

https://www.udemy.com/feature-engineering-for-machine-learning/learn/v4/content


How to detect the match precision of OneVsRestClassifier

Yes, of course.

Assuming that you have used sklearn's OneVsRestClassifier and so you have a decision function for example a Support Vector Classifier with say linear kernel. Use set_params to change probability key to True, default is False. Use this in the OneVsRestClassifier classifier and then go with the inbuilt function predict_proba like

from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC
mod = OneVsRestClassifier(SVC(kernel='linear').set_params(probability=True)).fit(samples,classes)
print mod.predict_proba(np.array([your_sample_vector]).reshape(1,-1))

Edit:

You can use your old LinearSVC with decision_function to find the distance from the hyperplane and convert them to probabilities like

mod = OneVsRestClassifier(LinearSVC()).fit(sample,clas)
proba = (1./(1.+np.exp(-mod.decision_function(np.array(your_test_array).reshape(1,-1)))))
proba /= proba.sum(axis=1).reshape((proba.shape[0], -1))\
print proba

Now you don't need tuning the parameters, I guess. :)


Get the average time between first and 2nd call (postgresql)

If you had a table containing only the first call details and another table containing only those of the second call you'd be able to join them, right? Then the two times would be in separate columns. You don't have to copy data or split the table to achieve this. It can be done logically by referencing the table twice in the query and using a self-join.
select
    AVG(second_call.time_of_call - first_call.time_of_call)
from TheTable as first_call
inner join TheTable as second_call
    on first_call.customer_id = second_call.customer_id
where first_call.call_attempt = 1
and second_call.call_attempt = 2;

Logically I split the table into two, one called first_call and one called second_call. The former only contains rows for call_attempt 1 and the latter for call attempt 2. This is enforced in the WHERE clause. I'm assuming each customer will only ever have one row for each call attempt number.
Then I join these tables on the customer_id. I use an inner join so only customers that have both first second calls will be processed. It would be a good idea to profile the data to see how this affects the quality of the calculation.
You'll likely want to change the subtraction inside the AVG() to get the format you need.
If run-time performance is critical it may be more run-time efficient to write this using window functions rather than a self-join, depending on what indexes exist and the cardinalities.


When creating an RNN, we generally assume there is some temporal correlation, for example: that the data has a time-series nature, like the price of a stock. We might consider a window of 30 days as the timestep for a single sample.

When creating input data for an LSTM layer, you need to consider how many timesteps are included in one of your samples, and put the data into that shape. The input shape should be (number_of_sample, num_timesteps, num_features):

Take a look at this answer for more information.

Your data then might be able to be reshaped:

new_shape = data.reshape((None, 100, 4))       # new input shape to LSTM

None means you let the number of samples be whatever it must in order to fit your original data shape.
100 means you pass 100 sequential datapoints/timesteps as a single sample
4 is your number of independent variables (features)

That reshape might throw an error if your data cannot be reshaped to make the dimensions you want. You might need to trim your data (i.e. leave some of teh oldest samples out)


How to fill df with data from another df - Python

First answers is close, the only thing you need to do is merge both data frames using two fields. You do not need both data frames to have the same length at all
Try:
pd.merge(df1, df2, on = ["number","trans"], how = "left")


ReLU for combating the problem of vanishing gradient in RNN?

Can't we use ReLU instead of tanh in RNNs for solving vanishing gradients too rather than opting for a more complex architecture?

Yes you can. I'd expect it to help a little with vanishing gradients, but not be as effective generally as a LSTM is. Also, without some kind of bounded function in the feedback loop, a basic RNN using ReLU will suffer very easily from exploding values, resulting in very large errors as the feedback is run over multiple time steps - so you may need some adjustment like gradient clipping to account for that.

If you already have a problem to analyse, have prepared your data and decided on a top level architecture (e.g. how many recurrent layers, whether this is a seq2seq type model etc), then it should be relatively easy to compare LSTM with a basic RNN using ReLU. It is likely different hyperparameters would suit each architecture so you may need to do a search, and you could compare best possible results for each model, or look at things such as number of parameters or training time.

I expect many researchers have already tried this comparison, although the examples I found on a quick search appeared to use tanh for the simple RNN.

Even using ReLU in simple RNN architecture, I suspect that by most measures and in most problems, the LSTM or GRU architectures will perform better. There will be some edge cases where RNN performs better, such as if your goal is to get to a certain accuracy quickly during training and not necessarily to get the best accuracy in the end.


Explainable AI, how did the computer classified Text?

There are libraries like ELI5 or LIME, which can provide explanations for text classification, here is a link to an example:
https://eli5.readthedocs.io/en/latest/tutorials/black-box-text-classifiers.html#textexplainer
import eli5
from eli5.lime import TextExplainer

te = TextExplainer(random_state=42)
te.fit(doc, pipe.predict_proba)
te.show_prediction(target_names=twenty_train.target_names)


How differential semi-gradient Sarsa updates estimated average reward?

In a continuing task, TD error can increase w to infinity unless its expected value is zero. By subtracting the TD average estimation, the expected value of our update value is zero and w cannot go to infinity. TD error is a biased estimation of the average reward ( with a bias that goes to zero as the number of updates goes to infinity assuming that every state can be reached from every state since the average reward is independent of the state action combination with which we start).


conversion of dictionary to json to be sent to requests.post

To build your data use sth like:
import copy, json

def make_objects(ref_obj, df):
    objects = []
    for i in range(len(df[df.columns[0]].values)):
        cobj = copy.deepcopy(ref_obj)
        cobj['data_sample'] = {}
        for col in df:
            cobj['data_sample'][col] = int(df[col].values[i]) if df[col].dtype == np.int64 else (float(df[col].values[i]) if df[col].dtype == np.float64 else df[col].values[i])
        objects.append(cobj)
    return objects

df = pd.DataFrame({'sub1':[10,20,30,40],'sub2':[5,10,15,20],'sub3':[1,2,3,4]})
objects = make_objects({'name':"abc",'id':"AB10",'address':"some_address"}, df)
# in case you want to json-encode the returned objects
json_output = json.dumps(objects)


Is there a replacement for small p-values in big data?

There is no replacement in the strict sense of the word.  Instead you should look at other measures.

The other measures you look at depend on what you type of problem you are solving.  In general, if you have a small p-value, also consider the magnitude of the effect size.  It may be highly statistically significant but in practice meaningless.  It is also helpful to report the confidence interval of the effect size.

I would consider this paper as mentoned in DanC's answer to this question.


Multi-label text classification with minimum confidence threshold

Looks like the common approaches to multi-class classification actually solve this challenge.

Building individual Naive Bayes Classifiers with only the training data for a single label is insufficient - we must also the include data from other labels as "everything else".

See Text-Classification-Problem, what is the right approach?


It is basically a matter of convention, which becomes a bit more clear if you write the whole thing in terms of elements, rather than vectors. Consider

$$ \theta^T \theta = \sum_{n=1}^N \theta_n \theta_n = \sum_{n=1}^N \theta_n^2 $$

What is usually meant if you write $\frac{df}{d\theta}$ is that you take the gradient of a scalar $f$, i.e. you get a vector $\frac{df}{d\theta}$ where each element $i$ is the derivative with respect to the respective coordinate $\theta_i$:

$$ \left( \frac{df}{d\theta} \right)_i = \frac{df}{d\theta_i} $$

Let's apply this to $\theta^T\theta$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = \frac{d}{d\theta_i} \sum_{n=1}^N \theta_n^2 = 2 \sum_{n=1}^N \theta_n \frac{d\theta_n}{d\theta_i}  $$

and since $\frac{d\theta_n}{d\theta_i} = \delta_{i,n}$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = 2 \theta_i $$

If you interpret this as columns or rows is pretty much up to you. Commonly we would take $\theta_i$ as the elements of a column vector $\theta$ and for practicality we usually want the gradient to live in the same vector space as our coordinates, so $\left( \frac{df}{d\theta} \right)_i$ would also be a column vector. Hence

$$ \frac{d}{d\theta} \theta^T\theta = 2 \theta $$


Spark 1.5.1: Train many binary classifiers, save them, then use them on new data

This is a general ML answer, not about Spark 1.5.1, but perhaps it can help you find a way to the solution.

Your problem is multi-label as well as multi-class (e.g. http://scikit-learn.org/stable/modules/multiclass.html).

Some algorithms have multi-label versions, and will deal with the problem internally so you don't have to. If you don't have a multi-label version of your algorithm, you can either change to an algorithm that does, or roll your own multi-label solution around your algorithm, for example using binary classifiers for each label. (This typically leads to worse performance as there are many complications and possible optimisations to take into account).


SVM - Shuffle image data before GridSearchCV or not?

GridSearchCV will not shuffle the data by default. However, as you noted, you can pass Kfold or StratifiedKfold objects to cv argument, that specify shuffling.

For example:

model = XGBClassifier()
param_grid = {"n_estimators": [100]}
cv = StratifiedKFold(n_splits=5, shuffle=True)
gs = GridSearchCV(model, param_grid=param_grid)


Also please explain what this array - array.mean() do?

Basically, it is doing memberwise subtraction operation after broadcasting. np.mean function finds the mean in your array and its result will be a scalar, a single number. Your array is a numpy array and the result of the latter term is a single value as mentioned. Consequently, the single value gets extended to the shape of the former term. then a memberwise subtraction will be performed for each entry of the array and the result will have the same shape as the former term.

  Can you please explain what does this normalize function do?

Normalizing data is done for accelerating optimization. If you have features with different scales, it will take too much time for your optimizer function to find optimal points. Suppose you have age feature which can change between 0 to 150 (!) and salary which can be changed from 0 to whatever, like 500,000,000 $. your optimization algorithm used in your ML model will take too much time, if possible, to find appropriate weights for each feature. Moreover, if you don't scale your data, your ML algorithm may take too much care to features with large scales.


No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM.

A normal LM takes an autoregressive factorization of the probability of the sentence:

$p(s) = \prod_t P(w_t | w_{<t})$

On the other hand, BERT's masked LM loss focuses on the probability of the (masked) token at a specific position given the rest of the unmasked tokens in the sentence.

Therefore, it makes no sense to use the token probabilities generated by BERT and multiply them to obtain a sentence level probability.

A secondary issue is the BERT's tokenization is subword-level so, even if it would make sense to compute a sentence-level probability with BERT, such a probability would not be comparable with a word-level LM, as we would not be taking into account all possible word segmentations into subwords.

UPDATE: there is a new technique called Masked Language Model Scoring (ACL'20) that allows precisely what the OP asked for. From the article:

  To score a sentence, one creates copies with each token masked out.
  The log probability for each missing token is summed over copies to
  give the pseudo-log-likelihood score (PLL).

So the answer is now YES. It is possible to score a sentence using BERT, by means of the described pseudo-log-likelihood score.


Clustering after PCA: Use the standardized data, or take into account the variation explained at each PC?

I had a more careful check, and I will try to answer my question based on the additional insights. I would appreciate further answers/comments regarding the accuracy of the provided answer.

The actual results of the PCs are not standardized, and their variance is proportional to the square root of the total variance (variation) explained by each PC. These values are the ones that should be used for the subsequent clustering. In this way, the weighting of the PCs is indeed proportional to the (square root of the) explained variance.

If the standardized PCs (mean=0, std=1) are used, then the values should be adjusted with the square root of the % of variance explained by each PC. In this way, the adjusted PCs will have the same weighting as the actual PCs, and the only difference will be on the range of the values.

I run the analysis for clustering based on 1) Actual PCs, 2) Standardized PCs, 3) Standardized PCs adjusted with the % of the variance, and 4) Standardized PCs adjusted with the square root of the % of the variance. 1) and 4) were practically equal (I assume some small differences are due to the fact that the range of the values affects the instances very close to 0) and the best ones, followed by 3). Option 2) was by far giving the worst performance. The performance was tested by the Sum of Squared Distances between each field and the allocated centroid on the original dataset (daily gridded data).


Machine Learning: Take into account a variable if a condition is met (depending on another variable)

If you are interested to see if there are any other girls, you'll likely need to create a new indicator variable in your modelling data - perhaps something like:

other_girls_in_class = 1 if gender = female and there are other girls in the class
other_girls_in_class = 0 if gender = female and there are no other girls in the class

You need to treat these kind of approaches with care as it will implicitly contain some information on student gender.


In my personal opinion the lecturers's attempt to "make things look easy" by showing "magic tricks" ("let us now divide by the norm as if we were dealing with one-dimensional vectors AND VOILA") is rather unhelpful as it does not really explain neither the methodology nor the math nor the intuition.

There are some basic, but important notions hidden behind that "voila" step, let me help you by writing them out explicitly.

One way to find the distance between two parallel lines is to:

Pick any point $x_1$ on the first line.
Pick any point $x_2$ on the second line.
Project the vector $x_1 - x_2$ on the direction, perpendicular to two lines and measure the length of the projection.

Here's an illustration, just in case: you have two lines, one point taken on each line, the vector $x_2-x_1$ connects the two points. The dotted line is a perpendicular to the two lines. The vector $v$ points in the direction of the perpendicular, and the red segment shows the distance you want to measure.

This red distance is exactly the length of the projection of $x_2 - x_1$ onto the dotted line.

At this point we shall need three facts:

(a) For any unit vector $v$ the value $x^T v$ is equal to $|x|\cos \alpha$, where $\alpha$ is the angle between the vectors. Coincidentally, this is exactly the (signed) length of the projection of $x$ on the line with direction defined by $v$.
(b) Any vector $v$ can be made a unit vector (i.e. vector of length 1) by dividing it with its actual length: $\frac{v}{|v|}$.
(c) The equation $w^T x + b = 0$ defines a line with the normal $w$. I.e. $w$ is a vector, perpendicular to the line.

We know from (c) that $w$ is perpendicular to the line, we know from (b) that $\frac{w}{|w|}$ is a unit vector, so we plug it into (a) to get that the (signed) distance between the lines must be equal to $$\frac{w^T(x_2-x_1)}{|w|}.$$

If we know that $w^T x_2 + b = -1$ and $w^Tx_1 + b = 1$ we can play a bit with the equations to get $w^T(x_2 - x_1) = -2$ and hence the (signed) distance must be $\frac{-2}{|w|}$. We can drop the sign easily. ... and voila!

This is certainly not the only way to solve the problem, so for completeness sake let me show you a similar, yet still useful alternative. Here, you need to know just one fact (which I suggest you memorize just like the facts above):

(d) (Signed) distance from point $p$ to a line defined implicitly by the equation $w^T x + b = 0$ (note the zero on the right) is equal to $\frac{w^T p + b}{|w|}$.

Now, we can compute the distance between two lines by taking a point on one of them and substituting it in (d).

Let $x_1$ be a point on the line $w^T x+ b = 1$. The second line has the form $w^T x + b = -1$ which is the same as $w^T x + b + 1 = 0$. The distance between the lines is therefore:

$$\frac{w^T x_1 + b + 1}{|w|} = \frac{1 + 1}{|w|} = \frac{2}{|w|}.$$

Yet another way to derive the necessary distance would be to note that the line $x = sw$ is perpendicular to the two lines of interest. If $x_1$ and $x_2$ are the intersection points of this perpendicular with the first and the second parallel line correspondingly, the necessary distance is just the Euclidean distance between the two points:
$$ d = |x_1 - x_2| = \sqrt{(x_1 - x_2)^T(x_1 - x_2)}. $$
I'll leave solving this to you as an exercise.


faster alternatives to sparse.model.matrix?

Try that but remove LHS variables from dataset before hands? We use that for the data processing part of putting our xgboost models in production.

fspmatrix <- function(df) {
  # Building sparse matrix with sparseMatrix to save time vs sparse.model.matrix

  # Details about the df to help build triplet
  nr <- nrow(df)
  nc <- ncol(df)
  nlevels <- pmax(1, sapply(df, nlevels))
  fac <- which(sapply(df, is.factor))
  # Check that all factors have at least 2 levels
  stopifnot(min(nlevels[fac]) > 1)

  # Building the value vector removing zero positions and first levels of factors after the first factor
  x <- as.numeric(unlist(unlist(df)))
  jx <- x # for use in building j
  # Positions of values coming from factors
  pos <- as.vector(sapply(fac * nr, function(x) x - nr + (1:nr)))
  # Positions of first levels of factors after the first factor
  pos1 <- pos[-1:-nr]
  pos1 <- pos1[which(x[pos1] == 1)]
  # Replacing the values of factors with 1 for one hot encoding
  x[pos] <- 1
  # Positions of non-zero values
  nzpos <- which(x[-pos1] != 0)
  # Values to build triplet around
  x <- x[-pos1][nzpos]

  # Building the i vector, the first dim part of the triplet
  i <- rep(seq_len(nr), nc)[-pos1][nzpos]

  # Building the j vector, the second dim part of the triplet
  jnlevels <- nlevels
  jnlevels[fac[-1]] <- jnlevels[fac[-1]] - 1
  jx[-pos] <- 1
  jx[pos[-1:-nr]] <- jx[pos[-1:-nr]] - 1
  cs <- cumsum(c(0, head(jnlevels, -1)))
  j <- jx + rep(cs, each = nr)
  j <- j[-pos1][nzpos]

  # Building the dimnames vector
  tag <- TRUE
  fspnames <- as.vector(unlist(sapply(names(df), function(n) {
    clvl <- as.character(levels(df[[n]]))
    isL <- is.logical(df[[n]])
    isF <- is.factor(df[[n]])
    indx <- if (tag & isF) {
      1:length(clvl)
    } else {
      -1
    }
    if (tag & isF) tag <<- FALSE
    paste0(n, clvl[indx], ifelse(isL, "TRUE", ""))
  })))

  fspm <- sparseMatrix(i = i, j = j, x = x, dims = c(nr, tail(cs, 1) + tail(jnlevels, 1)), dimnames = list(NULL, fspnames))

  return(fspm)
}


Multiple binary dummy features Vs Multi-values single feature

As @K3 suggested, as @Sean Owen told me the name (One Hot Encoding), I can provide at least some elements to answer myself.

As found on sklearn documentation :

  [...]integer representation can not be used directly with scikit-learn
  estimators, as these expect continuous input, and would interpret the
  categories as being ordered, which is often not desired.

It is implemented in sklearn as OneHotEncoder and in Pandas as get_dummies .


How to accomodate different database / schema names when writing M queries in Power BI?

There is no public documentation on generating column names... So hopefully I can help shed some light here.

When creating the data source with MySQL.Database, you can actually pass in a function that will be used to generate the column names, NavigationPropertyNameGenerator

First, we need to create the function that will generate these names. Basically, I want the exact same behaviour as the default except to not change the name depending on the schema / selected db.

Create a function to generate names, I created a blank query and used the below code (which will replace all instances of "abc." to "xyz.", where abc is the actual name of the schema.)

let
    ListAccumulateFunction = (state, current) =>
    let
        effectiveName = Text.Replace(current, project_database & ".", "xyz."),
        occurances = List.Count(List.FindText(state{1}, effectiveName)),
        newName = effectiveName & (if occurances > 0 then " " & Text.From(occurances + 1) else ""),
        newNamedList = List.Combine({ {effectiveName}, state{1} }),
        newColumnsList = List.Combine({state{0}, {newName} })
    in
        {newColumnsList, newNamedList},

    ColumnNameGenerator = (p, a) =>
    let
        working = List.Transform(a, each _[TargetTableName] & " (" & Text.Combine(List.Distinct(_[SourceKeys])) & ") "),
        renamedDuplicates = List.Accumulate(working, {{}, {}}, ListAccumulateFunction)
    in
        renamedDuplicates{0}

in
    ColumnNameGenerator

Now we can use this function in throughout the project in data sources as the name generator

Source = MySQL.Database("127.0.0.1", project_database, [ReturnSingleDatabase=true, NavigationPropertyNameGenerator=ColumnNameGenerator]),


Can you sum up gradients and apply in batch?

Reading the relevant paragrpah, we can gain a little more insight. It seems to come down to the idea of delayed reward. By only updating the model (i.e. the agent) every once in a while, we are inherently inducing the agent to make moves which are benficial over longer periods of time.

Here is the paragraphs from the post:

  Delayed reward — Keeping the pole in the air as long as possible means moving in ways that will be advantageous for both the present
  and the future. To accomplish this we will adjust the reward value for
  each observation-action pair using a function that weighs actions over
  time.

  To take reward over time into account, the form of Policy Gradient we
  used in the previous tutorials will need a few adjustments. The first
  of which is that we now need to update our agent with more than one
  experience at a time. To accomplish this, we will collect experiences
  in a buffer, and then occasionally use them to update the agent all at
  once. These sequences of experience are sometimes referred to as
  rollouts, or experience traces. We can’t just apply these rollouts by
  themselves however, we will need to ensure that the rewards are
  properly adjusted by a discount factor

  Intuitively this allows each action to be a little bit responsible for
  not only the immediate reward, but all the rewards that followed. We
  now use this modified reward as an estimation of the advantage in our
  loss equation. With those changes, we are ready to solve CartPole!

The agent might otherwise want to just do the optimal move to keep the pole upright for the next milllisecond, but with this buffer, we are essentially only allowing the agent to consider larger chunks of time.
This will take longer to train, but should in theory (well, the assumption behind these rollouts...) lend itself to creating a more thoughtful agent.

In other contexts, such as a standard CNN used to classify images, this approach would make less sense as there isn't a temporal dimension to the task.

Saving up the errors and making one big update can have several other explanations in general:

I can save compute, as we could formulate the graph to then only perform backprop after our buffer is full. For buffer size not close to 1, this would potentially halve the training time (making the gross simplication of an assumption that learning occurs at the same rate)
It may be equivalent to using batches, in that it could help smooth the learning process. If you have a batch size of 1 sample, the error that is measures and used to update weights may be quite erratic. Using a batch has a nice averaging/smoothin effect. Saving up errors in a buffer, one could argue, may have a similar effect.

I should add, I haven't personally seen this in practice outside of reinforcement learning.


Best methods to solve class imbalance problem and why?

There are 2 Techniques:

Oversampling: There are many techniques under this, ROSE and SMOTE are the most famous techniques used for oversampling. In ROSE it just increases the minority classes. In SMOTE it synthetically generates more number of rare minority classes for balancing. Most of the Scenarios SMOTE gives better results than ROSE but you should try both. Other than that there is just another techniques which just duplicates the records to the make it equal n number. This Link, is for implemenation of SMOTE in Python.
UnderSampling: There are many techniques under this too, but this Link-1, Link-2 gives you better idea about undersampling. Generally I don't prefer Undersampling, as you would loose some infomation.

The reason why you need to use these techniques are, if we don't use then the model accuracy will be very high it will be able to predict with 99% accuracy correctly the cases which are not fraud, which we don't want. if it can predict with same accuracy to find out fraud, then that is a great insight.
This can be achived only by using either of the above techniques.

Do have a look and Let me know if you have any additional questions.


Is there any text similarity databse available for phrases?

This is a difficult problem, but definitely worth exploring.
An interesting resource to look into is DBpedia. It aims to extract structured information from the Wikipedia project. It is available under a free license (CC-BY-SA).
You can conveniently explore the project online, e.g.:

IBM
Beam Suntory

Note that you are restricted to the extensive but ending knowledge on Wikipedia, for example Synergy Telecom/SynTel seems not to have an entry. Your creativity would be required to overcome this limitation.


What is the exact definition of VC dimension?

The definition of VC dimension is: if there exists a set of n points that can be shattered by the classifier and there is no set of n+1 points that can be shattered by the classifier, then the VC dimension of the classifier is n.

The definition does not say: if any set of n points can be shattered by the classifier...

If a classifier's VC dimension is 3, it does not have to shatter all possible arrangements of 3 points.

If of all arrangements of 3 points you can find at least one such arrangement that can be shattered by the classifier, and cannot find 4 points that can be shattered, then VC dimension is 3.


How to train a neural network on multiple objectives?

What you're referring to is called multi-task learning, where your goal is to have a single network learn multiple tasks (in your case "click" and "purchase").
The benefit of having a single model learn both tasks is that the network can use information extracted for one task to improve its performance on the other.
Technically you need your network to have two output layers (i.e. one for predicting "clicks" and one for "purchases") and likewise two loss functions (one for each objective). These loss functions usually contribute equally towards the total loss, unless you don't want to consider both tasks of equal importance.
If you're using keras an example for multi-output models can be found here.


Target data values are not evenly distributed

From scikitlearn LogisticRegression docs:

  class_weight : dict or ‘balanced’, default: None

  Weights associated with classes in the form {class_label: weight}. If
  not given, all classes are supposed to have weight one. The “balanced”
  mode uses the values of y to automatically adjust weights inversely
  proportional to class frequencies in the input data as n_samples /
  (n_classes * np.bincount(y)). Note that these weights will be
  multiplied with sample_weight (passed through the fit method) if
  sample_weight is specified. New in version 0.17:
  class_weight=’balanced’

So try to add class_weight='balanced'in your call to LogisticRegression()

Or maybe if this doesn't work, try to use as trainSet an evenly split dataset: where the number of samples of class 1 is equal to class 0.


You have plenty of alternatives for this problem

Using sapply

data$abstract <- sapply(data$abstract,
                        function(x){gsub(pattern = "no abstract available",
                        replacement = " ", x)})

Using mapply

data$abstract <- mapply(gsub, pattern = "no abstract available",
                        replacement = " ", data$abstract)

Using the stringr package

library(stringr)
data$abstract <- str_replace(data$abstract, "no abstract available",
                             " ")

Also, check this this question on StackOverflow for more information, like solutions with match and the qdap package.


Remove correlated features before or after splitting test and training set?

By principle in supervised ML any decision which affects the model should be made using only the training set, in order to avoid data leakage. Following this principle requires the training/test split to always be done first.
Quite often the decision seems minor and has little chance to cause data leakage, like in your case. It's tempting to make our life easier and use the whole data for some pre-processing like this. However it's a good idea to follow this principle as strictly as possible not because the effect of data leakage would be huge, but because it would be impossible to detect it if it happens: when it happens, data leakage means that the performance on the test set is biased (overestimated), and there's simply no way to know by how much (except by collecting a new test set, of course).
Your concern that the split of the data could impact the analysis of correlation is a good example of the problem: suppose there would be a difference indeed. This means that there is data leakage, since the features are different depending directly on the test data. In this case the evaluation on the test set is likely to be biased.
Another way to explain the same idea is this: if something happens or doesn't happen in the training set by chance, then it's important to see if the model is able to focus on the real patterns and ignore the noise. The only way to obtain a reliable answer to this question is if nothing in the model depends on the test set.


Why is ElasticNet performs worse than both Lasso and Ridge?

The ElasticNet model is not being tuned. By default in scikit-learn, ElasticNet's l1_ratio parameter, the mixture of L1 and L2 penalty, is set to .5. A .5 l1_ratio represents an even mixture of L1 and L2 penalty and does not fit the data very well. Best practice is a cross-validation grid search for the optimal value of l1_ratio.