Neural network example not working with sigmoid activation function

I found the answer by myself. The reason of the difference is that the definition of prime of tanh in BogoToBogo (tanh_prime) takes arguments that's already applied with activation function:

def tanh_prime(x):
    return 1.0 - x**2

while sigmoid_prime is not. It calls sigmoid in it:

def sigmoid_prime(x):
    return sigmoid(x)*(1.0-sigmoid(x))

So the definition of sigmoid_prime seems more accurate than tanh_prime. Then why not sigmoid is working? It's because their parameters are already applied with the activation function.

Background

The derivatives of sigmoid ($\sigma$) and tanh share the same attribute in which these derivatives can be expressed in terms of sigmoid and tanh functions themselves.

$$
\frac{d\tanh (x)}{d(x)} = 1 - \tanh (x)^2
$$
$$
\frac{d\sigma (x)}{d(x)} = \sigma(x) (1 - \sigma(x))
$$

When performing backpropagation to adjust their weights, neural networks apply the derivative ($g^{'}$) to the values that's before applied with activation function. In BogoToBogo's explanation, that's variable $ z^{(2)} $ in

$$
\delta^{(2)} = (\Theta^{(2)})^T \delta^{(3)} \cdot g^{'}(z^{(2)}).
$$

In its source code, the variable dot_value holds such values. The Python implementation, however, calls the derivative with the vector stored in variable a. The vector is after applied with activation function. Why?

I interpret this as optimization to leverage the fact that derivatives of sigmoid and tanh use their parameters only to apply the original function. As the neural network already holds the value after activation function (as a), it can skip unnecessary calculation of calling sigmoid or tanh when calculating the derivatives. That's why the definition of tanh_prime in BogoToBogo does NOT call original tanh within it. However, the definition of sigmoid_prime, on the other hand, calls sigmoid function unexpectedly, resulting in miscalculation of derivative function.

Solution

Once I define sigmoid_prime in such a way that it assumes the parameter is already applied with sigmoid, then it works fine.

def sigmoid_prime(x):
    return x*(1.0-x)

Then calling the implementation with

nn = NeuralNetwork([2,2,1], 'sigmoid', 500000)

successfully outputs:

(array([0, 0]), array([ 0.00597638]))
(array([0, 1]), array([ 0.99216467]))
(array([1, 0]), array([ 0.99332048]))
(array([1, 1]), array([ 0.00717885]))


How to wisely choose machine algorithm for the given data set?

I used a simple Neural Network with 3 Inputs - 1 hidden layer with 8 neurons - 3 Outputs (Labels) with k-fold crossvalidation in keras for your data set.

import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from keras.utils import np_utils
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import LabelEncoder, scale
from sklearn.pipeline import Pipeline

path = r"User\train.csv"
data = pd.read_csv(path)

data.location.value_counts()
dataset = data.values

x = dataset[:,0:3]
x_scaled = scale(x)
y = dataset[:,3]

encoder = LabelEncoder()
encoder.fit(y)
encoded_Y = encoder.transform(y)

dummy_y = np_utils.to_categorical(encoded_Y)

def model():
    model = Sequential()
    model.add(Dense(8, input_dim=3, activation='relu'))
    model.add(Dense(3, activation='softmax'))

    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model

estimator = KerasClassifier(build_fn=model, epochs=200, batch_size=5, verbose=0)
kfold = KFold(n_splits=10, shuffle=True)

results = cross_val_score(estimator, x_scaled, dummy_y, cv=kfold)
print("Model Prediction: %.2f%% (%.2f%%)" % (results.mean()*100, results.std()*100))

The used Model can categorize around 94% of the data correctly. Maybe this helps for the start.

Out:
Prediction: 93.95% (3.13%)

In data science and statistics in general you have to try and train several models to see which suits your dataset best (model benchmarking). In the end it is often a trade-off between computational costs and accuracy.


Does it make sense to do train test split when trainning GANS?

Training GANs is only a partially unsupervised task, IMHO. It's certainly unsupervised for the Generator, but it's supervised for the Adversarial Network. So it might be useful to test the Disciminator's ability to distinguish fake and true cases on new data it has never seen before.

In other words, it makes sense to split your dataset in train(-validation)-test if you want to understand the Discriminator's ability to generalize its task on data it has never seen before. In case this is not of your interest, I guess you don't need to do it.


Understanding similarity between two phrases has two aspects

How similar are the unique tokens in the phrases ?
How much should the individual tokens contribute to the overall phrase similarity?

To answer 1, you can use vector similarity which can give you high similarity for tokens similar in meaning. To answer 2, you should look at giving importance/weights to the tokens. You can use a measure like tf-idf. While comparing hospital machine and building machine, machine being a frequent word in your corpus should get a lower score and hence would contribute lesser to the overall similarity. Most of the similarity would be then determined by the similarity between hospital and building which would solve your issue.


YOLOv3 Predicting bounding boxes for grid containing multiple centers

It depends on the combination of thresholds of the objectness score and non-max suppresion (NMS):
Objectness score $C$
Apart from the location and dimensions predictions ($t_x, t_y, t_h, t_w$) for each box, an objectness score $C$ is also predicted for each of the boxes. It tells us how likely an object is inside  of a certain predicted bounding box (BB).
But How much it needs to be to ensure an object is inside?
This has not a precise answer, instead we use a certain threshold. For example, we could set that a BB contains an object if and only if the objectness $C>0.5$ (threshold = $0.5$). This value depends on the implmentation and it can be asked to the user as in this repo
Non-max suppresion (NMS)
Now that we know that there is an objectness score threshold, we can rephrase the question  in the following way:
What if two BBs of the same grid cell have an objectness score $>$ threshold? Would one of these two BBs be eliminated?
This depends on the threshold used in non-max suppression. If this concept is new to you, I recommend you this tutorial about it.
Broadly speaking it's used to discard bounding boxes that have a lower objectness score than the bounding box assigned to the same class that has the highest objectness score, and that have an overlap with it bigger than the threshold we are talking about now.
This means that higher thresholds used in NMS will lead to discarding a fewer number of boxes (we will be allowing a bigger overlapping between the boxes).
Conclusion
So, to sum up, In the situation of the question (more than one object per grid cell). Only boxes that have values of $C$ higher than its threshold will be considered. And out of these boxes, if there are some that are assigned to the same class then the box that have the highest $C$ will be considered and also the ones that don't get eliminated after applying non-max suppresion.


Significant testing - repeated observation over multiple days

It depends on what you are testing and what your assumptions are.
Here are couple of scenarios:

If you are looking at single day, then it does not matter.

If you are comparing multiple days but looking at each day independently, they you might want to control for the multiple comparisons problem.

If you are looking for changes over days, then you might want to treat it as a time series problem.


Interesting question! Maybe the pretrained models in Keras can help. Either by means of transfer learning, so that you might have to label only a small number of images by hand to retrain the higher layers. Or by using them for feature extraction and see if a certain keyword appears frequently for watermarked images.

Or just upload the pictures some place that does not allow watermarks and see if they get flagged ;-)


LSTM regression bias increases when targets go close to 0

I found a solution for the problem: scaling the features and the labels.
By performing min-max scaling over the features and labels, the curves become:

The error is still showing a pattern related to the target value, but the normalized error is now really low...
I think this is a nice example of the benefits of normalizing features...


Can the number of epochs influence overfitting?

Yes, it may. In machine-learning there is an approach called early stop. In that approach you plot the error rate on training and validation data. The horizontal axis is the number of epochs and the vertical axis is the error rate. You should stop training when the error rate of validation data is minimum. Consequently if you increase the number of epochs, you will have an over-fitted model.

In deep-learning era, it is not so much customary to have early stop. There are different reasons for that but one of them is that deep-learning approaches need so much data and plotting the mentioned graph would be so much wavy because these approach use stochastic-gradient-like optimizations. In deep-learning again you may have an over-fitted model if you train so much on the training data. To deal with this problem, another approaches are used for avoiding the problem. Adding noise to different parts of models, like drop out or somehow batch normalization with a moderated batch size, help these learning algorithms not to over-fit even after so many epochs.

In general too many epochs may cause your model to over-fit the training data. It means that your model does not learn the data, it memorizes the data. You have to find the accuracy of validation data for each epoch or maybe iteration to investigate whether it over-fits or not.


Server log analysis using machine learning

I don't think you necessarily need to convert the individual log entries into vectors for use in an algorithm.  I would guess that what you are interested in is a sequence of log entries, which represent a series of events, ordered in time, which together make up a series of 'cases'.  Here the relationship between a series of collected log entries is important.

If this is the case then you might want to consider using Process Mining techniques.  This allows you to build models of your process (the use of your application) and determine patterns of process steps, along with errors and rework steps.

There is a good introduction course on Coursera, here.  There are even some developed, commercial packages like 'disco' to help you with the analysis and visualisation


Why the estimated Lasso coefficients of almost all variables are equal to zero?

This was due to the fact that I should normalize my data before passing it into the algorithm. For example,

scaler = StandardScaler()
scaler.fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

Or, the "normalize" parameter of Lasso should be set as True.

Lasso(alpha=float(alpha), normalize=True, random_state=42)


How to evaluate performance of a time series model?

The best summary on evaluating time series forecast is probably explained in detail on Rob Hyndman's site. I typically use the mean absolute percentage error which is baked in Keras. However, what I found in a different setting is that the MAPE prevents the neural network from converging if combined with the Adam optmization. I had much better success with the rooted mean square error (RMSE). Since you have poor experience with that maybe you could use the symmetric MAPE.


Why do we need to add START  + END  symbols when using Recurrent Neural Nets for Sequence-to-Sequence Models?

Because of the encoder-decoder structure. The encoder reads the input sequence to construct an embedding representation of the sequence. Terminating the input in an end-of-sequence (EOS) token signals to the encoder that when it receives that input, the output needs to be the finalized embedding. We (normally) don't care about intermediate states of the embedding, and we don't want the encoder to have to guess as to whether or not the input sentence is complete or not.

The EOS token is important for the decoder as well: the explicit "end" token allows the decoder to emit arbitrary-length sequences. The decoder will tell us when it's done emitting tokens: without an "end" token, we would have no idea when the decoder is done talking to us and continuing to emit tokens will produce gibberish.

The start-of-sequence (SOS) token is more important for the decoder: the decoder will progress by taking the tokens it emits as inputs (along with the embedding and hidden state, or using the embedding to initialize the hidden state), so before it has emitted anything it needs a token of some kind to start with. Hence, the SOS token.

Additionally, if we're using a bidirectional RNN for the encoder, we're definitely going to want to use both SOS and EOS tokens since the SOS token will signal to the reversed-input layer when the input is complete (otherwise, how would it know?).


Unsupervised learning to identify most common basketball plays from spatial data

This is a really interesting problem! Like most really interesting problems, you're unlikely to find an out-of-box solution for this, but I think the field of graph/subgraph similarity has some promise here. I'll go into more detail, but, at a high level, I think you can view players' paths during a play as a collection of five traversals through x,y space, with a vertex being any x,y point you have available (presumably there's some level of time granularity here), and an edge describing a player's movement from one point in x,y space to the next over time. It should be possible to cluster your data using a similarity metric (e.g., see Koutra et al., 2011 for a nice overview). Then, using your own domain expertise, you should be able to identify whether the clusters you've derived have some real-world meaning in basketball.


numer.ai: how does their leaderboard system work?

I found the answer in the comments section of their blog:

  Earnings listed on the public-score leaderboard are potential winnings. Actual winnings are determined by the private-score leaderboard. At the time of withdrawal and when new datasets are released, your actual winnings will be revealed to you.


What are the best ways to tune multiple parameters?

Generally people perform a grid search, which in its simplest "exhaustive" form is similar to Method 1.  However there are also more 'intelligent' ways to choose what to explore, which optimize in parameter space in a fashion similar to how each individual model is optimized. It can be tricky to do greedy optimization in this space, as it is often strongly non-convex.

This page describes the basics of optimizing model parameters.


very important features, but rarely input

A tree-based model, especially one that allows missing data in the input, seems a good choice: the tree(s) can split large capital gains out (presumably separating a nearly pure node from the rest).

Also consider something more basic: just hard-code an exception for those with large gains, and model for the rest.


Memory-efficient metric calculation for ultra high dimensional data

The way to go was to use relative similarity metric:

\begin{equation*}
d(x,y)=\left(1-\frac{\left\vert{x \cap y}\right\vert }{\left\vert{x \cup y}\right\vert }\right)^k
\end{equation*}

Where x and y are sets containing hashes identifying features and k is experimentally selected factor that allows "spreading" the range of distances.

Given that data point, being really a binary vector of size 2^32 have bits set only on the positions denoted by the hashes, we can calculate the distance as size of intersection of hash sets divided by size of sum of hash sets.


The automatic construction of new features from raw data

'high predictive value' is only defined if you a target which you are trying to predict upon. It seems that you don't, and your goal is to cluster data points according to some scale defined by a variety of factors. These can undoubtedly be used to cluster data points, and I'd advise you to look into the various methods available: some that may be interesting for you are Agglomerative and Hierarchical clustering.

Now to answer the question, you can surely generate new features from the ones present in your dataset that may or may not help you achieve your goal. You can:

Bin your data: define some categories such as 'rich', 'average', 'poor' with specified ranges and create a new feature that maps a numerical value (wealth) to a bin
One-hot-encode categorical variables

After these pre-processing steps are done, you could go ahead an apply the clustering methods I mentioned to group data together into their respective 'Social Status'. Of course, lots of tweaking and experimentation will be needed. As far as what I've encountered, there are not really automagic ways of generating new features, and mostly the available methods will depend greatly on the type of data and problem you are working on.


What is the BLEU score used in Google Brain's "Attention Is All You Need" paper?

BLEU (Bi Lingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated (MT) from one natural language to another. BLEU is typically measured on a 0 to 1 scale, with 1 as the hypothetical “perfect” translation.

Google uses the same definition of BLEU but multiplies the typical score by 100.


How to find similarity of two series over time containing periodic trends?

Maybe it is a huge oversimplification, but you can try: What if you do just arithmetic difference between the two signals (or divide one signal to another one) and look at the result?

I would expect that you will have spikes where signals become very different and you can make a threshold to find where they are on time.


I think that this system of equation is incorrect. If you know that (3, -1), (3, 1) and (1, 0) are support vectors then you need to solve the next system:

3*w1 - 1*w2 - w0 = ±1
3*w1 + 1*w2 - w0 = ±1
1*w1 + 0*w2 - w0 = ±1

Support vectors on the same side of the separation line have the same sign on the right side of the equation (same sign here: ±1). In this example, (1, 0) locates on one side, while (3, -1) and (3, 1) are on the other side. Thus, the system will be the following:

3*w1 - 1*w2 - w0 = 1
3*w1 + 1*w2 - w0 = 1
1*w1 + 0*w2 - w0 = -1

The solution is: w0 = 2, w1 = 1, w2 = 0. You can plot all the points and check that this solution is correct.


Optimum values of all predictors in logistic regression

Theoretically you won't be able to get the value $1$. However, suppose your variable is bounded and you can indepedently set the values of the variables, you can let it be as large as possible.

Look at the coefficients that you obtained. For those coefficients that you obtained, if they are positive, set the value to be as large as possible. For negative coefficients, set the variable to be as small as possible. If the coefficients is zero, any value is fine.


How to train model on specific classes from dataset for object detection?

You need to change xml or csv files of bounding boxes and from there you need to remove the all classes other then your 5 classes.


I think this problem could be represented as a Markov Chain.
Maybe a Markov model (or Markov Random Fields) could be used to estimate the probabilities. Normally it's easier than with the more standard Hidden Markov Model since there is no hidden state.
My suggestion would be to estimate the probabilities of the model (parameters) from the data, then it should be easier to discover the patterns based on these probabilities.


For Text classification Multinomial Naive bayes is a good choice to start with.

We can start with MNB,
when

1. There is no dependence between the predictors
2. Good if class is not unblanced

(Here is an approach to handle unbalanced data with MNB )

I would start with bag of words and then TFIDF vectors to check the model performance before moving further.


How to use bigrams for a text of sentences?

One way is to loop through a list of sentences. Process each one
  sentence separately and collect the results

Brian is totally right about the solution but I think that's actually the part that is missing from his answer :) Brian's code assumes that the sentences have already been segmented, which was Ahmad's original question as far as I can tell.

With that in mind, here's an updated proposal for adding the sentence splitting to the script:

import itertools, nltk

def ngrams_wrapper(sent):
    return list(nltk.ngrams(sent, 2))

raw = "To Sherlock Holmes she is always the woman." + \
      "I have seldom heard him mention her under any other name."

# Given a string as input (`raw`), split it into sentences.
# Returns an object of type 'list: str'
sentences = nltk.sent_tokenize(raw)
# ['To Sherlock Holmes she is always the woman.',
# 'I have seldom heard him mention her under any other name.']

# Next, tokenize every sentence (string) in the list of sentences. The tokenizer takes
# strings as input so we need to apply it on each element of `sentences` (we can't apply
# it on the list itself). For that, we can use the function `map`, which applies any
# callable Python object to every element of a list.
# The output of this step will be an object of type
#   'list: list: str'
# such that
#   'sentences: sentence: word'
tokenized = map(nltk.tokenize.word_tokenize, sentences)
# [['To', 'Sherlock', 'Holmes', 'she', 'is', 'always', 'the', 'woman', '.'],
# ['I', 'have', 'seldom', 'heard', 'him', 'mention', 'her', 'under', 'any', 'other',
# 'name', '.']]

# We now generate the ngrams for each tokenized sentence in the list of sentences. Since
# we want to generate the ngrams for each sentence specifically, we again must apply the
# function to each element of the list separately, not the whole list. So, we use the
# `map` function again.
# The output of this step will be an object of type:
#   'list: list: tuple: <str, str>'
# such that
#   'sentences: sentence: ngram: <word_1, word_2>
bigrams = map(ngrams_wrapper, tokenized)
# [[('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.')], [('I', 'have'),
# ('have', 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'),
# ('mention', 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other',
# 'name'), ('name', '.')]]

# Finally, since we want all the bigrams in a single list to work with them more easily,
# we flatten the list, that is, we concatenate all the elements at the first level of
# depth in the list. So, if the list was an object of type
#   'list: list: tuple: <str, str>'  (sentences: sentence: ngram)
# we will transform it into:
#   'list: tuple: <str, str>'        (sentences: ngram)
bigram = list(itertools.chain.from_iterable(bigrams))
# [('To', 'Sherlock'), ('Sherlock', 'Holmes'), ('Holmes', 'she'), ('she', 'is'), ('is',
# 'always'), ('always', 'the'), ('the', 'woman'), ('woman', '.'), ('I', 'have'), ('have',
# 'seldom'), ('seldom', 'heard'), ('heard', 'him'), ('him', 'mention'), ('mention',
# 'her'), ('her', 'under'), ('under', 'any'), ('any', 'other'), ('other', 'name'),
# ('name', '.')]

freq_dist = nltk.FreqDist(bigram)
# [(('always', 'the'), 1), (('woman', '.'), 1), (('Holmes', 'she'), 1), (('seldom',
# 'heard'), 1), (('Sherlock', 'Holmes'), 1), (('him', 'mention'), 1), (('I', 'have'), 1),
# (('any', 'other'), 1), (('under', 'any'), 1), (('the', 'woman'), 1), (('her', 'under'),
# 1), (('other', 'name'), 1), (('To', 'Sherlock'), 1), (('name', '.'), 1), (('she', 'is'),
# 1), (('heard', 'him'), 1), (('mention', 'her'), 1), (('have', 'seldom'), 1), (('is',
# 'always'), 1)]

prob_dist = nltk.MLEProbDist(freq_dist)
number_of_bigrams = freq_dist.N()


Why are ensembles so unreasonably effective

For a specific model you feed it data, choose the features, choose hyperparameters etcetera. Compared to the reality it makes a three types of mistakes:

Bias (due to too low model complexity, a sampling bias in your data)
Variance (due to noise in your data, overfitting of your data)
Randomness of the reality you are trying to predict (or lack of predictive features in your dataset)

Ensembles average out a number of these models. The bias due to sampling bias will not be fixed for obvious reasons, it can fix some of the model complexity bias, however the variance mistakes that are made are very different over your different models. Especially low correlated models make very different mistakes in this areas, certain models perform well in certain parts of your feature space. By averaging out these models you reduce this variance quite a bit. This is why ensembles shine.


How to improve results from ML model? (spam classification)

In your results you can observe the usual problem with imbalanced data: the classifier favors the majority class 0 (I assume this is class "ham"). In other words it tends to assign "ham" to instances which are actually "spam" (false negative errors). You can think of it like this: with the "easy" instances, the classifier gives the correct class, but for the instances which are difficult (the classifier "doesn't know") it chooses the majority class because it's the most likely.
There are many things you could do:

Undersampling the majority class or oversampling the minority class is the easy way to deal with class imbalance.
Better feature engineering is more work but it's often how to get the best improvement. For example I guess that you use all the words in the emails as features right? So you probably have too many features and that probably causes overfitting, try reducing dimensionality by removing rare words.
Try different models, for instance Naive Bayes or Decision Trees. Btw Decision Trees are a good way to investigate what happens inside the model.


How to cache GridSearchCV optimizer result in Google Colab?

We can save the trained model or any other file via Google Colaboratory.

How I'm using it?

  I have mapped my Google Drive with Google Colaboratory notebook and
  saved trained model as a pickle file in it. You can create a file and save your data in it.

How to integrate Google Drive with Google Colaboratory notebook?

#Add and execute below mentioned line of code in Google colaboratory notebook cell.
from google.colab import drive
drive.mount('ndrive')
#Once you execute these two lines, it will ask you to authorize it. Please follow the steps as visible on the Google Colaboratory Notebook.

That's it :).

  Few important points:

  Now you can run usual linux command to browse the directories.
  By default it mapped to content folder and you can create sub directories in it.
  You can visit these sub-directories via Google-Drive application as well.


Multi Label Classification on Data Columns in Tables

@DPascal Here is something you could definitely try doing:

Using the features you generated, you could add a label to each of the column for these features.
You could generate these feature value for different time slices.
Once that is done, you can then run something as simple as Random Forest Classifier on this data.


Transfer Learning on Resnets/VGGs -- Validation accuracy can never be over 75%

There was a Contest in Kaggle - SIIM-ISIC Melanoma Classification.  In this contest also tasks for participants to classify two class: malignant & benign.
But in the dataset, there were the same images in both the class. And this produces the Dataleakage.
Dataleakage,  also can produce in the same class images both in the training and in the validation set.
And to solve this problem Chris Deotte(Kaggle Grandmaster), Applied RAPIDS cuML kNN using CNN embeddings for finding the similar images and then remove those image.
Here are link of the Chris Deotte Kaggle Notebook.
This is the one solution to detect similar images for classes if the model does not work well. There are obviously other factors to improving model accuracy. Best of luck.


COVID: How is the R coefficient calculated?

This article provides a good explanation.
Realistic models are much more complicated, but the basic ideas are:
$R_0$ is the basic reproductive number. If you put an infected individual in a population that is fully susceptible to the disease, then that individual is expected to infect $R_0$ people.
$R_t$ is the effective reproductive number (same as $R_e$). People include the subscript to distinguish $R_t$ from $R_0$ because $R$ is ambiguous. In a population $P$ at time $t$, an infected individual causes $R_t$ new infections on average. Therefore, given the susceptible population $S\subseteq P$ (i.e. the people who are not immune at time $t$), we have $R_t=R_0\frac{|S|}{|P|}$. Again, the actual $R_t$ is affected by many other factors, but if we only considered $R_0$ and the fraction of susceptible people in a population at a given time, then this is the interpretation of $R_t$.
Therefore, the growth ($R_t>1$) or decay ($R_t<1$) in the number of cases is exponential.

For live updates on COVID $R_t$ values in the US per state, you can check out: https://rt.live


Is there a good deployment module using Golang's amazing single binary compilation model?

I code up a Golang package call gotch that can take Pytorch pretrained model and do further training/finetuning or inferring.
Have a look at “example” subfolder as well as Gophernotes notebook links in README.md.


How do you map phrase IDs to sentence IDs in the Stanford Sentiment Analysis dataset?

Only full sentences are used for testing and validation, though sentences and phrases are used for training. See Yoon Kim 2014, Convolutional Neural Networks for Sentence Classification for an overview of various text classification experiments.

Here is code that creates training, dev, and test .CSV files from the various text files in the dataset download.

"""
Put all the Stanford Sentiment Treebank phrase data into test, training, and dev CSVs.

Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive Deep Models
for Semantic Compositionality Over a Sentiment Treebank. Presented at the Conference on Empirical Methods in Natural
Language Processing EMNLP.

https://nlp.stanford.edu/sentiment/
"""

import os
import sys

import pandas

def get_phrase_sentiments(base_directory):
    def group_labels(label):
        if label in ["very negative", "negative"]:
            return "negative"
        elif label in ["positive", "very positive"]:
            return "positive"
        else:
            return "neutral"

    dictionary = pandas.read_csv(os.path.join(base_directory, "dictionary.txt"), sep="|")
    dictionary.columns = ["phrase", "id"]
    dictionary = dictionary.set_index("id")

    sentiment_labels = pandas.read_csv(os.path.join(base_directory, "sentiment_labels.txt"), sep="|")
    sentiment_labels.columns = ["id", "sentiment"]
    sentiment_labels = sentiment_labels.set_index("id")

    phrase_sentiments = dictionary.join(sentiment_labels)

    phrase_sentiments["fine"] = pandas.cut(phrase_sentiments.sentiment, [0, 0.2, 0.4, 0.6, 0.8, 1.0],
                                           include_lowest=True,
                                           labels=["very negative", "negative", "neutral", "positive", "very positive"])
    phrase_sentiments["coarse"] = phrase_sentiments.fine.apply(group_labels)
    return phrase_sentiments

def get_sentence_partitions(base_directory):
    sentences = pandas.read_csv(os.path.join(base_directory, "datasetSentences.txt"), index_col="sentence_index",
                                sep="\t")
    splits = pandas.read_csv(os.path.join(base_directory, "datasetSplit.txt"), index_col="sentence_index")
    return sentences.join(splits).set_index("sentence")

def partition(base_directory):
    phrase_sentiments = get_phrase_sentiments(base_directory)
    sentence_partitions = get_sentence_partitions(base_directory)
    # noinspection PyUnresolvedReferences
    data = phrase_sentiments.join(sentence_partitions, on="phrase")
    data["splitset_label"] = data["splitset_label"].fillna(1).astype(int)
    data["phrase"] = data["phrase"].str.replace(r"\s('s|'d|'re|'ll|'m|'ve|n't)\b", lambda m: m.group(1))
    return data.groupby("splitset_label")

base_directory, output_directory = sys.argv[1:3]
os.makedirs(output_directory, exist_ok=True)
for splitset, partition in partition(base_directory):
    split_name = {1: "train", 2: "test", 3: "dev"}[splitset]
    filename = os.path.join(output_directory, "stanford-sentiment-treebank.%s.csv" % split_name)
    del partition["splitset_label"]
    partition.to_csv(filename)

There is a gist for the code here.


If I understand you correctly, you're looking to take the text of these questions and train classifiers to identify which of 10 categories they belong to. And you'd like to come up with a decent feature representation in order to do this.

I think your finding about part-of-speech is intuitive. It makes sense that in grammatical English (assuming your question data is written in English), most questions would follow similar part-of-speech sequences since grammatically correct questions follow a particular syntactic form (at least when posed interrogatively as in the case of "When was George Washington born?")

So, you've ruled something out - which you should actually view as progress. If you haven't tried it already, one simple thing you might do is use the actual words within the questions as features. You could use any order n-gram you like, but unigrams stick out as an immediate linguistic feature to try. It seems likely to me that while the POS-tags are similar across classes, making them difficult to distinguish between, the actual words being used in the questions may vary from class to class, giving your model a better shot of differentiating between classes.

That is, maybe words like "time", "year", and "when" co-occur more highly with the Date class while words like "numerical" and "quantity" co-occur more with the Number class (obviously, this is speculation - I haven't seen your data). You might also look at bigrams, trigrams, or any other number n-gram for this feature set as well.

Finally, there may be other features you could generate using NLP methods that may be useful. I'm not familiar with the Shiftreducer software, but Named-Entity Recognition could be helpful in generating features for the Factoid class if there are many questions about proper nouns. Other really simple features such as length of the question (counted in number of tokens). A final thought would be to use only the tagged verbs from your POS-tagger, tab these up and to see whether they differ between classes. This may be a useful feature for identifying questions present in your Verb class. Hopefully, those are some ideas to get you started.


Clustering evaluation metrics with subquadratic time complexity

Rand index is not more expensive than NMI, it's O(n) when implemented correctly.

There are like 40 internal evaluation indexes. I am fairly certain there are several that are not quadratic in runtime. Just go through the list, and document your findings for the next one with the same question.

Spark sucks for clustering. It can do k-means and GMM but both are very slow compared to other tools. They don't scale. The attempts at implementing DBSCAN for Spark are even worse. They were found to return incorrect results to crash, and to scale worse than single-core implementations such as ELKI (not to speak of parallel HPC implementations):

  Helmut Neukirchen.
  Survey and Performance Evaluation of DBSCAN Spatial Clustering Implementations for Big Data and High-Performance Computing Paradigms

It's just not what Spark is good at. And given all the "extensions" for it - that often have a very low quality - you quickly run into problems and don't know who is to blame and where to fix them.

Make sure to benchmark.


Several independent variables based on the same underlying data

For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.

Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying variable across all the derived ones.

In some cases, it might not help at all: a tree model on your second example can already easily discover the derived variable given only the original.  It may be actively harmful: adding too many of these derived variables might provide noise for your model to overfit to, rather than useful signal.

In some cases, it might help a lot: in your first example, a linear classifier won't see the derived feature at all from the original, and a tree model would require several consecutive splits to see it. A neural network could build it, but it's not clear whether the training process would find it.


Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad?

One small correction- G here is the sum of squares of the gradient of a particular parameter which is kept track of and is monotonic in nature.
That being said, the way I see the equation functions is, it lets the past dictate the future or simply put, the past gradients of the parameter decides by how much it's corresponding learning rate drops.
In SGD no matter what they gradients were in the distant past their importance drops exponentially(assuming $\eta < 1$). But AdaGrad, seems to be meant for data with sparse features and here, though the update rule gives equal importance to the past gradients, the update is sporadic instead of being regular.
I understood your question and for me at least the concept of sparse features makes the update rule of AdaGrad sense.
Quoting from the paper for quick reference-
In many applications of online and stochastic learning, the input instances are of very
high dimension, yet within any particular instance only a few features are non-zero. It is
often the case, however, that infrequently occurring features are highly informative and
discriminative.
.
.
.
In contrast, our algorithms dynamically incorporate knowledge of the geometry of the data
observed in earlier iterations to perform more informative gradient-based learning.
Informally, our procedures give frequently occurring features very low learning rates and
infrequent features high learning rates, where the intuition is that each time an
infrequent feature is seen, the learner should “take notice.”


It could be normal to have such large loss value if your predicted values are far from the ground truth values (because of the square in the mean squared error loss).
Try to print the predicted values and ground truth with tf.Print to check that.
You can normalize your ground truth values so that their range is not too big. For example it should be easier for the network to predict values between 0 and 10 than values between 0 and 1000.


How should a decision tree handle an attribute that can be anything?

I think your description of anything matches the idea of missing values. Basically by stating that a value could take any value you say that you do not know the value. In standard Breiman description of CART or random forests there is a way to handle missing values. If you use Python stack, however, the implementations does not allow missing values. If that is the case one surrogate would be a different new value for categorical data. For numerical continuous variables I don’t know something better than imputing with average, but it is not quite the same thing.


Using neural networks with jumps in stock returns

In standard models that try to characterise jumps, such as the Jump Diffusion model from Mertons (a short introduction), the model consists of two main parts:

Brownian motion; a random walk to account for the random path, perhaps with some drift - when the values head upwards or downwards in a consisten manner. And
An additive Possion process, which with some probability add a jump in a time-step, with a given probability.

Drawing from this approach, you could also consider using two models that work independently (or separately) to model the overall market and trend, along with a model that introduces jumps at certain timesteps and itnervals.

You could try using different input data for the second model, such as signals taken from text, such as news feeds or newpapers that discuss current market dynamics/politics, possible decisions from SEC and the like. The first (stable) part could be modelled by your current neural network.

This is just a high level idea, and I haven't actually seen any research that already tried it, so unfortunately cannot provide any links to literature.


how to implement a hierarchical clustering technique using parallel execution in R

The complexity of that algorithm is O(n³), and it needs O(n²) memory.

So if your data grows "exponentially", you better settle for a sampling-based approach!

Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for the clustering. Then estimate how long this will take (and how much memory you'll need). Even if you are very optimistic, divide this by the number of threads M you can run. A more realistic value will be M/2 if you can run M threads.

When you are really convinced that you'll be okay with the resulting runtime, just implement the algorithm yourself. It is a very simple algorithm. No need to study the Fortran code. But most likely your code will be 2x slower (20x if using pure Python or R) than a well written old Fortran code...


Elimination of features based on high covariance without affecting performance?

$|d| \gg 0$ means there is a very strong correlation between $x_1$ and $x_2$. This means one can be expressed (almost completely) in terms of the other, thus one of two is almost redundant.
A simple example:
Consider that $x_2$ is simply a copy of $x_1$, ie $x_2=x_1$. Does $x_2$ offer any new information about the the label $y$ apart from the information $x_1$ provides? No, it is clear.
Now consider that $x_2$ is simply a shifted version of $x_1$, ie $x_2=c_0+x_1$. Does it now offer any new information apart from what $x_1$ provides? Again no, by simply changing the reference point $c_0$, which is equivalent to changing the baseline of the measuring system, does not  by itself offer new information.
Thirdly consider that $x_2$ is a re-scaled version of $x_1$, ie $x_2 = s
 \cdot x_1$. Does it offer now different information than what $x_1$ provides? Again, no since scaling is equivalent to changing the measuring unit of $x_1$, but this is simply a re-naming of units, no new information is added.
One can see that any transformation of the form $x_2 = c_0 + s \cdot x_1$ does not offer any new information from what $x_1$ already provides, by the arguments above.
If one wants to be more realistic one can add some noise, ie $x_2 = c_0 + s \cdot x_1 + \epsilon$, where $\epsilon$ is some random variable with zero mean and very small variance.
A correlation value $|d|$ which is quite high actualy means that $x_2$ can be expressed as such a transformation of $x_1$, ie $x_2 = c_0 + s \cdot x_1 + \epsilon$ (or vice-versa) (there are some non-trivial exceptions to this). So $x_2$ offers no new information apart from what $x_1$ offers and is redundant.
So if such a high correlation exists between variables/features, one of them can be always eliminated without diminishing performance as being redundant (in the sense that, as far as second-order statistics are concerned, the two variables are identical).
This is one of the main rationales of feature selection and dimensionality reduction in machine learning.


What methods can be used to detect anomalies in temporal texual data?

This assumes you’re happy with neural networks. If you’re not, this answer probably isn’t of much use to you.

Firstly, a little bit about anomaly detection via autoencoders. Apologies if you’re already familiar with this.

An autoencoder is a neural network which learns to reproduce its own input when compressed through a “bottleneck” layer. For example, you may want to find a lower-dimensional feature representation of a set of 100 x 100 images. Your neural network architecture has an input layer of 10,000 elements, an output layer of 10,000 elements, and one of its hidden layers will be relatively narrow compared to the input space: say 100 nodes.

The objective is to train the network to produce an output as close to the input as possible, while throwing away all but 100 nodes’ worth of activation. You are trying to produce as lossless a compression of the input as possible, so those 100 nodes should be a very information-rich representation of the kind of data you trained it on.

“How does this have any bearing on anomaly detection?” I hear you ask. Well, if you train your autoencoder on non-anomalous data, it will learn a non-anomalous lower-dimensional feature representation. This will mean the reconstruction error from pushing something through the autoencoder will be lower for data similar to what it was trained on than it will for other arbitrary data. If it receives input that is substantively different from what it was trained on, the reconstruction error will be higher. So, given a set of novel inputs, those inputs with the highest reconstruction error are the most anomalous, as they are poorly reconstructed from the non-anomalous feature representation.

If your data has a temporal structure, and you have plenty of training samples, you might want to consider constructing an autoencoding LSTM. An LSTM is a neural network architecture for encoding and decoding sequentially dependent data, and a full description of how this works is beyond both the scope of this post and my own abilities. There are many magnificent resources available online for getting to grips with this.

Here is a relevant paper on using LSTMs for anomaly detection in time series in general. It may be that LSTMs are unnecessary for your purposes if the data isn’t strongly sequentially dependent.


Calculating correlation of slightly out of sync data

At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.

The problem here is that the missing data point isn't necessarily associated with a timestamp that is the average of the timestamps above and below. If it were, the procedure you suggested would be equivalent to a linear interpolation, i.e. drawing a line between the points before and after the missing timestamp, and using that line to construct a local prediction for the missing observation. Which is what you should actually do instead.

For example in the case above, the line for the point your trying to interpolate is $\text{r_person}(ts) = 3.0 +\frac{.1}{5}(ts-23)$, so $\text{r_person}(25) = 3.04$ (I think... doing this in my head right now).


Is it conscientious to use a threshold for a model output in order to play on the recall and precision?

Good question. Using a threshold is perfectly fine and is not "manual overfitting".

It is not manual because this is a step that can (and should) be done automatically. It is not overfitting as it doesn't modify the model itself. It modifies how you interpret the model's output.

What the user did is actual called cost-sensitive learning. It is a technique where you define which error is more costly and these costs are reflected in the performance metric you use.

I can see why it may feel like overfitting but it is important to understand that all you are doing here is handcrafting a performance metric with which you will evaluate your model. Your model will simply try to optimize this metric.


Are there any machine learning techniques to identify points on plots/ images?

A first derivative, on the surface, would do it.  However, the data you show have a great deal of noise in them, so we need a way to evaluate the first derivative in a somewhat noiseless way, or at least within a frequency domain that eliminates the jitter and preserves the major derivative change.

Wavelet analysis could achieve this for you, especially if you use the first derivative of a Gaussian as your mother wavelet.  R has some decent wavelet packages (see r-project.org for starters).  If you do the wavelet transform at short scales, this will identify the locations of the bits of jitter in the steering.  If you do it at larger scales (i.e. lower frequency), you can likely find just the lane shifts and not the little jitters.

If you train the transform up with a reasonable data set, you should be able to identify a scale or range of scales that corresponds to lane changes.  But note that if you don't figure that out, this is something like O(n^2), so try to narrow the scale range down a bit to save compute time.


A/B testing with non-Gaussian distributions

Welch's t-test assumes normal distribution. I'd assume your sample size is big enough to see that these two distributions are different, based on mean, variance and range differences


Least Squares Regression $Ax=b$ when $A$ is fixed and $b$ is varied

In least squares regression, the problem that you solve is:

$X^T X \beta = X^T y$

where $\beta$ are the parameters, $X$ the regressors and $y$ the predicted variable. If you want, in your notation $A$ to be constant and $b$ to change, you need to have constant regressors and change the predicted variable $y$. I don't think there are applications that the regressors are not changed but the predictors do change, but if you want to find one it will probably be in online learning.


Deep Reinforcement Learning - mean Q as an evaluation metric

I think your main problem is use of relative distance as the core feature. It has two major weaknesses:

The distance to an object does not give the direction to the object. The best action choices are all critically dependent on direction. For example an enemy laser bolt 0.1 units directly above the player is an immediate danger requiring evasive action, whilst one 0.1 units to the left or right is not a danger and about to leave the game window. Your feature of relative distance does not distinguish between those scenarios, but it is a critical difference.

Slightly less important, but the raw distance does not capture any sense of movement. If enemies move consistently turn by turn, but not always in the exact same direction or same speed, then their velocities should also be part of the state.

One way you could improve the features is to add a velocity component for each item, showing how quickly it is approaching or receding from the player. This might help a little, but my feeling is that you need more data than distance and speed.
I think you should use normalised $x, y$ position as features for each item being tracked, plus normalised velocity $dx, dy$ for any object type that can change direction (if enemy lasers are always falling straight down you might not need anything for those).
In addition:

If the window edges are important, you should include at least the relative $x$ of one of them, so the agent knows its absolute position on screen and how much space it has to maneuver. This is true whether the player is blocked from moving further left or right, or whether the player "wraps around" to the other side of the screen. Both types of effect will significantly affect how the game plays near the screen edge.

In order to track predicted value, you need to track location of player missiles. It is not enough to just let the agent predict when it is best to fire - in order to accurately track a value function it needs to "see" whether the missile it fired some time steps ago is likely to hit or miss a target.

For both enemy lasers and player missiles, it is OK to filter and sort the data by some criteria (such as distance to player). As long as this is consistent it may even help a lot to have such pre-processing.


Which is the best algorithm for entity extraction for unstructured document

Try to clean your document and use the flair library, it's a user friendly library from Zalando Research that allows you do do all sorts of nlp tasks very quickly. Especially NER.


How to handle NaNs for ratio feature for binary classifier?

The usual solution is a mix of a few methods :

remove these records, but this should be avoided if possible
encode it based on general business knowledge, you could pick a
distinct number ( extremely large or small ) and then train the
model on it.
finally, I like usually like to plot one-way charts and
impute with values with the dependent (say churn rate) close enough
to the observed dependent rate over the NaN values for that
variable.
Edit - more recently, I have been using the package MICE and think it  could be used for imputing when you cannot do point 2 and there are far too many variables to do point 3 individually.


Is there a way to measure the "sharpness" of a decision boundary of a CNN?

You might enjoy looking into the literature on "adversarial examples".  Given an instance $x$ with a label $y$, an adversarial example is a (typically carefully constructed) instance $x'$ that is very similar to $x$, but whose label differs from $y$.  The research literature suggests that it is often possible to find adversarial examples that are very close to the original $x$.  You could use the distance $d(x,x')$ as a measure of sharpness of the decision boundary near $x$; or you could average this over many $x$ to get a global measure of sharpness.

There are many methods for finding adversarial examples.  A standard simple one is the gradient sign method, originally described in the following paper:

Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy. Explaining and harnessing adversarial examples. arXiv:1412.6572, 2014.

Since then there have been numerous improvements that find even closer adversarial examples, e.g., iterative gradient sign (arxiv:1607.02533), Deepfool (arxiv:1511.04599), and others.  You might also be interested in  Cleverhans, a software library to assist with finding adversarial examples.


Guide:

To solve the problem, we have to assume that there is the outcome of the dice and the outcome of the coins are independent.

We let $C$ denotes the event that the coin lands on tail and $D$ be the event that the dice outcome is $1$.

You have been given that the coin lands on tails or  the dice lands on $1$, and you are interested in finding out that the coin lands on tails and  the dice lands on $1$.

You want to compute $P(W|Q)=\frac{P(W\cap Q)}{P(Q)}=\frac{P(W)}{P(Q)}=\frac{P(C)P(D)}{P(Q)}$.

To compute $P(Q)$ where $Q= C \cup D$. You can either use $$P(Q)= P(C)+P(D)-P(C\cap D)$$

or $$P(Q)=1-P(Q^c)=1-P(C^c \cap D^c)$$

Given all these formulas, hopefully you can solve for $P(W|Q)$.


Temporal data is across time. Data points are correlated and you model them using a variant of RNNs (LSTMs, GRUs, Tree recursive NNs etc). example would be text script of speech. Notice how each data point would be correlated with the previous data point

local data is ambiguous and could mean a lot of things. local to a particular location? local data as opposed to server data? local to a particular task? etc etc. Please provide context


Training the Variational Autoencoder After applying the reparameterization trick

Have you read Doersch's tutorial on VAE? In paragraph 1.1 the author shows main intuitions behind the latent model and in the following sections he goes deeper into the explanation of how a VAE works. I think it's a good starting point to approach to VAEs.


Why is my neural net getting a poor accuracy?

Reason is you did not train the network.

You need to run the step op in your code. You defined it, but didn't use it anywhere later. Running the  loss only calculates the loss, but does not train your network.

after fixing your code (remove duplicate softmax, using adadelta, run step) you should get:


Evaluating performance of classifier on lopsided dataset

sklearn has weighted accuracy score which works just fine:

sklearn.metrics.balanced_accuracy_score()


Gaussian Naive Bayes (GaussianNB) classifier not working with large number of features

It is not the number of features that is the problem with Gaussian Naive Bayes (GaussianNB). It is the decision boundary that GaussianNB is learning. Naive Bayes is constrained to the learn the marginal distribution of the data because "naive" assumption. Often times the conditional distribution is useful to make predictions. Given the performance of GaussianNB on that dataset, it might make sense to apply a different classifier that can learn a different decision boundary.


Drastic increase in accuracy while using pickle file with sklearn

In the first code you split the data X randomly with this line:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1)

Then after training the model you correctly apply it to the test set, which is 20% of the instances:
y_pred = xg.predict(X_test)

Whereas in the second code you apply the model to the full data X instead of only to the test set:
X_ = data.predict(X)

So in this second code you are testing on the whole dataset which includes the 80% training set. It is of course incorrect to test on the training set, this is why you obtain artificially high performance: the model "is cheating" since it has seen 80% of the instances during training.

[answer to comment]
There are at least two possibilities to split the data properly and use the saved model only on the test data:

option 1: split the data before running the training code into two different files, one for the training set and one for the test set. The training code should be provided only the training data as input, it doesn't perform any splitting and just outputs the model. The testing code takes the model and the test data as input and outputs the predictions (or performance).
option 2: apply exactly the same splitting in the testing code as in the training code. This can be done thanks to the random_state parameter in the call to train_test_split: by using the same value for this parameter the function will always separate the data exactly the same way. Then of course use only the test set for predicting and evaluating (ignore the training set part of the data).

Option 2 is much easier to implement given your current code (you just have to copy the train_test_split call). Option 1 is more general and makes the ML design cleaner, but it's less convenient.


Optimisation strategy webstores with shipping costs

I have done the opposite of your problem - I have written code to implement shipping costs for e-commerce sites which runs on their sites.

Shipping cost rules can be almost completely arbitrary logic. In the general case, you have no good choice but to implement them as logic that is assessed per order. That means your optimiser will have to run that logic for every combination of purchases it considers. Which makes this a planning/combinatorics problem.

This may not be so bad - e.g. for purchasing 10 items across 5 stores you have $5^{10}$ combinations which is 9765625. That is possible by brute force.

For larger orders, or more choices of store, you may want to look at dynamic solutions to optimise cost, such as simulated annealing, genetic algorithms. Reinforcement learning run per order may also work.


What is the best ML/DL model to choose to calculate mobile network utilization increase

I do not see why having a percentage greater than 100% might be a problem to you? You might normalize it by dividing by 100, so 100% might become 1. That feature will be your label.
I suggest you try several regression algorithms, with different settings, and choose the best model i.e. the model which provides the best result. You have a bunch of regressors in scikit learn, you may also try XGBoostRegressor or CatBoostRegressor.


Say you have a set of pairs $(x_i,y_i)$, Somers D is linked to Kendall's tau, which is defined by :
$$\tau=\frac{N_C-N_D}{n(n-1)/2}$$
Where $N_C$ is the number of concordant pairs (ranks of both elements agree, $x_i>x_j$ and $y_i>y_j$ or $x_i<x_j$ and $y_i<y_j$) and $N_D$ the umber of pairs that are not. First it is possible that there might be a problem with the special case where $x_i=x_j$ or $y_i=y_j$, of which you have a lot of examples. This case might be handled differently depending on your implementation. Overall it quantifies the number of difference between the number of concordant and discordant pairs. By construction Kendall's tau is symetric in X and Y. (And this is probably what you wanted to use.)
Somers D, treat the variables differently. Somers D of $Y$ with respect to $X$ is defined as : $$D(Y,X)=\tau(X,Y)/\tau(X,X)$$
That is it quantifies the difference between the number of concordant and discordant pairs, divided by the number of pairs with $X$ values in the pair being unequal. You can see that it is not symmetric. It explain that there is a difference between your cases.
As to why there is such a difference : the main explaination would be a bout what is your "reference" in one case, if you set B as a reference, it is about 2 weak people (among 4) that A believe are strong, if you set A as the reference, the disagreement seems about 2 (among 38) strong people that B believe are weak.(2/4 is 50%, 2/38 is near 95%, not sure where the discrepency come from exactly)


Computing confidence interval of average output from random forest in scikit learn

In the link that you post there is a portion that describes how to compute and plot errorbars. These bars are the confidence intervals. If you prefer to have this plotted as bands you could follow this example


LSTM Timeseries recursive prediction converge to same value

If you want to use an LSTM for time series prediction, you have to try every possible parameter to find the best possible result. In order to check your model performance, you should also check your train/validation loss (this will probably show that 10 epochs is too few). Check your network architecture. Why do you use three layers? More layers won't mean better results. Try different optimizers with different learning rates. Finally, ask yourself why you are even using an LSTM for prediction? Have you tried other methods? LSTMs aren't magic, they won't have perfect prediction without any pattern to draw from, and you have pretty small samples for training.


Why for logistic regression the error is given by [y ln(sigma(x)) + (1 − y) ln(1 − sigma(x)]

This is the log-likelihood:

$\log P(x; w) \equiv \log \prod_i P(x_i | w) = \sum_i \log P(x_i | w)$, where $P(x_i | w) \equiv \left\{ \begin{array}{rl}\sigma(x_i), & y_i =1 \\ 1 - \sigma(x_i), &y_i = 0\end{array} \right.$

Why the log-likelihood? When you have a probabilistic model, such as logistic regression, it's one way (the MLE) of finding the parameters that fit best. Recall that in logistic regression we are, contrary to the name, trying to classify rather than regress, and the MSE is a regression loss; it seeks to minimize the distance from a point, while we wish to penalize being in the wrong subspace (the parts that don't correspond to the correct class). If you squint a bit, you can see that the negative log-likelihood minimizes the cross entropy.


Resize instead of transposed convolutions

Presumably, you have the seen the Distil article discussing checkerboard artifacts due to transposed convolutional layers, where they recommend upsampling (resizing) and then doing regular convolution. This is known to be a reasonable approach, so it is likely something specific to your case. Without architectural details, I can only guess that it might be (a) your hyper-parameters have been tuned to work with transposed conv layers, so it breaks with this different approach, or (b) you have introduced some kind of vanishing gradients or convergence issue (e.g., not using residual connections, not using batch norm, too large of a learning rate, etc...). The loss (e.g., $L_2$ reconstruction) and it's gradients may be of a different magnitude than what the other (hyper-)parameters were expecting, for example. Feel free to post back with more info.


Those rule of thumbs for ratio make no sense. The use case for a test set is to measure your performance in a real scenario on unseen data. How much you need for an accurate measure depends on the level of accuracy you need, the amount of variance you expect, not so much on the size of the original dataset. Something to be aware of is that you do decrease the size of your training set, which is a cost of testing. But when you have 10 million rows of data it makes no sense to validate on 3 million, when 10,000 could be more than enough. Outside of the cost of losing training data you should look at absolute numbers in your test set.


Why am I getting very different results between SVC, LinearSVC and Naive Bayes?

These models are used for convex optimization. which means that there is only one solution for the problem. LinearSVC and SVC are used for same purpose but the optimization techniques used are different. E.g. LinearSVC intercept is penalized while in SVC it isn't. there might be differences scaling or default loss function. Hence they produce different results. LinearSVC tends to optimize faster. in multiclass classification, liblinear does one-vs-rest by default whereas libsvm does one-vs-one.

if you want to know how they can produce similar results. You can check out this question.
https://stackoverflow.com/q/33843981/5947203

Now the reason why SVN produced better results than Naive Bayes is that in this problem features are important. Naive Bayes treat them as independent while SVN looks at the interactions between them to a certain degree. Mathematically, One is probabilistic while other is geometrical. The dependencies are not captured by Naive Bayes hence it does not produce good results.


Which accuracy metric of a ML classifier can maximize map@K of a recommender system for an unbalanced dataset?

If you want to stay in Python, I would suggest going for xgboost instead of GradientBoostingClassifier - among other advantages, it supports map / ndcg as metrics. If you are considering R, gbm supports both map and ndcg.


GPU computing: how much VRAM do I need for mini batch gradient descent?

Generally memory cost increases linearly while mini-batch size increases. If batch size 32 costs you 2GB memory, then batch size 64 will cost you 4GB memory.

In practice, deep learning framework(for instance, keras) will load the entire full-batch into your memory, thus you cannot observe apparent memory increment or decrement during mini-batch tuning.

References

https://github.com/fchollet/keras/issues/146


How to estimate missing values when calculating NDCG

Movie 2 get second best score but it has not been rated by user although it's highly relevant for user A based on other user ratings.

The issue with this is that you're injecting your opinion into what items are relevant to user A.

How NDCG@3 should be calculated in this example ?

You cannot evaluate your model on items the user has not rated. Generally with explicit feedback data you want to ignore missing values. Imputation may be possible, but it is more challenging and not commonly used. The correct modeling pipeline is to first make a training & testing set. Train your model on the training set and then for each user rank their known items that are present in the testing set (source: p.g. 24). Then you can compute various cutoff-sensitive metrics.
In your example, you should not be predicting ratings for movie 2 or 5. If your training set was movie 1 & movie 3 then you would make predictions for movie 4 -- obviously in a real application you should have more data so that there is more than one example in your test set.


Can anyone explain me the fisher score working

From Feature Selection for Classification: A Review (Jiliang Tang, Salem Alelyani and Huan Liu)

  Fisher Score: Features with high quality should assign similar values to instances
  in the same class and different values to instances from different
  classes.

From Generalized Fisher Score for Feature Selection (Quanquan Gu, Zhenhui Li, Jiawei Han)

  Fisher Score: The key idea of Fisher score is to find a subset of features,
  such that in the data space spanned by the selected
  features, the distances between data points in different classes are
  as large as possible, while the distances between data points in the
  same class are as small as possible.

If you are interested in the mathematical explanation, I would suggest the second paper, on section 2 (A Brief Review of Fisher Score)


Yes, you should save result files before you make major mods to the code.  Disk space is cheap, so you're unlikely to run into issues unless your results sets are prolific.  I would suggest storing old results sets with folder names that include a time stamp of when they were generated.

As far as time shots of your code, using github (or some other code repository tool) is as easy as can be, will save version information, allows for collaboration, and is an all around great way to backup and version your code.

The combination of these two things, you'll effectively have an easy way of mapping a result set to a specific version of the code.


Theoretical treatment of unlabeled samples

You already dropped the key word "semi-supervised" in your question. Indeed, semi-supervised learning is the answer to your question. Search for this term with your favourite search engine or library catalog to find out about algorithms for semi-supervised learning.


Audacity is a free and open source audio editing software, available for all common desktop operating systems. It can be used to annotate audio, by using Label Tracks. You can have a label at a certain position, or covering a selection of time.

Annotating

Open or import your audio into Audacity
Select the audio track you want to annotate.
Click Track -> Edit Labels to add a new Label track
Select where in the track you want to annotate
Insert new label using keyboard shortcut Ctrl + B

Repeat until done with the track.

Export annotations

Export the labels using File -> Export -> Export Labels

Loading annotations

To load the labels into Python, you can use this snippet of code:

import pandas

path = 'PATH/TO/FILE'
labels = pandas.read_csv(path, sep='\t', header=None,
                        names=['start', 'end', 'annotation'],
                        dtype=dict(start=float,end=float,annotation=str))


For point 1. and 2., yes. And this is how it should be done with scaling. Fit a scaler on the training set, apply this same scaler on training set and testing set.
Using sklearn:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit_transform(X_train)
scaler.fit(X_test)

Regarding binarizing, I think you should not have this problem. Only if you want to choose the threshold dynamically regarding a dataset. If so, you should only use training dataset to choose it.


How to Extract Information from the Image(PNG)

If all of your images are similar to this one(or have a small set of possible designs), you can simply reference the location (pixel-wise) on the image where this fields are and slice it.

After slicing you can use any OCR algorithm to extract that data.

If your data has more variation than that, you can use OCR on the entire image, which is usually a slow algorithm.

If you have less than 50 or so images to do that, it is more efficient to do this by hand (not worth writing code)


Reinforcement Learning: Policy Gradient derivation question

Actually in the article there is a $\approx$ rather than an $=$. This is because you can approximate expectation values by sampling the respective distribution.

Assume you want to compute

$$ E \left[ f(x) \right] = \int p(x) f(x) dx$$

The integral might not be tractable, you might not even know the distribution $p(x)$. But as long as you can sample from $p(x)$, you can approximate quite well using the Monte Carlo estimator

$$ E \left[ f(x) \right] \approx \frac{1}{N} \sum_{i=1}^N f(x_i)$$

with $x_i \sim p(x)$ for all $i$. This approximation would get better for larger $N$. The distribution $p(x)$ is, in some sense, represented by the samples $x_i$ and their respective frequency.

This is what is going on in the article. You want to compute the expectation value over all possible trajectories, but that is infeasible. Luckily though, you can sample the distribution by running simulations of the environment. The expectation is then approximated using samples of trajectories. The $\pi_\theta(\tau)$ is represented in the sampled $a_{i,t}$ and $s_{i,t}$.

In short, the expectation over all possible paths is approximated by the mean over samples of paths.


Starting search from rightmost column and moving left pandas

Found the answer, putting

if "final cost" in df.columns[k]:

instead of

if df.columns[k] = "final cost":

seemed to have fix it


What is the proper proportion for train and test set for classification system?

In short theres no best proportion or rule of thumb.
It's highly dependent on how much data you have, its distribution in relation to the number of labels, and if samples are related to each other in anyway or are they completely independent of each other.  You could look into k-fold training. Say you split your data into 5 sections each 20 percent of the size of the entire dataset. For each split of 20 percent of data use the remaining 80 percent of data as the training set. Go through each 20 percent of data and use that as the test and the rest as the training. Make sure your labels are evenly distributed between splits.
K-fold reference


Topics to cover for software developer interested in data analytics

Welcome to the community!! There can be a lot of answers to this question but I would suggest you the approach I took when I shifted from software development to the data science field.

1) Refresh your statistics and probability concepts. You should not go into too much details but you must understand basic things like Gaussian Distribution, Mean, Variance, Probability,etc.

2) Go through the basics of Machine Learning concepts. I prefer Andrew Ng's machine learning course on Coursera. That will help you build a strong foundation and will give you a great start in the field

3) Choose a particular language Python/R for building models. Though it's totally upto you but I prefer python as it has great libraries for machine learning as well as Deep learning.

4) Take part in competitions. We learn by doing not by taking only lectures. I suggest you should join Kaggle and the slack community out there namely, 'KaggleNoobs'. It's a great community. I learn everyday a new thing from there.

P.S: Data Science is a vast field. It demands from you various skill set like Data Analysis, Data Visualisation, Machine Learning,etc. So sometimes it can become frustrating too. But once you start enjoying it, you will become a master eventually.


Plotting different values in pandas histogram with different colors

There isn't any built-in function to do this directly in pandas, but by getting the array collection of AxesSubplot, iterating on them to retrieve the matplotlib patches you can achieve the desired result.

Here's some dummy data to play with:

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randint(low=0, high=3, size=(1000,16)))

Now, here's the magic:

import matplotlib.pyplot as plt

# Plot and retrieve the axes
axes = df.hist(figsize=(12,6), sharex=True, sharey=True)

# Define a different color for the first three bars
colors = ["#e74c3c", "#2ecc71", "#3498db"]

for i, ax in enumerate(axes.reshape(-1)):
    # Define a counter to ensure that if we have more than three bars with a value,
    # we don't try to access out-of-range element in colors
    k = 0

    # Optional: remove grid, and top and right spines
    ax.grid(False)
    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)

    for rect in ax.patches:
        # If there's a value in the rect and we have defined a color
        if rect.get_height() > 0 and k < len(colors):
            # Set the color
            rect.set_color(colors[k])
            # Increment the counter
            k += 1

plt.show()


I see you have done encoding of feature line using the below code:

line_dict = {'INSPECTION':'0', 'REPAIR':'1' }

if we see the type of the value as below:

   >>> line_dict['INSPECTION']
       '0'
   >>> type(line_dict['INSPECTION'])
   <class 'str'>

it seems it is of object type. This could be the problem. Suggest you do to one hot encoding technique for this feature

Also it is good to an info on your dataset to identify the object columns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 153381 entries, 0 to 153380
Data columns (total 12 columns):
bodyId        153381 non-null int64
bodyTypeId    153381 non-null int64
color         153381 non-null int64
line          153381 non-null object
shiftNo       153381 non-null int64
Inspection    153381 non-null object
mYear         153381 non-null int64
mMonth        153381 non-null int64
mDay          153381 non-null int64
mHour         153380 non-null float64
mMinute       153380 non-null float64
mSecond       153380 non-null float64
dtypes: float64(3), int64(7), object(2)
memory usage: 14.0+ MB

if you check the data type line  it says object which means the line feature is String


How does sklearn KNeighborsClassifier compute class probabilites?

The class probabilities are the normalized weighted average of indicators for the k-nearest classes, weighted by the inverse distance.

For example: Say we have 6 classes, and the 5 nearest examples to our test input have class labels 'F', 'B', 'D', 'A', and 'B', with distances 2, 3, 4, 5, and 6, respectively.

Then the unnormalized class probabilities can by computed by:

(1/2) * [0, 0, 0, 0, 0, 1] + (1/3) * [0, 1, 0, 0, 0, 0] +
(1/4) * [0, 0, 0, 1, 0, 0] + (1/5) * [1, 0, 0, 0, 0, 0] +
(1/6) * [0, 1, 0, 0, 0, 0] =

[1/5 ,1/2, 0, 1/4, 0, 1/2]

And after normalizing we get:

[0.13793103, 0.34482758, 0.0, 0.17241379, 0.0, 0.34482758]

If you choose weights='uniform' then it becomes a simple average, which can be computed by multiplying each indicator by (1/k) and summing, with no need for additional normalization.


Feature transformation possible at selected features or only at all?

Either can be good or bad. It's not necessary to apply the same transformation to all attributes; but if youe attributes differ that much, you likely have some major problem anyway.

It depends on what you want to achieve what is appropriate. Try to approach it from having to explain your choices later. Can you argue why you transform the data this way? What is the statistical meaning of this transform for your problem? Here you'll quickly realize that with mixed data you can't really argue much - your results are based on guessing, not in a sound model.


Does Non Negativity Constrains increases the estimation error

Are you talking about train error or test error?
Non negativity constrains make your search space of possible solutions much smaller. This means that the train error would be always higher or equal when you add non-negativity constraint.
But higher train error can mean that you are overfitting. If the assumption of non-negativity make sense for the solution of your problem, then it is likely that this assumption would decrease the test error as you are limiting your function space and decrease the chance of overfitting.


ValueError: y should be a 1d array, got an array of shape (1045, 5) instead

Your data is multilabel: each row can have more than one label. But sklearn's Naive Bayes doesn't support that format of problem. You can use MultiOutputClassifier to wrap the Naive Bayes classifier, effectively training one model for each of the labels.
See the User Guide, especially the MultiOutput classification section. You may also want to consider the ClassifierChain (in the next section).

The last error you see is because you're using pandas structures, but reshape is a numpy method. Anyway, as @Oxbowerce observed, reshaping won't help; you need to deal with having multiple labels.


Is there an algorithm for categorizing unlabeled samples into K classes?

Best solution would be to use the inception pertained model in your case as it's very good with over 1000 classes. Also you can label some of the data and make a CNN in Tensforflow, and then use this data to classify other data.


Doubt regarding the number of weights in 2 layer neural network

All units in one layer gets connected to all units in the next layer. To accomplish this all units in the first layer will need a weight per unit in the last layer. Therefore you get the number of weight be multiplication and not addition like you assumed.

Let's for example add one more layer with 3 units in the end and maybe it will be clearer. We have layers with units [10, 5, 3]. Then we will have $W_1: 10\times5=50$ weights between the first two layers and $W_2: 5\times3=15$ between the last two layers and together they will be $W_1 + W_2 = W <=> 50 + 15 = 65$.


How to find the various matrix sizes in designing a CNN

For a CNN layer with input of dimensions h * w * d, kernel size k * k and number of kernel filters as f, we have the number of parameters as k * k * d * f, if we ignore the biases. If use biases then the number of parameters becomes (k * k * d + 1) * f

For e.g., the 1st conv layer has 5 * 5 * 2 * 20 parameters if we are ignoring the biases. With bias, the number of parameters would become (5 * 5 * 2 + 1) * 20.

Note that the number of parameters does not depend on the stride, padding, pooling, dropout nor on the spatial dimensions of the input or output!

To find the total number of parameters in the network one needs to add the parameters of all the individual layers in the model.


Cast regression problem into classification problem

The statement in principle seems very vague, how can one state that there are better classification than regression algorithms?
With that said I would rephrase the statement to:
Sometime it is feasible to turn a regression into a classification problem due to for the problem itself, it makes sense to predict a range/bin instead of a continuous value.
When doing this we have to be careful since we are not facing a "common" classification problem but we have an ordinal classification problem in which we have a natural order from the new target (bins)
For reference check:
https://stats.stackexchange.com/questions/493254/why-ordinal-target-in-classification-problems-needs-special-attention


How to derive association from a regression model?

I think NN is a little problematic here, because you run into the bias-variance-tradeoff. NN are really good in producing estimates with low variance. However, if you look for a causal influence of $x$ on some $y$, you would choose an unbiased estimator (with higher variance). Options are OLS or Lasso/Ridge estimators. Especially with DNA information, Lasso is a good choice since you may face a problem with (relatively) high dimensionality.

If you really want to explore feature importance in a NN setting, I guess you need to resort to stepwise forward/backward techniques similar to the method as used for feature selection. See here and here for some background. To my best knowledge Keras does not come with a predefined function for the problem.


building a 2-layer LSTM for time series prediction using tensorflow

First question: How many layers?

This is architectural question and one of them most important when constructing NN. Generally the more complex the task the more layers you should use to approximate (until a certain point than there is overkill, motivation for ResNet)
If you are looking for some guidelines there are some good posts, but the research and general trend nowadays is that we are over-doing it in the first place, and we can achieve some good results with some smart tricks without making it too deep.

TL;DR it depends on the problem, but not as deep as we think for 98% of problems.

Second (third?) question: why is there a need for the adding a Dense(72) layer ? what is the function Dense() doing ? Well you said that Finally, since 72 predictions are made, the dense layer outputs 72 predictions. For the question what is dense function doing in short its producing an (output) vector. In long dense layer represents a matrix vector multiplication. Values in the matrix are actually the trainable parameters (weights) which get updated during backpropagation, and if you seen mathematically representation of NN with matrizes (which all of them are thats how you utilise power of GPU-s bla bla) thats exactly what this dense layer represents.


IndexError: index 804 is out of bounds for axis 0 with size 800

One of the values being returned by touch.x or touch.y are bigger than the dimensions of the sand array.

You could try adding a try/except block around it and print a useful message - just as an example, as I have no idea what this programme is doing (in the on_touch_down function):

def on_touch_down(self, touch)
...
...
    try:
        sand[int(touch.x), int(touch.y)] = 1
    except IndexError:
        print("Touchdown did not occur within the end-zone! No goal!")


Well, you want to identify change in usage you could try something like:

$$ f(day_1,day2) = \frac{day_2-day_1 + \delta}{||day_2-day_1+\delta||} \times \Biggr|\Biggr|\frac{day_2+day_1}{(day_2+day_1+1)(day_2-day_1+1)}\Biggl|\Biggl| $$

where $\delta$ is the eps of your machine (minimum value needed to be summed to differ it from other floats)

that will give you
$$f(100,0) \approx -98.02$$
$$f(0,100) = 100$$
$$f(100,100) \approx 0.995$$
$$f(0,0) = 0$$

You can look at my experiment here

This will map all non-changes from $[0,1]$ where $f(0,0)$ maps to $0$ and $f(\infty,\infty)$ maps to $1$

Where is it from? Just tuned the function manually. But I think this might suffice for your application

Explaining the idea

You want to have a feature that packs a lot of information:
 - Is the usage greater than zero?
 - Is it increasing or decreasing?
 - If it is stalled, how much is the usage?

Well, your usage vary in integer values so you can map the entire non-changing but above 0 case to a previously non-used interval.

The function above will map in $[0,1]$ all non-changing possibilities, in a exponential kind of way ($a^{(-\frac{1}{usage})}$) also you can extract the actual value from positive changes and the approximate value for negative change (been a better approximation when the drop is high)

This is not the perfect scenario but it is the maximum information I could compress into 1 variable with little loss.


Should I apply 1D or 2D CNN on binary text classification?

The standard way would be to apply 1D convolution.

While technically there is nothing preventing you to implement a 2D convolution over your textual representations, they would be "less expressive" than normal 1D convolutions:

In a normal 1D convolution, the kernel would have depth 100 and the width you choose (e.g. 3, 5). The kernel is slid through the "time" dimension and is computed over the whole channel dimension:

In order to compute a 2D convolution over your text, the 2 dimensions would be time and channels. You could have the kernel height be 100 and therefore take the whole channel dimension, but this would be analogous to a 1D convolution, no there is no point in doing so. Therefore, for having a 2D convolution, you would have a kernel that does not cover the whole channel dimension at a time:

Therefore, while it is possible to have a 2D convolution over text, it would be less expressive that the 1D convolution kernel, which covers the whole channel dimension and can, therefore, express any computation a 2D convolution can

So the answer is that you should use a 1D convolution. This is what makes sense and what people do.

(Images were taken from here and here)


chart x-axis spacing terminology question

It's usually called "Tick" and as it's for x axis, let's say "xtick". You can label it as xticks and give it two values "evenly spacing" and "actual spacing" (This is just an idea as there is no standard terminology as far as I know).


Formulate a MDP for a problem given below

I would like to know, is there any procedures or rules, that needs to be considered before formulating an MDP for a problem

For a simple problem like this, I would do the following:

List the states.

You have missed a state - the terminal state of leaving the building

Mark terminal states

In this case, only exiting the building is a terminal state. The hungry tiger doesn't actually do anything to Bunny, the problem states that the MDP should just continue.

List the actions possible in each state.

In your starting diagram, you do not show actions, and this is already limiting your ability to express the MDP.

List the possible transitions and probabilities of them

For most action choices in the example, there are two possible outcomes from each action. The problem doesn't actually describe what happens when there is only one door, for the room with the tiger, but it seems reasonably safe to say that it is not possible for Bunny to get confused when there is only one door, so exiting the room with the tiger is one exception.

Assign rewards. Rewards should relate to the goals stated in the problem definition. Negative rewards for things to avoid happening (or in general to avoid time wasting). Positive rewards for reaching desired targets.

In your case you have two goals - avoid tiger and exit building. There does not appear to be any time constraint.

Typically you would show action choices as labelled arrows going from states to an action node (usually just a small solid circle) where the state transitions then take effect, with further arrows labelled with probabilities going to states. So State A should have two arrows going out "choose left door" and "choose right door", then these choices will branch again - "choose left door" will go to state B with p=0.9 and state C with p=0.1.

You have some free choices over where to put reward labels, and there are a few different standard approaches. In this case, I would put the non-zero rewards on the state transitions that lead to goal-affecting states. E.g. maybe a +1 reward on the state transition from B taking the exit door and ending up outside. Note that for both the exit and the room with the tiger, that there are two possible ways to get to them due to the random transitions (thanks to Bunny getting confused).

Note there are no actions, and no possibility of different transitions, where Bunny starts in state C and ends up in state C on the next timestep. I think you are confusing state C as being a terminal state - although this is understandable as some encounters with hungry tigers in the real world could well be terminal, the problem clearly states that everything just carries on for Bunny if he encounters the tiger. It is just something he wants to avoid. If he ends up in state C, then he still has an action he can take to go through the door. When he takes this action, he cannot "get confused" and pick the wrong door, so should always end up in state B with p=1.0.

If you decide to draw a transition diagram, you have free choice of layout. There is no fixed approach for this, and for many MDPs it may not be possible to draw a nice 2D diagram. However, in your case, it is definitely possible to layout the states in a simple 2D arrangement. A good place to start would be to place the states in locations that correspond to the imaginary room map. Leave plenty of space initally, as the states do not link directly (as you have started), but link via action nodes which themselves branch to the next state.

I hope this enough guidance for you to complete the problem by yourself. Solving the remainder on your own will be much better for your learning than turning it into a worked example here.


When to use Linear Discriminant Analysis or Logistic Regression

This question was asked and answered at the Cross-Validate SE. The answer is a few years old but it is still useful. If you want you can click here to find their answer.


How does Neural Network denoise an image?

in general CNN can generate images, so we can feed them noised images and images without noise at output and it can learn how to figure out pixel color based on his neighbours, actually you can't reconstruct perfectly because noise cause removal of some information from image and all you can do it to find "as best as possible" values to restore original image

for more info this clip is very informative
https://www.youtube.com/watch?v=z4vl3Z6NFW8


Can we optimize regression problems that have categorical variables by encoding them if on the other hand we are inserting multicollinearity?

Multicollinearity can be a problem if you choose to optimize linear regression with ordinary least squares (OLS). Because the data matrix $X$ can have less than full rank, therefore the moment matrix $XᵀX$ cannot be inverted.

If you choose to optimize linear regression with gradient descent, multicollinearity is not an issue in finding an optimal solution.


Clustering Weekday Weekend Data and Multicollinearity

Yes its called correlation clustering.

Even though correlation can cause problems with many clustering algorithms by giving extra weight on these attributes, it would be best to drop highly correlated variables for example with PCA

However, there exist correlation clustering algorithms that are meant to process data containing multiple correlations, and cluster objects based on the correlations they exhibit, using your problem exactly to an advantage.


Yep - you've overfitted !

Can you see what your accuracy is as a function of the number of training cycles? It would be interesting to see how that looks.

Is there any way you could increase your data? 15000x300 isn't a huge amount of depth for a neural network, considering the width of the data set.

You could look at the impact of increasing your regularisation parameters, or decreasing the depth of your network


Thrift is generally faster because the data being exchanged is smaller.  Stargate offers a web service which is an integration method that is widely supported, which is a concern when you are working with commercial products with limited integration possibilities.

In a closed environment where everything is controlled, I would prefer Thrift.  If I'm exposing data to external teams or systems I would prefer Stargate.


How to plot train test error for classification models like Support Vector Classification(SVC)

Well you haven't defined what "error" means, so I'll just assume that you want the log loss.

First, you need to create your SVC object telling it you'll want probability estimations:

model = sklearn.svm.SVC(probability=True)

Then you can compute the log loss given said estimation:

probs = model.predict_proba(x)
loss = sklearn.metrics.log_loss(y_true=y, y_pred=probs)


Determine relationship between users and age?

If you are using pandas, all you need to do is:
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

corrMatrix = df.corr()

Then you can print the correlation matrix and also plot it using seaborn or any other plotting method.
sns.heatmap(corrMatrix, annot=True)
plt.show()

Hope this helps.


1) Split data in smaller blocks

2) learn clf on first block

3) Dump model into pickle

4) load second block

5) load model from pickle

6)
    learn clf on second block and so on


What do I initialise each model in cross validation with in a multi-layer Perceptron?

I think you are trying to do cross validation with hyperparameter tuning, so here is how it's done with k-fold CV (from this answer):

You can split your data into 2 datasets: training and test. k-folds cross validation takes a model (and specified hyperparameters) and partitions the training dataset into k equally sized subsets. Then, it does the following k times:

Trains the model on k-1 of the subsets
Evaluates the model accuracy on the subset that wasn’t trained on.

It then reports the average error. To do hyperparameter tuning, do the steps above using every time a different hyperparameter combinations. Then, choose the set of parameters for which k-folds reports the lowest error. However, be careful to not excessively minimize the k-folds error, since it will often lead to overfitting.

Ultimately, we want a measure of how well our final model will generalize. This is why we created the test set at the beginning—evaluating the model’s accuracy on this set is a useful estimation of its success.

So, k-fold doesn't mean k different models, but k folds of the dataset!

To reply to your questions:

  Will the training of these individual model be as usual(backward propagation)?

Yes each training is as usual, just changes the training set and the hyperparameters.

  What do I initialise each model with? (Random Weights?)

Yes, (still) as usual in neural networks. You are not re-using old weights.

  After completing the cross-validation, I have the best model(say B) with me now, what does it mean to train this model on the entire dataset? (Does it mean, I initialise the weights of the new model being trained on the whole dataset, with those of B).

Well, you are mistakenly exchanging again weights and hyperparameters, but, if you have a very big dataset and cross validating on the entire dataset takes too long, you can:

take a portion of your dataset (maybe 10%), let's call it A
Use A to find the best hyperparameters using k-fold CV as I described before
Now you can use the entire dataset for training (except a test set) the model using those best hyperparameters. With the hope that's the real best model.


Optimal points of $f(x,y)=x^2 + y^2 + \beta xy + x + 2y$

If the function is convex, then all local minimum are global minimum.
If $4-\beta^2 >0$, then the function is convex and hence the local minimum is indeed the global minimum.
If $\beta = \pm 2$, then we have $f(x,y)=(x\pm y)^2+x+2y$, $f(x, \mp x)=x\mp2x$ of which we can make it arbitrary large or small.
If $4-\beta^2 < 0$, then it is indefinite, the stationary point is a saddle point.
It can't be negative definite, it doesn't have global maximum.