How to verify that a trained neural network is trustworthy?

A definitive and deterministic answer regarding the existence of adversarial samples is only possible by checking the entire sample space, which you said is finite and hence feasible but maybe too computationally expensive.
I would suggest to generate a grid of samples over the entire space to get a somewhat representative subspace and evaluating the model on it. As you suggested, the normal error distribution out of that whould give you a more or less reliable probabilistic statement on future model errors, depending on the size of the subspace I guess.
However, I also am curious if there is a better way to evaluate ones model reliability.


An alternative algorithm to sklearn KMeans that separates values by similarity?

Welcome, Dmitry!

A very first red alarm ... If you are doing clustering, then let it tell you who is similar to who! If you already know your clusters, then why are you doing clustering indeed?!

And about the algorithms: Yes there are many more, for example, Spectral Clustering which makes use of different similarity measures (via Gaussian Kernel or K-NN connectivity matrix) to find clusters or DBSCAN which is based on similarity according to the density of data points.

If you needed more help please drop a comment here!

Good Luck!


Train, test and submission files - what am I supposed to do with all of them?

The submission file is just there as a reference. You are not supposed to do anything with it, but when you make your own submission it be on the same format.

Regarding your high accuracy. Try making your own submission file and submit it to Kaggle and it will show you if your score is real or not. :)

link for submission


What is the purpose of 'oversampling' when the test set is still unbalanced?

Could someone please explain why do we use oversampling when both
  training and testing sets need to have the same distribution?

We use it because train and test set does not have to have the same distribution of labels. What is important is that the test set comes from the same distribution as your "real" data so that it provides a reliable measure. Post split the training set can be sampled as you see fit, as long as the results are good on your test set.


How to model predict on Tensorflow model that has feature columns?

You can find prediction examples in the next tutorial: Classify structured data using Keras Preprocessing Layers.
The list of tutorials in the series is in the bottom left part:

Specifically, the prediction examples are in the "Inference on new data" section (the last one in the tutorial).


Simple example of genetic alg minimization

Here is a trivial example, which captures the essence of genetic algorithms more meaningfully than the polynomial you provided.  The polynomial you provided is solvable via stochastic gradient descent, which is a simpler minimimization technique.  For this reason, I am instead suggesting this excellent article and example by Will Larson.

Quoted from the original article:

  Defining a Problem to Optimize Now we're going to put together a simple example of using a genetic algorithm in Python. We're going to
  optimize a very simple problem: trying to create a list of N numbers
  that equal X when summed together.

  If we set N = 5 and X = 200, then these would all be appropriate
  solutions.

lst = [40,40,40,40,40]
lst = [50,50,50,25,25]
lst = [200,0,0,0,0]

Take a look at the entire article, but here is the complete code:

# Example usage
from genetic import *
target = 371
p_count = 100
i_length = 6
i_min = 0
i_max = 100
p = population(p_count, i_length, i_min, i_max)
fitness_history = [grade(p, target),]
for i in xrange(100):
    p = evolve(p, target)
    fitness_history.append(grade(p, target))

for datum in fitness_history:
   print datum
"""
from random import randint, random
from operator import add

def individual(length, min, max):
    'Create a member of the population.'
    return [ randint(min,max) for x in xrange(length) ]

def population(count, length, min, max):
    """
    Create a number of individuals (i.e. a population).

    count: the number of individuals in the population
    length: the number of values per individual
    min: the minimum possible value in an individual's list of values
    max: the maximum possible value in an individual's list of values

    """
    return [ individual(length, min, max) for x in xrange(count) ]

def fitness(individual, target):
    """
    Determine the fitness of an individual. Higher is better.

    individual: the individual to evaluate
    target: the target number individuals are aiming for
    """
    sum = reduce(add, individual, 0)
    return abs(target-sum)

def grade(pop, target):
    'Find average fitness for a population.'
    summed = reduce(add, (fitness(x, target) for x in pop))
    return summed / (len(pop) * 1.0)

def evolve(pop, target, retain=0.2, random_select=0.05, mutate=0.01):
    graded = [ (fitness(x, target), x) for x in pop]
    graded = [ x[1] for x in sorted(graded)]
    retain_length = int(len(graded)*retain)
    parents = graded[:retain_length]
    # randomly add other individuals to
    # promote genetic diversity
    for individual in graded[retain_length:]:
        if random_select > random():
            parents.append(individual)
    # mutate some individuals
    for individual in parents:
        if mutate > random():
            pos_to_mutate = randint(0, len(individual)-1)
            # this mutation is not ideal, because it
            # restricts the range of possible values,
            # but the function is unaware of the min/max
            # values used to create the individuals,
            individual[pos_to_mutate] = randint(
                min(individual), max(individual))
    # crossover parents to create children
    parents_length = len(parents)
    desired_length = len(pop) - parents_length
    children = []
    while len(children) < desired_length:
        male = randint(0, parents_length-1)
        female = randint(0, parents_length-1)
        if male != female:
            male = parents[male]
            female = parents[female]
            half = len(male) / 2
            child = male[:half] + female[half:]
            children.append(child)
    parents.extend(children)
    return parents

I think it could be quite pedagogically useful to also solve your original problem using this algorithm and then also construct a solution using stochastic grid search or stochastic gradient descent and you will gain a deep understanding of the juxtaposition of those three algorithms.

Hope this helps!


Using a pre trained CNN classifier and apply it on a different image dataset

Do Read - When Transfer Learning is Disadvantageous ?(thanks to @media)
(looks very informative to me, so added here to make this answer a complete one...)
Answer to your Question..(starts here)

Transfer Learning is What you are Looking for..

When we are given a Deep Learning task, say, one that involves training a Convolutional Neural Network (Covnet) on a dataset of images, our first instinct would be to train the network from scratch. However, in practice, deep neural networks like Covnet has a huge number of parameters, often in the range of millions. Training a Covnet on a small dataset (one that is smaller than the number of parameters) greatly affects the Covnet’s ability to generalize, often result in overfitting.

Therefore, more often in practice, one would fine-tune existing networks that are trained on a large dataset like the ImageNet (1.2M labeled images) by continue training it (i.e. running back-propagation) on the smaller dataset we have. Provided that our dataset is not drastically different in context to the original dataset (e.g. ImageNet), the pre-trained model will already have learned features that are relevant to our own classification problem.

When to fine tune Models?

In general, if our dataset is not drastically different in context from the dataset which the pre-trained model is trained on, we should go for fine-tuning. Pre-trained network on a large and diverse dataset like the ImageNet captures universal features like curves and edges in its early layers, that are relevant and useful to most of the classification problems.

Of course, if our dataset represents some very specific domain, say for example, medical images or Chinese handwritten characters, and that no pre-trained networks on such domain can be found, we should then consider training the network from scratch.

One other concern is that if our dataset is small, fine-tuning the pre-trained network on a small dataset might lead to overfitting, especially if the last few layers of the network are fully connected layers, as in the case for VGG network. Speaking from my experience, if we have a few thousand raw samples, with the common data augmentation strategies implemented (translation, rotation, flipping, etc), fine-tuning will usually get us a better result.

If our dataset is really small, say less than a thousand samples, a better approach is to take the output of the intermediate layer prior to the fully connected layers as features (bottleneck features) and train a linear classifier (e.g. SVM) on top of it. SVM is particularly good at drawing decision boundaries on a small dataset.

Fine-tuning Techniques

Below are some general guidelines for fine-tuning implementation:

The common practice is to truncate the last layer (softmax layer) of the pre-trained network and replace it with our new softmax layer that are relevant to our own problem. For example, pre-trained network on ImageNet comes with a softmax layer with 1000 categories.

If our task is a classification on 10 categories, the new softmax layer of the network will be of 10 categories instead of 1000 categories. We then run back propagation on the network to fine-tune the pre-trained weights. Make sure cross validation is performed so that the network will be able to generalize well.

Use a smaller learning rate to train the network. Since we expect the pre-trained weights to be quite good already as compared to randomly initialized weights, we do not want to distort them too quickly and too much. A common practice is to make the initial learning rate 10 times smaller than the one used for scratch training.

It is also a common practice to freeze the weights of the first few layers of the pre-trained network. This is because the first few layers capture universal features like curves and edges that are also relevant to our new problem. We want to keep those weights intact. Instead, we will get the network to focus on learning dataset-specific features in the subsequent layers.

You need to train them again in this case as if I am not wrong then the wallpapers is not a class of Image-net models.. It won't be tough to build a model from scratch to do so(preferably a shallower one will also do here..)

Source of my answer is translation from these amazing courses..

CS231N
fast.ai

For Further Readings,

There is a difference between using pre-trained weights and transfer learning.,
Karpathy's  Latest NN's Training Recepie BlogPost..


In supervised learning, are more data entries always better?

That is a clue your data is missing-not-at-random.
The missing data is a result of a systemic process that also effects predicting target performance.


Getting TypeError: expected bytes, Descriptor found while importing tensorflow

I resolved the issue by creating new environment with tensorflow by the below two commands:

conda create -n tensorflow_env tensorflow
conda activate tensorflow_env


K-Fold Crossvalidation in Tensorflow when using flow_from_directory for image recognition

The easiest way I found was replacing flow_from_directory command to flow_from_dataframe (for more information on this command see).

That way you can split the dataframe. You just have to make a dataframe with images paths and labels.

 i = 1
 df_metrics = pd.DataFrame()
    kf = KFold(n_splits = 10, shuffle = True, random_state = None)
    for train_index, test_index in kf.split(dataframe):
        trainData = dataframe.iloc[train_index]
        testData = dataframe.iloc[test_index]
        print('Initializing Kfold %s'%str(i))
        print('Train shape:',trainData.shape)
        print('Test shape:',testData.shape)
        epochs = 30

    train_datagen = ImageDataGenerator(rescale=1./255,validation_split=0.2)
    test_datagen = ImageDataGenerator(rescale=1. / 255)

    train_generator=train_datagen.flow_from_dataframe(
    dataframe=trainData,
    directory="./train/",
    x_col="id",
    y_col="label",
    subset="training",
    batch_size=batch_size,
    shuffle=True,
    class_mode="categorical",
    target_size=(img_width, img_height))

    validation_generator=train_datagen.flow_from_dataframe(
        dataframe=trainData,
        directory="./train/",
        x_col="id",
        y_col="label",
        subset="validation",
        batch_size=batch_size,
        shuffle=True,
        class_mode="categorical",
        target_size=(img_width, img_height))

    test_generator=test_datagen.flow_from_dataframe(
        dataframe=testData,
        directory="./test/",
        x_col="id",
        y_col="label",
        batch_size=1,
        shuffle=False,
        class_mode="categorical",
        target_size=(img_width, img_height)

       .
       .
       .

       i +=1


Are view() in Pytorch and reshape() in Numpy similar?

Yes, for most intents and purposes, they can do the same job. From this link, an example:

>>> import torch
>>> t = torch.ones((2, 3, 4))
>>> t.size()
torch.Size([2, 3, 4])
>>> t.view(-1, 12).size()
torch.Size([2, 12])

If you are concerned with memory allocation, here is another answer on StackOverflow with a little more information. PyTorch's view function actually does what the name suggests - returns a view to the data. The data is not altered in memory as far as I can see.

In numpy, the reshape function does not guarantee that a copy of the data is made or not. It will depend on the original shape of the array and the target shape. Have a look here for further information.


found \N in my data does not count as missing values in r

The reason its shorws an error, bacause '\' is a part of base regex expressions in R.
As states here:

The metacharacters in extended regular expressions are . \ | ( ) [ { ^ $ * + ?

So the comparisson like this

country$indepYear=='\N'

or this ( Its not a valid comparisson at all)

country$headOfState==\N

will throw an error. You need to "escape" the "\" symbol.
Try something like this instead if you want to replace "\N" with NA:

country$indepYear[country$indepYear=='\\N'] <- NA

If you need just to count them, you can use this approach:

sum(country$indepYear=='\\N')

Hope this helps.


How to extract keywords from a list of URLs?

urllib parse seems like the function for you. With this you are able to extract keywords from the net location and the path separately if you desire to process them separately or even if you want to join them back again later.

The result should look something like this:

from urllib.parse import urlparse

o = urlparse('https://www.forbes.com/sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts')

ParseResult(scheme='https', netloc='www.forbes.com:443', path='sites/forbesagencycouncil/2018/09/06/how-to-use-content-marketing-to-boost-your-recruiting-efforts',
           params='', query='', fragment='')

A second step will have to do with string parsing and string splits so something like

.split("/")
.split("-")

Will work just fine to split the path of the example's URLs into words. Also, remember to transform each word into lowercase.
After that stemming is a good idea so that all related terms are grouped into the same category. It's not necessary but it's a good idea.

Finally counting the occurrences of the words will give you the rank of the top "keywords" in those URLs.


Not sure how big the data you need but this link has data for binary classification. https://www.kaggle.com/tags/binary-classification


Why does downsampling leads classification to only predict one class?

For the looks of it the sampling method you used isnt doing the trick.
The way you use pandas slicing and assignment is a bit off.

Something like this might work:

dfs = []
for key in class_dict:

    if class_dict[key] > median:
        factor = int(count_df/7)
        key_df = df[df.grade == key].sample(factor, replace=False)
    else:
        key_df = df[df.grade == key]

    dfs.append(key_df)
new_train_df = pd.concat(dfs, sort=False)

While this might work, there are more straight forward ways down sampling the data(like setting the exact number of samples expected from each class).
This question has nice answers too.


When to split data into multiple regression models instead of one model?

You should split them by category since their features do not apply to each category.

Under certain circumstances that perhaps you manage to group some categories together based on some business logic, then perhaps you can build less models.


Find the most relevant columns for each single class in pandas

In the end, this is what I've found around the internet. Really helpful

def best_features(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    #apply SelectKBest class to extract top 10 best features
    bestfeatures = SelectKBest(score_func=chi2, k=15)
    fit = bestfeatures.fit(X,y)
    dfscores = pd.DataFrame(fit.scores_)
    dfcolumns = pd.DataFrame(X.columns)
    #concat two dataframes for better visualization
    featureScores = pd.concat([dfcolumns,dfscores],axis=1)
    featureScores.columns = ['Specs','Score']  #naming the dataframe columns
    return featureScores.nlargest(15,'Score')  #print 10 best features

def worst_features(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    #apply SelectKBest class to extract top 10 best features
    bestfeatures = SelectKBest(score_func=chi2, k='all')
    fit = bestfeatures.fit(X,y)
    dfscores = pd.DataFrame(fit.scores_)
    dfcolumns = pd.DataFrame(X.columns)
    #concat two dataframes for better visualization
    featureScores = pd.concat([dfcolumns,dfscores],axis=1)
    featureScores.columns = ['Specs','Score']  #naming the dataframe columns
    return featureScores.nsmallest(15,'Score')  #print 10 best features

def largest(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    model = ExtraTreesClassifier()
    model.fit(X,y)
    feat_importances = pd.Series(model.feature_importances_, index=X.columns)
    feat_importances.nlargest(15).plot(kind='barh')
    plt.show()
    print(feat_importances.nlargest(15).index)

def smallest(df,target):
    features = list(df.columns)
    features.remove(target)

    y = df[target]
    X=df[features]
    model = ExtraTreesClassifier()
    model.fit(X,y)
    feat_importances = pd.Series(model.feature_importances_, index=X.columns)
    feat_importances.nsmallest(15).plot(kind='barh')
    plt.show()
    return feat_importances.nsmallest(15).index.to_list()


What do you call a feature that always has the same value?

I think there are a few terms, but the one I have seen most often is "Zero Variance Predictor" or "Zero Variance Feature"


Classifying whether a comment or review is a complaint or appreciation of product and extracting the Topic?

Your (basic) task is sentiment analysis, covered in many places.
There is a number of algorithms proven good for that, including LSTM but you need a good deal of data to train that (and compute power). Others is fastText - tool by Facebook - where you have already embeddings for a number of languages including EN.
But you need anyway 'ground truth' - samples of positive and negative posts / reviews - to train final classification model. Assuming you have standard EN reviews - meaning a lot of language typical to online reviews... - you can use some of existing datasets to train your model and boost results. Otherwise you would have to manually select representative (meaning the more the better...) samples and label them (pos/neg). E.g. there is yelp review dataset, actually a number of versions, one of them.
The other tasks - extracting product and complaint/appreciation part - are  entity extraction and topic modelling respectively, each a seprate story.
For entity (product) extraction, possibly you can just search the post for the names from your inventory. Assuming again EN language, beware only of plural versions. And you will need a good data structure / indexes for it to get results fast.
For topic modelling, there is LDA implementation in gensim. You will have to manually go through the topics and process them, depending on the exact scenario you want to use them, e.g. assign each a meaningful name, selecting meanwhile those that make sense for you, merging some.


You have a problem of data leakage.
The "days since invitation was sent" feature contains all the information on the concept. Therefore, adding it as a feature will prevent most common classifiers from using rules based on other features and will lead to misleading results.

Duplicating the negative row for each predefined time value will lead to a different distribution than the one you should be tested one, which need needles problems.

Note that even that way, the feature might leak information (e.g., a small ratio of the positives will replay in two weeks while its distribution in the negatives will be higher, making it a good predicator for negative).

In case that you will have more complex representation of time (e.g., was the email sent on weekend? was the reply sent on weekend), the duplication of rows will be more complex.

I would try a different direction. It is likely that the negative rows has no influence on the time to reply in the positive ones.
Therefore I would do a first research aimed to differ between replies and no replies without using the predefined time.

After that, do a second research only on the positives where the concept will be the predefined time, aiming to find what is influencing out given that the user had replied.


Text Mining - Similarity among words to determine thresholds

Have you tried word2vec models ?. One of the main application is computing similar words, as they are very closer in the feature vector space. Thus apart from High_school and H_school, also secondary_school, public_school may also be obtained as closer words. The important thing is that a huge amount of sentences are needed for this unsupervised learning model.

Ref: gensim, word2vec in java


What is the relationship between clustering and association rule mining?

First of all, the problem of finding frequent itemsets (ARM) differs from the similarity search (clustering).

If you think about market baskets, clustering tries to find objects that have a large fraction of their baskets in common, the absolute number of those objects (basket occurences) is not of interest.
With ARM we are interested in the absolute number of baskets that contain a particular set of items.

CLustering:  Allocates objects in such a way that objects in the same group (called a cluster) are more similar (given a distance metric) to each other than to those in other groups (clusters).

ARM: Given many baskets (could be actual supermarket baskets) find which items inside a basket predict another item in the basket.

Sources


You can be sure that saving the model to disk won't make the model have higher accuracy when loaded again. Saving the models to disk would only cause worse performance if parameters are truncated, for example.

To establish a fair and rigorous comparison method, you have to keep track of your training/test splitting result. In that sense, you can:

Make your code reproducible, using random seeds conveniently
Before saving the model, you can save your splitting results by either saving the data or returning the indices for the training set and the test set. For that purpose you have to store the data-set in a list (for example, a list with the file names…)


Understanding XG Boost Training (Multi class classification)

What can I understand/interpret from the training & test loss graph?

This checking out the quality of the model. If the train and test set loss decreasing according to the number of epoch in the same way(i.e plots should overlap each other), that means that model is good. Otherwise, we have a problem. In your graph plots fastly separates(from x-value moreover 10 or 20). In this case we could say that's overfitting.

  I am not sure whether the model is overfitting ?

It's typical overfitting.

  How can I reduces the error b/w training & testing, only through hyperparam tunning or this is how the XG Boost model works ?

More data.
Feature selection.
Cross-validation.
Wide range hyperparameters to explore.
Random search.


The way I read this, you are actually asking two questions:

How do character n-grams help to encode knowledge that is often encoded with the help of techniques such as stemming ?
Why are n-grams language independent? I'm not totally sure on what you mean by this one, but I'll take a stab at it

Character N-grams

Some languages (most languages that I know of, but some more than others) have grammatic rules that change the morphology of a word: one house, two houses. In a vanilla Vector Space Model house and houses are not identical and form two dimensions of your model. As if they were as different as house and apple.

We know that English applies these morphological operations on the words and we can counter that by bending the words back into their 'stem'.

For instance, the Snowball stemmer would bend them back to a token that does collide:

  House  -> Hous
  Houses -> Hous

Note: Hous is not what we think of as a stem, hence my use of the quotes around 'stem'

N-gramming is basically splitting text into all subsequences sequences of length N. If we apply that (N=3, forgetting about start and stop for simplicity) on our example strings we would get something like:
>

  House -> Hou, ous, use
  Houses -> Hou, ous, use, ses

Note that we end up with 4 new dimensions, 3 of which collide. This reduces our previous found need for stemming. We could argue that it still doesn't collide fully, but then again, we might not want it to.

Language Independence

I don't think n-grams are language independent. Certainly not on the word level: some languages are more regular and some languages are more context-free. Some languages follow a pattern that looks more like adding words to the end as you go, and those languages are probably more suited for modelling with n-grams. And this probably holds too for character-based n-grams.

However, as previously argued, they add flexibility to the V.S.M. when it comes to handling morphology. That morphology would normally require 'manually' encoding knowledge about the language at hand. With this more or less out of the way your system needs less configuration to the language at hand, which makes it less dependent on the language.


Sci-kit Pipeline and GridsearchCV returns indexError: too many indices for array

It turns out that the Pandas dataframe is the wrong shape.

estimator.fit(train.values, train_labels[0].values)

works, although I also had to drop the penalty term.


Getting monthly revenue predictions for outlets

LTSTM models are very good at time-series data, so look into that for starters.

Additionally I'd like to ask why you need to adjust and deadjust your model at all?

Even simpler models like boosted trees, random forest or even a plain regression should give you good predictions by simply including trends, seasons, etc. as predictors.

Edit:

I try to be a bit more comprehensive in my answer. Your problem is actually a classical problem of sales prediction, especially for retail stores like supermarkets, etc.

A great resource to see solutions to this problem can be found here:

Kaggle Walmart Sales Forecast

You will find a lot of notebooks with great approaches and solutions (e.g. based on the aforementioned LTSTM model).

But let's also try to point out a basic solution right here:

*Data**
Let's assume we have data that looks basically like this:

Store  |  Month  |  Sales  |  Store_Size  |  ...
East   |  Jan    |  $0.9M  |  500         |  ...
East   |  Feb    |  $1.1M  |  500         |  ...
North  |  Jan    |  $1.5M  |  800         |  ...
North  |  Feb    |  $2.0M  |  800         |  ...

Some of the ... variables will be constant for each store (store info), some will be related to the date (seasonal) and some might vary for the store and the month (trends).

Problem

Trying to predict the sales of store West for March given all constant variables associate with that store but NOT the trend variables.

It is important to understand seasonal variables here! These are engineered features based on the month. E.g. we can easily compute the influence on sales for each month (e.g. trends like christmas sales, etc.) these are not trends because they are tied ONLY to months and not any other pattern.

Solution

Suppose we fit an XGB model to predict the sales, the simplest solution would be to simply only use features in the model that well be available at the point of prediction, so only seasonal and store info.

This might already deliver a very good model and might be enough for prediction. If you have no information on trends, you cannot add them to the model and therefore should not.

However you easily feature engineer trend data based on GAN or MICE models to augment your data.

Finally please understand: machine learning models work best if you simply throw all available data at them and do not get too cute with feature selection and engineering. XGB and similar models will identify trends by themselves and model the ideal relation between the other factors for you.

Degree of freedom, etc. really does not apply here.


You can create keras.json in the same way as on linux. The 'backend' key will have either 'tensorflow' or 'theano'. Place the file in %USERPROFILE%/.keras


In Weka, how to draw learning curve evaluated on both test and training set?

This is only possible with KnowledgeFlow. In WekaManual.pdf (which is included in Weka package) for version 3.7.12 there is an example in Chapter 7.4.2 "Plotting multiple ROC curves" with picture and step-by-step instructions. For other Weka versions it is the same, just find the appropriate chapter.

To give an impression on how it goes, I extracted the picture from the manual. It will draw two curves for two classifiers. For your question it is very similar. You use one classifier and then connect trainingSet to one ClassifierPerformanceEvaluator and testSet to another.


what to do if the missing data in one column is based on some value/condition in another column in r?

If you plan to use LightGBM or XGBoost, the advise is "Do not do anything". These methods treat NA in a specific way, different in each decision tree and the results obtained are usually much better than using imputing.


Meltdown patch impact on data processing speeds

It looks like the Tech Web Sites that reported Meltdown, may have overblown the negative impacts.

According to Google, in their cloud infrastructure, there seems to be little to worry about:

  There has been speculation that the deployment of KPTI causes significant performance slowdowns. Performance can vary, as the impact of the KPTI mitigations depends on the rate of system calls made by an application. On most of our workloads, including our cloud infrastructure, we see negligible impact on performance.

Techspot  ran some benchmarks on Windows 10, and they concluded...

  Desktop users have little to worry about in terms of performance loss, particularly gamers. We've yet to test older CPUs, but given the type of workloads we’re seeing impacted by the patch, I don’t think there’s going to be an issue with any desktop hardware, but we’ll certainly report back if there is.


How to calculate pedigree function in diabetes prediction?

I had much better results with the decision forest. The diabetes pedigree function requires good diabetes diagnostic history of the persons relatives. Based on this, it might be neglectable.


There are basic things you can do with any set of data:

Validate values (String length tolerance, data type, formatting masks, required field presence, etc.)
Range correctness (Does this seemingly correct data fall within expected ranges of values)
Preliminary processing (If I attempt to analyze this data, can I perform the basics without running into errors)
Preliminary reporting (run a report against a data set and ensure that it passes a sanity test)
Defining null vs. empty vs. zero vs. False for any given column of data
Identifying data that is out of place (numeric values dramatically different than other values in a data set, string values that look like they might be misspelled, etc.)
Eliminating or correcting obviously errant data

Understanding data to identify errors is a whole different ball game, and it is very important.

For instance, you can have a rule that says a serial number must be present in a given data set and that serial number must be alphanumeric with a maximum string length of 255 and a minimum string length of 5.

Looking at the data, you may find one particular serial number value reads "PLEASE ENTER SERIAL"  It's perfectly valid, but wrong.

That's kind of an obvious one, but say you're processing stock data and you had a price range for 1000 stocks that was under a dollar.  A lot of people would not know that a stock price so low is invalid on certain exchanges and perfectly valid on others.  You need knowledge about your data to understand if what you are seeing is problematic or not.

In the real world, you don't always have the luxury of understanding your data intimately.

The way I avoid problems is by leveraging the people around me.  For small data sets, I can ask someone to review the data in it's entirety.  For large ones, pulling a set of random samples and asking someone to do a sanity check on the data is more appropriate.

Further, questioning the source of the data and how well that data source can be trusted is imperative.  I often have multiple conflicting sources of data and we create rules to determine the "source of truth".  Sometimes one data set has great data in a given aspect, but other data sets are stronger in other areas.

Manually entered data is usually what I'm most skeptical about, but in some cases it is stronger than anything that can be acquired through automation.


How to handle categorical features in K-means?

Label encoding is not a good idea if the nature of categories are not ordinal (it is actually not my favorite anyways). Use one-hot encoding and see how it works. You may apply a feature extraction on top of it, e.g. PCA, to reduce the noise coming from sparsity. The other idea is to label categories by their fraction in the feature, for example:
[a,b,b,c,a,a] --> [3/6, 2/6, 2/6, 1/6, 3/6, 3/6]


API for Company Data Enrichment Suggestions

Check out Clearbit.com -- https://dashboard.clearbit.com/docs#company-api

Super comprehensive company lookup. Not free, but 50 free calls a month.


Does increasing the n_estimators parameter in decision trees always increase accuracy

There are a lot of misconceptions about regression random forest. Those misconceptions about regression rf are seen also in classification rf, but are less visible. The one I will present here is that regression random forests do not overfit. Well, this is not true.

Studying the statistical properties of the random forests shows that the bootstrapping procedure decreases the variance and maintain the bias. This property should be understood under the bias-variance tradeoff framework. It is clear that the random forests approximate an expectation, which means the mean of the true structure remains the same, while the variance is reduced. From this perspective, the random forests do not overfit. There is a problem, however, that problem is the sample itself which is used for training. The expectation is taken conditional on data. And if the data is not representative of the problem, it is normal that in the limit when the number of the tree grows to infinity. In plain English, this means that the regression forest will learn too well the data and if the data is not representative, then the results are bad. In which way the data might not be representative? In many ways, one would be that you do not have enough data points in all region of interests for example. This problem is seen often with testing error, so it might not affect you so much, but is possible to see that in CV also.

Another issue with regression trees is the number of significant variables and the number of nonsignificant variables in your data set. It is known that when you have few interesting input variables and a large number of noise variables the regression forests does not behave well. Boosting procedures does not have this behavior. There is a good reason for that. Regression forests produce more uninteresting trees which have the potential to move the learned structure away from the true underlying structure. For boosting this does not happen since at each iteration only the region of interests have large weight, so the already learned regions are affected less. The remedy would be to play with the number of variables selected on learning time.

There is a drawback however even if you increase the number of variables take into account at learning time. Consider two randomly grown trees. If you would have 100 input variables and select 10 of them for learning, there are small chances that the trees look similar. If instead you would select 50 variables for learning then your trees have better chances to look similar. This is translated into the fact that if you increase the number of variables for testing candidates at learning time, then the correlations between them increases. If the correlation increases, then they will not be able to learn a more complex structure, because of their correlations. If the number of variables selected is small you have the potential to learn more due to diversity, but if the significant variables are small compared to nonsignificant, this would lead to learn noise, too much noise. This affects most of the time the CV performance.


Nothing just the starting randomly initialized paragraph vector is concatnated/ averaged with the randomly initialized word vector. Everything else pretty much works the same.


How to extract important phrases (which may contain company name) from resume?

Sounds like you want named entity recognition.  There are a variety of approaches to NER, and plenty of implementations, like the Stanford NER package.

After you find named entities, determining what the named entity refers to is called concept normalization.


error when submitting machine learning project

The return type of pandas.DataFrame.to_csv is None.

So your first line assigns sub = None:

sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class}).to_csv('sub.csv')

This is probably what you wanted:

sub = pd.DataFrame({'instructions': test['instructions'],'compiler':new_pred_class})
sub.to_csv('sub.csv')
sub.head()


What's the best way to generate similar words?

But the alternative is to navigate a treebank, through "type of" relationships… much, much faster and cheaper.

WordNet provides exactly this: it is a lexical database in which words are grouped by synonyms, with several types of relations between groups in particular hypernyms/hyponyms (more general/more specific).
The database can be downloaded and there is a library to use it through nltk.


Benchmarks based on neural networks libraries to compare the performance between different GPUs

Here is my list with collection of benchmarks: https://github.com/mrgloom/Deep-Benchmark-List

https://github.com/dividiti/ck-caffe looks more promissing, but complicated.


If the problem you are trying to solve is "I need to guess if the new songs are fit to user's taste with the song feature data I got."

AND if you have some indication what user has liked / listened-to in past; you can use "Recommendation Engine" approach.

Example : https://tampub.uta.fi/bitstream/handle/10024/101198/GRADU-1495623946.pdf?sequence=1

It works on the basis of "likeness"; E.g.:

Assumption : All Hip-Hop songs have some common pattern among the
features like energy , loudness etc.
If a user likes Hip-Hop (can be judged from user's history based on features of songs), then other Hip-Hop songs can be suggested


Is Adam's optimization susceptible to Local Minima?

All stochastic gradient descent (SGD) optimizers, including Adam, have randomization built and have no guarantees of reaching a global minima. The randomization is a result of training on a sub-sample of the data at each step. There are no guarantees of reaching a global minima because gradient descent optimizers are first-order, iterative optimization techniques.


One Hot Encoding for any kind of dataset

I would recommend to use the one hot encoding package from category encoders and select the columns you want to using pandas select dtypes.
import numpy as np
import pandas as pd
from category_encoders.one_hot import OneHotEncoder

pd.options.display.float_format = '{:.2f}'.format # to make legible

# make some data
df = pd.DataFrame({'a': ['aa','bb','cc']*2,
                   'b': [True, False] * 3,
                   'c': [1.0, 2.0] * 3})

cols_encoding = df.select_dtypes(include='object').columns
ohe = OneHotEncoder(cols=cols_encoding)
encoded = ohe.fit_transform(df)

Note that you can change the way you handle unseen data with

handle_unknown: str

options are ‘error’, ‘return_nan’, ‘value’, and ‘indicator’. The default is ‘value’. Warning: if indicator is used, an extra column will be added in if the transform matrix has unknown categories. This can cause unexpected changes in dimension in some cases.


Feature importance for random forest classification of a sample

So let me clarify your query: you have trained a random forest model to classify a dataset into multiple classes, e.g. A, B, C, or D. Now, you want to understand for each different class label (e.g. A), which features contributed to it being classified as class A vs. not class A, and so on.

To find the importance of a variable in random forest, each variable is permuted among all trees and the difference in out of sample error of before and after permutation is calculated. The variables with highest difference are considered most important, and ones with lower values are less important.
this method gives importance for the entire multinomial classification, i.e. A vs. B/C/D, B vs. A/C/D, etc. and not just for one label. So you can't claim that feature1 is more important to identify class A, but feature 2 is more important to identify Class B.

I've been using the random forest package implemented in R and it does not seem to provide any way to decipher the internal details on the one-vs-all categorization.

If you're really interested in doing just that, then you should fit a set of random forest models each for classifying class A vs. the rest, class B vs. the rest, etc. The variable importance of these separate models would give you the feature importance for classifying each label.

You can use the same library to fit these models using scikit-learn.  You may refer to their help page here - http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html


Which Spark MLlib regression algorithm is suitable for numeric predictions based on non-numeric features?

I would suggest -ensembles of trees (Random Forests and Gradient-Boosted Trees).

Here is a nice reference for handling such data with decision trees.


Input and output layer activation functions of neurons in Orange

The activation functions are used in hidden layers and output layer. The output layer will usually have sigmoid or softmax activation while hidden layers usually use ReLU. ReLU is available in Orange at least (there is a dropdown which you can explore but ReLU works best mostly, read this). Also, check this link for looking at neural networks in Orange.


If I train a model using only my testing data, will the accuracy be 100%?

It will likely be not 100% even though it is possible. But, for example, a linear model might not be able to represent a non-linear relationship in your data. Or your features don't contain sufficient information to separate all classes.

Your other question doesn't seem to make sense. If you train your model on the test data, the test data wouldn't really be test data anymore?!

There are tons of resources on bias-variance trade-off and overfitting.


Why I am getting prediction score 1 i.e. 100%

You are using your target variable location_id as a feature. You need to remove it from data/train1/ X variables.

In other words, you are trying to predict location_id by location_id.

If you use reg.feature_importances_ you will see, that location_id affects your prediction 100% and others have no effect on prediction result.


Linear Regression to predict a growing variable with time

When you have an exponential shape in $x$, you can transform $x$ by taking logs.

So while a model $y=\beta x + u$ will likely perform badly, a model such as $y=\beta \log(x)+u$ will likely perform better.

However, based on your minimal description of the problem, you cannot expect a really detailed answer.


Any advantage of sklearn wrappers for xgboost over python API?

One advantage of using scikit-learn wrappers is that they are more likely to work within the scikit-learn ecosystem. For example, it might be easier to put the algorithm into pipelines.


Represent time-series data in much compact form

Simulate some data:

library(ggplot2)
library(purrr)
library(ggthemes)

days <- seq(as.Date("2015-08-01"), as.Date("2015-08-31"), by="1 day")
hours <- sprintf("%02d", 0:23)

map_df(days, function(x) {
  map_df(hours, function(y) {
    data.frame(day=x, hour=y, val=sample(2500, 1), stringsAsFactors=FALSE)
  })
}) -> df

Check it:

ggplot(df, aes(x=hour, y=val, group=day)) +
  geom_line() +
  facet_wrap(~day) +
  theme_tufte(base_family="Helvetica") +
  labs(x=NULL, y=NULL)

Since you're only trying to convey the scope of the variation, perhaps use a boxplot of the values of hours across days?

ggplot(df, aes(x=hour, y=val)) +
  geom_boxplot(fill="#2b2b2b", alpha=0.25, width=0.75, size=0.25) +
  scale_x_discrete(expand=c(0,0)) +
  scale_y_continuous(expand=c(0,0)) +
  coord_flip() +
  theme_tufte(base_family="Helvetica") +
  theme(axis.ticks=element_blank()) +
  labs(x=NULL, y=NULL)

That can be tweaked to fit into most publication graphics slots and the boxplot shows just how varied each day's readings are.

You could also use boxplot.stats to get the summary data and plot it on a line chart:

library(dplyr)
library(tidyr)

bps <- function(x) {
  cnf <- boxplot.stats(x)$conf
  data.frame(as.list(set_names(cnf, c("lwr", "upr"))), mean=mean(x))
}

group_by(df, hour) %>%
  do(bps(.$val)) %>%
  ggplot(aes(x=hour, y=mean, ymin=lwr, ymax=upr, group=1)) +
  geom_ribbon(fill="#2b2b2b", alpha=0.25) +
  geom_line(size=0.25) +
  theme_tufte(base_family="Helvetica") +
  theme(axis.ticks=element_blank()) +
  labs(x=NULL, y=NULL)


This sounds like a classic frequent pattern mining problem, where you're trying to find sets of items that are frequently found together within users' sets of purchases.

Start here: https://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html


Digits Localization on Streets View House Numbersm

Since you are saying that you got box coordinates in the original image, why not reduce coordinates by exactly same scale.

For example a coordinate say $(100,100)$ in the image of say $(1024,1024)$ size when resized to say $(256,256)$ will be $(25,25)$ (assuming you didn't crop anything in the mean while), which is in general

$$(x_{new},y_{new}) = (\frac{x_{old}*l_{new}}{l_{old}},\frac{y_{old}*b_{new}}{b_{old}})$$ where $l$ is the length of the image, $b$ is the width.

In times of fractional answers after reduction make sure your bounding box covers a bigger area. For example, Y-coordinate for the left side of bounding box should be pushed towards the Y-axis while on the right should be pushed towards the ceil value like say you got $25.5$ by reducing, go to 25 when on the left side while $26$ when on right side of bounding box. Similarly up and down for $x's$.

Hope this helps.


I think you might be a little confused with the terminology here.

Whether you train locally on your own computer or in the cloud (via AWS, Microsoft Azure, GoogleCloud, FloydHub, ...) there are only two three options:

using a CPU $\rightarrow$ whatever is in your computer!
using a GPU $\rightarrow$ probably an Nvidia card supportung CUDA
using a TPU $\rightarrow$ a type of GPU to accelerate Machine Learning)

Your deep learning framework of choice (Pytorch, Tensorflow, CNTK, ...) will make use of the hardware with which you provide it.

All cloud services will offer instances with or without GPUs (they will almost vertainly have a CPU).

Training on a GPU is of course much faster thatn a CPU, as it can parallelise many of the operations that are performed (especially for CNNs, compared to e.g. LSTMs). It is of course possible to offload certain parts on training onto each of them, so you would be using both together.

A GPU in your computer at home and GPU in a big server-rack in an AWS data-center would both perform as equally as can be hoped for. The only differences will be cost, but that is a different discussion.


Why is my MLP with 2 features is doing worse than MLP with 1 feature where the one feature is a combination of feature1*feature2?

$h$ - height
$w$ - width
$p$ - weight

In real world we know that weight is volume times density. And volume is product of height, width and depth. If you assume constant depth and density the weight is just some constant multiplied by width and height i.e. $p = c \times hw$. If you have $hw$ as a feature the NN quickly learns $c$. This gives good performance.

Lets say your NN had identity as activation function i.e. no activation function. All it can learn is linear combination of input features.

So it will just learn $p \approx h w_h + w w_w + b$ where $w_h, w_w$ are weights and $b$ is bias.

If you have activation function like ReLU/Sigmoid it still learn linear combinations of outputs of ReLU/Sigmoid. Inputs to those is just linear combination of $h$ and $w$. In all that there is never a simple $hw$ term. This makes model complicated without having the important feature that is volume(area).

I think that is why the performance of $hw$ model is better.

If you have a NN with 2 neurons in the hidden layer with sigmoid activation, and 1 output neuron with no activation, then this is how $hw$ term shows up.

$\frac{1}{1+\exp(hw_{11} +w w_{12} + b_1)} + \frac{1}{1+exp(hw_{21} +w w_{22} + b_2)} = \frac{\ldots}{\ldots + \exp(w_{11} w_{22} hw + w_{12}w_{21}hw) } $

Thats very tricky place to show up if you just want to learn a constant multiplier.


Regarding 2) Your task sounds like a classical case for survival analysis. From the description of your problem, I cannot really say if this would be an option, but I just want to let you know about it: https://en.m.wikipedia.org/wiki/Survival_analysis


How many examples needed for named entity disambiguation?

It depends on the signal-to-noise in the dataset. The amount of data to perform named entity disambiguation will depend on the tf-idf score of the occupation and skills, rare occupations and skills will require less data to build a performant model.

For example, that the sentence "I am a cook that multitasks well." "Cook" is an occupation and "multitask" is a relevant skill. In a similar sentence, "I multitasked while I cooked." "Cook" is no longer an occupation and "multitask" is no longer a relevant skill. However, the phrase "saturation diver" is less frequent than "cook", thus much easier to build a model to identify as an occupation and find relevant skills.

Annotator performance is easier to measure. Cohen's kappa is a common method of judging inter-rater reliability. Again, the number of needed raters depends on their agreement on the task. If task performance is easy, the number of raters and the number of items per rater can be lower. It is best to benchmark your system and then decide how much data you need to raise the benchmark scores.

One way to automatically create ontologies from a text is the TextRank algorithm.


How to find the mean of a column relative to another column?

You need to change inplace to False (which is the default).

Setting it to True changes the dataframe in place, so you don't have to assign the column again. Also, setting inplace=True returns None. So, either:

train['GarageYrBlt'] = train['GarageYrBlt']
     .fillna(train.groupby('YearBuilt')['GarageYrBlt']
     .transform("median"), inplace=False)

or

train['GarageYrBlt']
     .fillna(train.groupby('YearBuilt')['GarageYrBlt']
     .transform("median"), inplace=True)


You are using R, and everything that you are currently working on will be held in memory hence the error. You just don't have enough of it.

You might be better off creating the frequencies of the terms from the original splits as opposed to creating one big file.

Then after you have the frequencies adding them all together.

Personally I use this code to create my wordclouds.

##Clean code for single column of a dataframe, in this case named Text
  x = alltweets
  tweets.text <- x$text
  tweets.text.cleaned <- gsub("@\\w+ *#", "", tweets.text)
  tweets.text.cleaned <- gsub("(f|ht)tp(s?)://(.*)[.][a-z]+", "", tweets.text.cleaned)
  tweets.text.cleaned <- gsub("[^0-9A-Za-z///' ]", "", tweets.text.cleaned)
  tweets.text.corpus <- Corpus(VectorSource(tweets.text.cleaned))
  tweets.text.final <- tm_map (tweets.text.corpus, removePunctuation, mc.cores=1)
  tweets.text.final2 <- tm_map (tweets.text.final, content_transformer(tolower), mc.cores=1)
  tweets.text.final2 <- tm_map(tweets.text.final2, removeNumbers, mc.cores=1)
  tweets.text.final2 <- tm_map(tweets.text.final2, removePunctuation, mc.cores=1)
  tweets.text.final2 <- tm_map(tweets.text.final2,removeWords, stopwords("English"), mc.cores=1)
  tweets.text.final2 = tm_map(tweets.text.final2, removeWords, c("amp", "&"))
#create corpus
   housing.tweets.corpus <- Corpus(VectorSource(tweets.text))

#clean up by removing stop words
   housing.tweets.corpus <- tm_map(housing.tweets.corpus, function(x)removeWords(x,stopwords()))

#install wordcloud if not already installed
   install.packages("wordcloud")
   library("wordcloud")

#generate wordcloud
   wordcloud(housing.tweets.corpus,min.freq = 2, scale=c(7,0.5),colors=brewer.pal(8, "Dark2"),  random.color= TRUE, random.order = FALSE, max.words = 500)


Why do I not get 100% Accuracy with KNN with $K=1$

In your training set (X_2,y), there are some samples with the same input features X_2 but different labels y. For example, the 73rd and 147th samples, which are labelled into class 0 and 1, respectively, have the same input values [6.3, 2.5]. There are more samples like this in the dataset. Therefore, you could not construct a perfect classifier for such data.


Can I use xgboost on a dataset with 1000 rows for classification problem?

Yes, XGBoost is famous for having been demonstrated to attain very good results using small datasets often with less than 1000 instances.

Of course when choosing a machine learning model to fit your data, the number of instances is important and is related to the number of model parameters you will need to fit. The greater the number of parameters in the model the more data you will need to reduce the bias of your final model. If you do get good results using a complex model on very few instances then there is a high probability that you are overfitting. For example 1000 instances is hardly enough to fit a deep neural network.

That being said, the distribution of your classes and the noise in the data is always going to be a limiting factor to how well any model you select will fit your data.


Imbalanced Dataset (Transformers): How to Decide on Class Weights?

I am not sure about the model you are using but I might explain what the procedure is for ML in general. You have three "vanilla" solutions for coping with unbalanced supervised dataset.

Reweighing class label so that there is the same number (calculated as sum of weights for given label) of samples per label. For example if a label with maximum number of samples has $n_{max}$ samples, and some other class has $n_i$ samples, then you would assign a weight $w_i=\frac{n_{max}}{n_i}$.
Undersampling - a basic procedure that gets rids of all the additional samples so that you end up with a balanced dataset.
Oversampling - creating copies of the unbalanced classes (with less samples than the $n_{max}$)
Hope that helps,

Max


Parameter Adjustment based only on tagged predictions

This is more a comment than an answer but the adjustment part is likely to depend on the model you have.
So it will be difficult to give you a general method. If your model is a simple rule base one, the adjustment could consist in adding another rule, if it is a neural network then you'll go with backpropagation to adjust the weights of the nodes, if it's a Bayesian tree then you could grow another tree taking into account your new outliers, if it's Bob, your dear colleague, you'll have to make sure that he takes the new info into account next time he classifies something as being an outlier.


I think the intuition here is that you want to push the negative loglikelihood (NLL) * the reward to be as negative as possible. Since our reward is often not differentiable because it is obtained through sampling, we can only change the NLL. For actions with high reward, we have greater "pressure" / gradient to push the NLL down, than if the reward was low, as the reward acts as a multiplier on the gradient on the NLL. So you'd imagine at convergence/equilibrium, the high reward actions would have greater likelihood than the low reward ones. Let me know if that makes sense!


Pandas time series optimization problem: add year

Make a pandas Timedelta object then add with the += operator:

x = pandas.Timedelta(days=365)
mydataframe.timestampcolumn += x

So the key is to store your time series as timestamps.  To do that, use the pandas to_datetime function:

mydataframe['timestampcolumn'] = pandas.to_datetime(x['epoch'], unit='s')

assuming you have your timestamps as epoch seconds in the dataframe x.  That's not a requirement of course; see the to_datetime documentation for converting other formats.


Pandas copy() different columns from different dataframes to a new dataframe

You are overwriting df3.
Method-1:
   import pandas as pd

    df1 = pd.read_csv('ASX_public_DB.csv', sep=';', skiprows=5)
    df2 = pd.read_excel('ASX_delisted_6months.xlsx')

    df3 = df1[['ticker_symbol']].copy()
    df3['ASX code'] = df2[['ASX code']].copy()

    print(df3)

Method-2:
    import pandas as pd

    df1 = pd.read_csv('ASX_public_DB.csv', sep=';', skiprows=5)
    df2 = pd.read_excel('ASX_delisted_6months.xlsx')

    df3 = pd.DataFrame()
    df3['ticker_symbol'] = df1['ticker_symbol'].copy()
    df3['ASX code'] = df2['ASX code'].copy()

    print(df3)
```


Matching content item to a persons profile

Look into collaborative filtering and content based filtering.
Essentially, build a user-space - using their personality, etc features. build an item space in a similar manner- using their tags, what not.
Decide on a metric for each space, so you can compare how much each item is similar to each other(same for users).
Then look at each user preferences and say, this user is similar to that user who liked this item who is close-like to that item.
If user A liked item 1 and user B is similar to user A then its probable he will like item 1 as well.

This is a simple enough example


make sure the files are saved in UTF
try Sys.setlocale("LC_ALL", locale_code) and have a look at the documentation of this function


How is the LSTM RNN forget gate calculated?

The equation and value of $f_t$ by itself does not fully explain the gate. You need to look at first term of the next step:

$C_t = f_t \odot C_{t-1} + i_t \odot \bar{C}_{t}$

The vector $f_t$ that is the output from the forget gate, is used as element-wise multiply against the previous cell state $C_{t-1}$. It is this stage where individual elements of $C$ are "remembered" or "forgotten".

Due to the sigmoid function, the vector $f_t$ behaves like a binary classifier for each element, with saturated values tending to settle on either not modifying $C$ at all (a value of $1$) or "forgetting" what the previous value was (a value of $0$). Of course intermediate values are also possible, and the analogy there between simply remembering and forgetting values is less direct.

The analogy with forgetting or remembering helps towards understanding the improvements regarding preserving gradients over multiple timesteps. For any step where an element of $f_t$ is close to $1$ (and thus the effect of previous timesteps is being "remembered"), then the corresponding gradient elements backpropagating from $\nabla_{\theta} C_{t}$ to $\nabla_{\theta} C_{t-1}$ remain the same, avoiding the gradient loss seen in more simple RNN architectures (especially related to saturated values). Once a previous element in $C$ is "forgotten", then the error gradient connection is cut off between time steps.


How to train an existing word2vec gensim model on new words?

Add this line before training with the new terms.

model.build_vocab([['potoatoes', 'and', 'farmers']], update=True)

After training,

print(len(model.wv.vocab))
=> 8


sklearn serialize label encoder for multiple categorical columns

LabelEncoder is meant for the labels (target, dependent variable), not for the features.  OrdinalEncoder can be used for features, and so can take a 2d array rather than the 1d array LabelEncoder requires, and so you can use a single transformer for all your categorical columns.  (You can use a ColumnTransformer to select those categorical columns, if you have continuous ones too.)

But, you should not use OrdinalEncoder before OneHotEncoder anymore:  OneHotEncoder has for some time now applied directly to string categorical columns.


Training a Variational Autoencoder (VAE) for Random Number Generation

You are describing surrogate modelling.
Because your situation is well studied, I recommend looking at what others have published. See, this paper for example.


Restricting the output of a model didn't improve the loss value of the model evaluation

I suggest you normalise your labels so that it is scaled between 0 and 1, rather than 0 and 80. Once you have a trained model then multiply your output at the end. The network should find it easier to learn values between 0 and 1 (Andrew Ng's coursera course has a good lecture on this).

Go back to using the standard ReLU: The problem with yours is that it cannot learn if it tries to output a value greater than 80, as there is no gradient with which to update the parameters.

I would also consider putting in a much smaller Dense layer at the end (eg 10 nodes).


On gradient boosting and types of encodings

This is actually a feature of tree-based models in general, not just gradient boosting trees.
Not exactly a reference, but this Medium article explains why ordinal encoding is often more efficient.
On the topic of safety, I think the author should have said that the use of ordinal encoding is more safe compared to linear methods, but still not perfectly safe.  It's possible for decision-tree methods to find spurious rules within ordinal encodings, but they don't have the strong assumptions about numeric semantics that linear methods do.

. . . I would like to know more about their supposed equivalence to encode unordered categorical variables . . .

Any rule derived with one-hot encoding can also be represented with ordinal encoding, it just might take more splits.
To illustrate, suppose you have a categorical variable foo with possible values spam, ham, eggs.  A one-hot encoding would create 3 dummy variables, is_spam, is_ham, is_eggs.  Let's say an arbitrary ordinal encoding assigns spam = 1, ham = 2, and eggs = 3.
Suppose the OHE decision tree splits on is_eggs = 1.  This can be represented in the ordinal decision tree by the split foo > 2.  Suppose the OHE tree splits on is_ham = 1.  The ordinal tree will require two splits: foo > 1 then foo < 3


How to find the initial hyperplane in a Support Vector Machine (SVM)?

Not sure how this is done in the any specific library, but here is what I would try.
Given two sets of points ($K$ points in the first set and $S$ in the second set): $\{\mathbf{a}_i\}_{i=1\dots K}$ and $\{\mathbf{b}_i\}_{i=1\dots S}$, all in $N$-dimensional space ($\mathbb{R}^N$)
I would compute the 'centres of mass' of the two sets:
$$
\begin{align}
\mathbf{A}=&\frac{1}{K}\sum_{i=1}^K \mathbf{a}_i \\
\mathbf{B}=&\frac{1}{S}\sum_{i=1}^S \mathbf{b}_i
\end{align}
$$
I would then use the mid-point between the two centres of mass,
$$\mathbf{M}=\left(\mathbf{A}+\mathbf{B}\right)/2$$
as the point for the hyper-plane.
Then I would use the vector connecting the two centres of mass,
$$\mathbf{C}=\mathbf{A}-\mathbf{B}$$
as the normal for the hyper-plane. Lets define
$$
\mathbf{\hat{n}}=\frac{\mathbf{C}}{\sqrt{\mathbf{C}.\mathbf{C}}}
$$
A single point and a normal vector, in $N$-dimensional space, will uniquely define an $N-1$ dimensional hyper-plane. To actually do it you will need to find a set of vectors
$$
\{\mathbf{v}_j\}_{j=1\dots N-1},\quad \mathbf{v}_j.\mathbf{\hat{n}}=0\,\mbox{for all } j
$$
This set can be created by Gram-Schmidt type process, starting from your trivial basis and then ensuring that every new vector is orthogonal to all vectors in the set and to $\mathbf{\hat{n}}$.
Once you did that, any point on the hyper-plane will be uniquely described by $N-1$ coordinates $\alpha_{i=1\dots N-1}$, and will correspond to the following point in the original $N$-dimensional space
$$
\mathbf{q}=\sum_{j=1}^{N-1} \alpha_j\mathbf{v}_j+\mathbf{M}
$$


How do I deal with additional input information other than images in a convolutional neural network?

I think there are three questions here:
How to incorporate non-spatial information into the network?
When combining different information modalities, a typical approach is to do it at the internal representation level, that is: the point where you lose the spatial information (normally with a flatten operation) after the convolutions. You can have your extra information be processed by an MLP and the result be combined with  the representations obtained by the convolutional layers by concatenating both.
How to represent the cards as input to the network?
In order to represent the card a user has, you can represent them as discrete elements (i.e. tokens), just like text is usually handled in neural networks. This way, you can use an embedding layer, which would receive as input the index of the card. As the user can have any number of cards, you could use an LSTM. In order to represent the "end of the card collection", you can have a special token, and yet another one to represent "padding", which would be useful to create minibatches with different number of cards.
How to represent the places?
You should decide if these are better represented as discrete or continuous values. Or maybe just try both options and choose the best performing one. For continuous features, you could add, as you suggested, another feature place. For discrete features, you would just have an embedding layer, and then concatenate the output to the other channels.
Update: Some clarifications:

A "token" is a term used in NLP to refer to a value which is discrete, that is, the number of values it can take is finite, normally small. In your case, the different values that a card token can take is 19. Usually, we refer to tokens by the index they occupy in the list of all possible values.

In order to represent discrete values in neural networks, we normally represent each different value as a fixed-size vector.

An embedding table is just a table with the fixed-size vectors used to represent your discrete elements. The embedding layer is normally the first in the network architecture. It receives as inputs token indexes and outputs their associated vectors. The entries of the embedding table are updated during the backpropagation process.

You don't "concatenate an MLP with a convolutional network", you concatenate their outputs. Specifically, once the output of the last convolutional layer is computed, you normally "flatten" it, meaning you remove the spatial information and just place the output tensor elements in a single-dimension vector. That vector is what you concatenate with the output of the MLP, which is also a single-dimension vector (apart from the minibatch dimension).


One reason Gaussian distributions are assumed is that it simplifies modeling since a Gaussian distribution only has 2 parameters to estimate (i.e., mu and sigma) and the properties of the distribution are well understood.
Gaussian distributions are often a reasonable assumption for acoustic data since acoustic data can modeled as a sum of independent random variables. According to the central limit theorem, the normalized sum of independent random variables  tends toward a Gaussian distribution.


Treating continuous data as a classification problem by predicting bins or quintiles

One option is scikit-learn's KBinsDiscretizer which bins continuous data into intervals.


You have got yourself a time series forecasting problem. And with multiple input variables it is called multivariate time series forecasting.

What Is Time Series Forecasting?

You can start with EDA on your data and find out if you can see any trend or seasonality. ( You might need to add or update your current features to get underlying trend/seasonality )

After EDA, you can start looking into following models, all of them are the go-to for time series prediction problem:

Classical, Statistical

ARMA for stationary data
ARIMA for data with a trend - Refer
SARIMA for data with seasonality
Holt-Winters Forecasting - Refer
Theta method - Refer
Fourier Transformation - Refer

Machine Learning

Quantile Regression Forest(QRF)
Support Vector Regression(SVR)
Recurrent Neural Networks(RNNs) (LSTM)

If you are not comfortable with Statistics then I would advise you to start with LSTMs for forecasting - Refer


How to convert night image to day image?

One problem with this is that dark images simply contain less information. Anyone with a background in photography will tell you it’s easier to decrease exposure on a bright image than increase exposure on a dark one (you can’t create what’s not there but you can throw away information you have). If you want all images on the same playing field, perhaps do the reverse. Or, try to extract only edges or high contrast regions from all images. Or, model them differently.


Regression model for continuous dependent variable and count independent variables

As you would like to retain all the predictors, you should try implementing ridge regression, which is a regularization technique popularly used for multi-collinearity problems like yours, by means of coefficient shrinkage.


is this a classification or clustering problem?

To me, what you described, sounds very much like collaborative filtering. In collaborative filtering, the features of the user(usually individual preferences) and features of the item(e.g. type and attribute of hotels) are trained collaboratively, gradually improving each other through iterations and converge to an optimal. Take a look at Andrew Ng's lecture, you can get a very quick idea of how it works.

In my opinion, collaborative filtering is a relatively basic technique for providing recommendations. Decision tree based models are more generally used in real life recommendations by big companies.


why we need data labelling tool for computer vision?

You need to label data in order to train classifier. Labelled data are at the basis of all supervised ML. XML files are just one of the many format you can use to store information, people usually use them since they can be easily read by many tools.


Support Vector classifier perform well with input features rather than transformed features in contrast to ANN-BP, random forest (other classifiers)

Thanks for the answer and it completely makes sense that results will be driven by data set in question!!

Is there any theory that says that SVM gives better results as compared to other ML algos like NN and RF when features are lower in number?
Another reason that I can think the reason for this behaviour is attributed to the information loss during transformations which were done.

Though SVMs results almost remained the same but for other algorithms improvement was significant(8-10%). Means transformed data showed patterns that RF and NN were able to recognize and do the classification while it hardly affected SVM outputs.


neural network for binary classification of xor gate

There are a few mistakes in the code, so I am going to present a revised version here with comments.

Setup

import numpy as np
import pandas as pd
x=np.array([[0,0],[0,1],[1,0],[1,1]])
y=np.array([[0],[1],[1],[0]])
np.random.seed(0)

# Optional, but a good idea to have +ve and -ve weights
theta1=np.random.rand(2,8)-0.5
theta2=np.random.rand(8,1)-0.5

# Necessary - the bias terms should have same number of dimensions
# as the layer. For some reason you had one bias vector per example.
# (You could still use np.random.rand(8) and np.random.rand(1))
b1=np.zeros(8)
b2=np.zeros(1)

alpha=0.01
# Regularisation not necessary for XOR, because you have a complete training set.
# You could have lamda=0.0, but I have left a value here just to show it works.
lamda=0.001

Training - Forward propagation

# More iterations than you might think! This is because we have
# so little training data, we need to repeat it a lot.
for i in range(1,40000):
    z1=x.dot(theta1)+b1
    h1=1/(1+np.exp(-z1))
    z2=h1.dot(theta2)+b2
    h2=1/(1+np.exp(-z2))

Training - Back propagation

    # This dz term assumes binary cross-entropy loss
    dz2 = h2-y
    # You could also have stuck with squared error loss, the extra h2 terms
    # are the derivative of the sigmoid transfer function.
    # It converges slower though:
    # dz2 = (h2-y) * h2 * (1-h2)

    # This is just the same as you had before, but with less temp variables
    dw2 = np.dot(h1.T, dz2)
    db2 = np.sum(dz2, axis=0)

    # The derivative of sigmoid is h1 * (1-h1), NOT dh1*(1-dh1)
    dz1 = np.dot(dz2, theta2.T) * h1 * (1-h1)
    dw1 = np.dot(x.T, dz1)
    db1 = np.sum(dz1, axis=0)

    # The L2 regularisation terms ADD to the gradients of the weights
    dw2 += lamda * theta2
    dw1 += lamda * theta1

    theta1 += -alpha * dw1
    theta2 += -alpha * dw2

    b1 += -alpha * db1
    b2 += -alpha * db2

Prediction

This is where you can kick yourself, you forgot to use the biases!

input1=np.array([[0,0],[1,1],[0,1],[1,0]])
z1=np.dot(input1,theta1)+b1
h1=1/(1+np.exp(-z1))
z2=np.dot(h1,theta2)+b2
h2=1/(1+np.exp(-z2))

print(h2)

When I run the above code I get a correct-looking output

[[ 0.01031446]
 [ 0.0201576 ]
 [ 0.9824826 ]
 [ 0.98584079]]

In summary your three big errors were the wrong dimension for the bias vectors in setup, incorrect derivatives for sigmoid function (using correct form, but with wrong variable) and forgetting to use bias at all when predicting at the end. Other details are still worth noting, but would not have prevented you getting something working.


Estimating data set size for grammar extraction

You can never get 100% coverage for real-world grammar extraction. Grammar is complex and undefined for real-world data. In addition, it is an "open world" data problem because novel grammar phrases can always be created.

You might get 100% coverage for a small data set through overfitting.

Statistical learning theory provides a framework for predicting the limits of machine learning.


Multi-task learning for Multi-label classification?

MTL is supposed to enable your shared layer to generalise better.For eg. in a text classification problem your encoder would be trained on separate tasks:
one could be multi class classification(original task) itself and let's take  identifying the sentiment of the sentence(again a classification task; but different than the original task) as the other task.

For each of these tasks encoder will be same but heads(decoders) will be different.

Decoder/head of one task won't be affected by the other task.

MTL always makes sense. Your encoder/shared layers will be able to generalise better putting less load on your decoders which in turn will increase the performance.


How are weights calculated in a feed-forward neural network before they are summed up with bias?

Initial weights are simply initialized by the programmer, usually according to some random distribution (e.g. Gaussian).
There are several well known initializers, such as xavier,
that have been shown to improve the training process.

If you read ML papers, you will see that researchers always mention how they initialized weights for their models so that their results are reproducible.


How does the naive Bayes classifier handle missing data in testing?

In evaluation (test) phase, when data point $x_n$ has $d$ missing features at indices $M=\{m_1,...,m_d\}$, corresponding terms $P(x_i|C_k),i \in M$ are simply removed from the classifier. That is, classifier
$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}P(C_k)\prod_{i}P(x_i=x_{n,i}|C_k)$$
is replaced with
$$C(x_n) = \underset{k \in \{1,..,K\}}{\mbox{argmax }}P(C_k)\prod_{\color{blue}{i:i \notin  M}}P(x_i=x_{n,i}|C_k)$$

where $i$ iterates over features, $x_{n,i}$ denotes the $i$-th feature of data point $n$, and there is $K$ classes in total.


Giving higher priority to certain inputs in SKLearn Random Forest

You've kind of answered your own question there!

One of the strengths of machine learning with algorithms like Random Forests is that they are able to determine the feature (input) importance themselves. My advice would not be to weight them for now (not sure you actually can with SKLearn's RF) and let the model determine it itself.

What you can do is give additional weight to the rows (observations) of your data set, or give additional weight to different classes of the label (for example, if there aren't many 'P's in your label but they're important, you can bump up their weight relative to the other classes).


This can be a cross between machine learning and simple matching exercise.

I think X_i tend to be rather defined and finite, while A_i can be vague and not finite. From a pure algorithm perspective I would search for instances where X_i = A_i and store the results into a container of sort. The more hits for certain X'es where X_i_n = A_i_k the more points X scores. X'es are then presented to A in the order of points from best match to lowest match.

Onto the machine learning mechanism, as the algorithm serves a lot of As (by that mean thousands and thousands, even millions) patterns will start to develop and certain combination of A_i's will be more prevalent, or in other words, worth more to other A_i's for a certain category of A. Using these patterns, the weighting of points will be re-balanced for higher chance of hitting the correct offers.

Kind of like how a search engine works.


If this distribution is row specific (each sample has a different associated distribution) or category specific this is not a bad approach to encode more information in your features. It's unclear how you have these distributions, are there empirical samples or do you have a parameterized distribution? A few approaches you could take to encoding this could be:

Fit a distribution family (or if you have one already) and use the location/shape/scale parameters as features
Add a few moments and other statistics of the distribution
Or similarly take a few percentiles

Depending on the parameterization of the distribution, the first might be less direct but it should be easy to test.


How can I extract bootstrap generated datasets into individual dataframes?

One way is to use the infer package.

library(tidyverse)
library(infer)

df <- read_table(pipe("pbpaste"))

five_bootstraps <- df %>%
  generate(reps = 5, type = "bootstrap")

five_bootstraps

# # A tibble: 60 x 5
# # Groups:   replicate [5]
#     replicate  Time Animal_ID Treatment Volume
#         <int> <int>     <int> <chr>      <int>
#   1         1     4         3 T2             8
#   2         1     2         1 Control        2
#   3         1     0         3 T2            21
#   4         1     0         3 T2            21
#   5         1     2         2 T1             6
#   6         1     0         3 T2            21
#   7         1     0         4 T3             4
#   8         1     2         3 T2             7
#   9         1     0         2 T1             6
#  10         1     2         4 T3             3
# # ... with 50 more rows

So each "bootstrap" replicate is labeled in the replicate column. If you'd like to write each replicate to a separate .csv file, you could do:

five_bootstraps %>%
  split(.$replicate) %>%
  walk2(.x = names(.), .y = ., ~ write_csv(.y, paste0(.x, ".csv")))


How to control for Co-variate shift in test data set compared to train data for regression task?

There are several options:

Change the training dataset

Use some of the test data as training data. This the best option since it better models the problem you are trying to solve.

Since it happens over time, take only the most recent data for training.

Manually engineer features. If you have knowledge of how the test data feature values are different, then explicitly put that knowledge in the model.

Increase regularization. For scikit-learn's ElasticNet, increase alpha hyperparameter. Unless your validation dataset is similar to your test data, you'll have no evidence that this will improve the modeling.

Switch algorithms to one that is better at generalization. Possibly, Random Forest could be better than regularized linear regression.


Understanding logistic regression loss function equation

Yes the first is the L2 regularization term to keep the norm of weights parameter as small as possible.


What is the difference between SGD classifier and the Logisitc regression?

Welcome to SE:Data Science.

SGD is a optimization method, while Logistic Regression (LR) is a machine learning algorithm/model. You can think of that a machine learning model defines a loss function, and the optimization method minimizes/maximizes it.

Some machine learning libraries could make users confused about the two concepts. For instance, in scikit-learn there is a model called SGDClassifier which might mislead some user to think that SGD is a classifier. But no, that's a linear classifier optimized by the SGD.

In general, SGD can be used for a wide range of machine learning algorithms, not only LR or linear models. And LR can use other optimizers like L-BFGS, conjugate gradient or Newton-like methods.


Appending the values of multiple columns into one whilst retaining UIDs

Assuming you already have a dataframe df, created as below:

In [1] : df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
                           columns=['a', 'b', 'c'])
In [2] : df
Out[2] :
   a  b  c
0  1  2  3
1  4  5  6
2  7  8  9

you could try doing this:

In [3] : df.melt("a").sort_values("a")
Out[3] :
   a variable  value
0  1        b      2
3  1        c      3
1  4        b      5
4  4        c      6
2  7        b      8
5  7        c      9

where the sorting might be optional in your case. Also, you should swap "a" for the name of your column containing the UID values.

You can additionally drop the autogenerated variable column too, so the final solution would be:

In [4] : df.melt("a").sort_values("a").drop("variable", axis=1)
Out[4] :
   a  value
0  1      2
3  1      3
1  4      5
4  4      6
2  7      8
5  7      9

From what I remember, the melt functionality is inspired on the melt function in the R programming language, where DataFrames are one of the core built-in data-types. It is really useful for plotting using the ggplot library (in R).


variance threshold, returning the names of the selected features

Use selector.get_support (Documentation).
This will give you a mask of the features that were selected and features that were discarded.
>>> selector.get_support()
array([False,  True,  True, False])

And here is how you get your selected features indices
>>> [ i for i, f in enumerate(selector.get_support()) if f ]
[1, 2]


Since answers change between different questions, your problem does not fit 'regular' classification problems.
And due to textual nature of your input/output, regression is not the best fit either.
This leaves me thinking K-NN is a good way utilizing supervised learning.

I don't have any good reference for this but this approach makes sense to me:

1) Embed both questions and answers into same space(using TF/IDF and PCA for example).
2) For a new(unseen) question find near neighboring labeled questions(from the training set), using K-NN.
3) Get neighboring questions answers.
4) Use K-NN(Or other distance based method) to find nearest neighbor in unseen question answer choices.


Cross validation is basically for hyper-parameter tuning.

You train a set of model hyper-parameter setting on kfold cross validation and take the average score from kfold cross validation as the approximate performance of each model hyper-parameter setting. Then, the model hyper-parameter setting which has the highest score will be the choice of your model setting. This hyper-parameter setting can be treat as the best you can get from this kind of model.

Later, you can use this model setting to test the general performance on your test dataset.


How does DQN solve Open AI Cartpole - v0?

You are mixing up two concepts from reinforcement learning, reward and return (aka utility)

Rewards are used to identify or specify goals of the agent. Whilst you can change them to help an agent focus on useful heuristics of the problem, it is more usual, especially in test/toy problems to have them very simple. In the case of CartPole, there is a positive reward for "not falling over" which importantly ends when the episode ends.
Returns (or utility) are what the agent learns to maximise. A return is typically the sum of all rewards, and might be discounted to prevent infinite results. In the case of CartPole, this means that the longer the agent can balance the pole into the future, the larger the return is.

With Q-learning, the action values predict the expected future return. So it doesn't matter that the rewards are dense. It matters how long the agent can keep balance going into the long term, the longer the better, because the return will be higher. A combination of state and actions that the agent associates with longer-lasting not failing will predict a larger return and be chosen in preference to shorter-term success. This is how the Q-learning agent handles a situation with dense positive rewards that may end on a mistake.

In practice, the OpenAI Gym CartPoleV0 environment does take a small liberty. Episodes will end at a fixed step in future. This is not available in the state information, and technically makes the problem non-Markov. However, it is possible to get away with this provided the timespan for maintaining stability is shorter than the maximum possible episode.


What is the difference between "Python interactive" and "Python 3" kernels

Interactive mode gives immediate feedback, i.e. every single line will be executed immediately, without possibility to script it i.e. following warnings:

  >>> if interactive:
  ...   print("do it")
  ... print("cant you?")
    File "<stdin>", line 3
      print("Done")
          ^
  SyntaxError: invalid syntax

so the fact that they are together confuses him, but if you seperate it

  >>> if interactive:
  ...   print("do it")
  do it

  print("cant you?")
  cant you?

So the main point is to seperate, and be carefull with new command lines.

Python 3 is standard scripting python version.

Why differentiate? Interactive to test quick and dirty, Python3 to write scripts


"Pip install tensorflow" only installs version 1.14 – how to install version 2 (with eager mode enabled?)

You install the version you want with:

pip install tensorflow=={version you want}

for example:

pip install tensorflow==2.0.0-beta1

If you are working in Google Colab it's even simpler, just type:

%tensorflow_version 2.x
import tensorflow as tf

and it will automatically import the latest version of TensorFlow 2.

Eager mode is default in TensorFlow 2.x, therefore you don't need to manually activate it. As soon as you import it it's immediately available.


How does Decision Tree with Gini Impurity Calculate Root Node?

From ISLR:

  ... we consider all predictors $X_1$, . . . , $X_p$, and all
  possible values of the cutpoint s for each of the predictors, and then
  choose the predictor and cutpoint such that the resulting tree has the
  lowest RSS ...

Since it's a classification problem, the best split is chosen by maximizing the Gini Gain, which is calculated by subtracting the weighted impurities of the branches from the original Gini impurity.

For c total classes with the probability of picking a datapoint with class, i is p(i), then the Gini Impurity is calculated as:

\begin{equation}
G = \sum_{i=1}^{c} [ p(i) * (1 − p(i)) ]
\end{equation}

1. Gini Impurity

Here, c = 2 , P(0) = 3/5 and P(1) = 2/5

G = [P(0) * (1 - P(0))] + [P(1) * (1 - P(1))]

G = [3/5 * (1 - 3/5)] + [2/5 * (1 - 2/5)] = 12/25

G = 0.48

2. Gini Gain

Now, let's determine the quality of each split by weighting the impurity of each branch. This value - Gini Gain is used to picking the best split in a decision tree.
In layman terms, Gini Gain = original Gini impurity - weighted Gini impurities So, higher the Gini Gain is better the split.

Split at 6.5:

Gini Impurity G_left  = [1/2 * (1 - 1/2)] + [1/2 * (1 - 1/2)] = 0.50
Gini Impurity G_right = [2/3 * (1 - 2/3)] + [1/3 * (1 - 1/3)] = 0.44
Weighted Gini  = (1/5 * .50) + (4/5 * 0.44) = 0.45
Gini Gain = 0.48 - 0.45 = 0.03

Split at 7.5:

Gini Impurity G_left  = [2/3 * (1 - 2/3)] + [1/3 * (1 - 1/3)] = 0.444
Gini Impurity G_right = [2/2 * (1 - 2/2)] = 0
Weighted Gini  = (3/5 * 0.444) + (2/5 * 0) = 0.27
Gini Gain = 0.48 - 0.27 = 0.21

Split at 8.5:

Gini Impurity G_left  = [2/4 * (1 - 2/4)] + [2/4 * (1 - 2/4)] = 0.500
Gini Impurity G_right$ = [1/1 * (1 - 1/1)] = 0
Weighted Gini  = (4/5 * 0.5) + (1/5 * 0) = 0.40
Gini Gain = 0.48 - 0.40 = 0.08

So, the best split will be chosen 7.5 because it has the highest Gini Gain.


Is there a way to print SVM margin obtained for each kernel used in sklearn?

You can check out this example on the sklearn site.


How would you teach a machine how to count?

I suggest using supervised learning and employing a linear model: linear regression.

This is a perfectly linear system (y=x+1), so linear regression will work just fine i.e. perfectly. Further, you have an infinite amount of data you can use to train the system, so it should be easy to train ;-)  I jest... I think that two data points will be sufficient, again, since it is perfectly linear!

Pedagogically, the triviality of this extremely simple linear system gets a little more interesting when you try analogue based methods like support vector machine (SVM) - which should also be able to provide a perfect result, decision trees or random forests, and even naive Bayes regressors.

Though its useful to start learning with very simple systems, I suggest quickly moving on to something more complex, i.e. don't get too stuck in your own head for the trivial linear model. Don't forget that the power of data science and machine learning lies in its statistical nature, so try to find a test case that includes some statistical variation in the input.

Hope this helps!


Would a decision tree classifier be applicable here?

As explained in here, decision trees have their disadvantages of a high chance of overfitting, instability with little variance, and low bias. Moreover, considering the fact that your data appears to be numerical, and continuous, there's a high chance that the decision tree might calculate the entropy of your features while only considering scenarios in your training data.

Decision trees tend to perform better in the scenarios where you have repetitive scenarios, and where your data is pretty much correlated, and most importantly, where you have discontinuous, categorical data in your features, which certainly isn't the case for the data which you mentioned.


Possible Challenges for a Data Science Escape Room

Fun! Cool idea, but it seems like a big undertaking. Anyways, here is my suggestion:

Summary

A manual fit of a 2D logistic regression which only becomes solvable once some previous step is completed.

Explaination

I imagine being greeted by a screen looking something like this:

The two boxes are inputs where the escapees can input numbers. If they try inputting at random it will be possible to figure out that the numbers represent $B_0$ and $B_1$ in: $$Y = B_0 + B_1 X1$$

This will still leave them confused since they will only see something like this:

However, when some other challenge in the room is finished the real challenge is revealed by colorizing the dots.

Then they just need to find the proper intercept and coefficient to finish challenge.

I think this would work quite well if there are enough samples so that you won't be able to hack it by trying values before they know the colors. If the challenge is too easy you could make them fit polynomials instead.


WHY or WHEN to convert numeric data to a categorical data?

One reason to convert numerical data to categorical data is to improve the signal-to-noise ratio. Fitting a model to bins reduces the impact that small fluctuates in the data has on the model, often small fluctuates are just noise. Each bin "smooths" out the fluctuates/noises in sections of the data.


Training Dataset for Sentiment Analysis of Movie Reviews

You can use the SAR14 dataset of 234K IMDb movie reviews. The construction of the SAR14 dataset is detailed in the paper "Sentiment Classification on Polarity Reviews: An Empirical Study Using Rating-based Features".


How does KNN handle categorical features

It doesn't handle categorical features. This is a fundamental weakness of kNN.  kNN doesn't work great in general when features are on different scales. This is especially true when one of the 'scales' is a category label.  You have to decide how to convert categorical features to a numeric scale, and somehow assign inter-category distances in a way that makes sense with other features (like, age-age distances...but what is an age-category distance?).

If all features are categorical, and inter-category distances are all treated as somehow equal, your job actually gets a little easier since you aren't converting between categorical and scalar scales.


Recommendations for storing time series data

There are two solutions that are worth looking at:

InfluxDB is an open source database platform specifically designed for time series data. The platform includes many optimized functions related to time and you can collect data on any interval and compute rollups/aggregations when reporting. The company recently launched a query app called Chronograf. I have not used this - but if its no good, you can also check out Grafana (which is very widely used and stable).

The alternative strategy you may want to pursue is an elasticsearch index. Elasticsearch is great at running aggregations and other mathematical functions on data. Many use it to store server log data and then query said data using Kibana.


What should I observe when choosing which optimizer suits my Deep Neural Network model?

High training score is not an indication of model performance, high test score is. Also, a faster convergence to the same, or better test score is an indication of optimizer performance.

Therefore, if Momentum optimizer reaches a better test score faster, it definitely means that it is the best out of those tested optimizers.

As a side note, be careful about the choice of "score", for example using "accuracy" for imbalanced classes is not a good choice, since it equates %1 error in a class with 100 members to 10% error in a class with 10 members. If classes are equally important, i.e. 1% error is equally bad for all classes, macro-f1 and AUC would be better replacements.

An important note: when we use a score to select a hyper-parameter or an optimizer, final model will be affected, thus, that score would be called validation score, not test score.


Sklearn Pipelines - How to carry over PCA?

When doing GridSearchCv, the best model is already scored. You can access it with the attribute best_score_ and get the model with best_estimator_. You do not need to re-score it in a cross validation.

Also, yes, the pipeline is entirely fitted when doing each split during the cv.


MATLAB @ symbol for fminunc / fmincg function

@(something) is used to call a function in MATLAB or octave.

Suppose you create a function within a code. And you set a keyword for that function.
we have a sum(x,y) function which takes two inputs and returns the sum.
Now you fix the value of y, say y = 3; And you want to change the value of x every time. You can design the inline function by following:

y=3
temp = @(p) sum(p,y);

ret = temp(4);  % What this will do is it will add 4 with 3 and return the answer.

This is used where we have to call a function multiple times and we don't want to write the arguments every time. It's a shortcut method.

In your case

fmincg (@(t)(lrCostFunction(t, X, (y == c), lambda)),initial_theta, options);

the value of t will be replaced every time with initial_theta. That is how the fmincg or fminunc functions work.


You'll find great datasets for this task called author verification from the PAN workshop series. Afaik the last one specifically on this task was in 2015: https://pan.webis.de/clef15/pan15-web/author-identification.html. I recommend exploring the website, there are many other datasets for related tasks.