How to Set the Same Categorical Codes to Train and Test data? Python-Pandas

First, note that Random Forests can handle categorical variables (moreover, if you have too much categories, reducing this number is a good practice).
If you want to apply a filter to your data, I'd suggest you using sklearn transformers (like OneHot Encoder, Label Encoding, ... pick the one you need according to what you want to do).
In this case, you have to fit the encoder in your train dataset, and then apply it in your test. If you want to apply this in a real case, you have to save your trained encoders alongside your trained model, so you can apply the encoder directly to the new data before predicting on it, so it has the same pattern.
Here is an example with Label Encoder
from sklearn import preprocessing
train, test = ... # SEPARATE YOUR DATA AS YOU WANT
le = preprocessing.LabelEncoder()
trained_le = le.fit(train)
train = trained_le.transform(train)
test = trained_le.transform(test)


The Why Behind Sum of Squared Errors in a Linear Regression

Simple google search on "stats why regression not absolute difference" would give you good answers. Try it yourself!

I can quickly summarise:

Your regression parameters are solutions to the maximum likelihood optimisation. That involves derivative, but absolute difference doesn't have a derivative at zero. There's no unique solution for least absolute regression.
Least absolute regression is an alternative to the regular sum of squares regression, commonly classified as one of the robust statistical methods.
You'd prefer least absolute regression if you care about outliers, otherwise the regular regression is generally better.

You might want to read about L1 vs L2:

  https://stats.stackexchange.com/questions/45643/why-l1-norm-for-sparse-models


You can use Keras, docs found here, to train a GAN. I have provided a GAN implementation in a previous answer here. Here is an alternative version that I have used more recently (found online).

import numpy as np
import time
from tensorflow.examples.tutorials.mnist import input_data

from keras.models import Sequential
from keras.layers import Dense, Activation, Flatten, Reshape
from keras.layers import Conv2D, Conv2DTranspose, UpSampling2D
from keras.layers import LeakyReLU, Dropout
from keras.layers import BatchNormalization
from keras.optimizers import Adam, RMSprop

import matplotlib.pyplot as plt

class ElapsedTimer(object):
    def __init__(self):
        self.start_time = time.time()
    def elapsed(self,sec):
        if sec < 60:
            return str(sec) + " sec"
        elif sec < (60 * 60):
            return str(sec / 60) + " min"
        else:
            return str(sec / (60 * 60)) + " hr"
    def elapsed_time(self):
        print("Elapsed: %s " % self.elapsed(time.time() - self.start_time) )

class DCGAN(object):
    def __init__(self, img_rows=28, img_cols=28, channel=1):

        self.img_rows = img_rows
        self.img_cols = img_cols
        self.channel = channel
        self.D = None   # discriminator
        self.G = None   # generator
        self.AM = None  # adversarial model
        self.DM = None  # discriminator model

    # (W−F+2P)/S+1
    def discriminator(self):
        if self.D:
            return self.D
        self.D = Sequential()
        depth = 64
        dropout = 0.4
        # In: 28 x 28 x 1, depth = 1
        # Out: 14 x 14 x 1, depth=64
        input_shape = (self.img_rows, self.img_cols, self.channel)
        self.D.add(Conv2D(depth*1, 5, strides=2, input_shape=input_shape,\
            padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        self.D.add(Conv2D(depth*2, 5, strides=2, padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        self.D.add(Conv2D(depth*4, 5, strides=2, padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        self.D.add(Conv2D(depth*8, 5, strides=1, padding='same'))
        self.D.add(LeakyReLU(alpha=0.2))
        self.D.add(Dropout(dropout))

        # Out: 1-dim probability
        self.D.add(Flatten())
        self.D.add(Dense(1))
        self.D.add(Activation('sigmoid'))
        self.D.summary()
        return self.D

    def generator(self):
        if self.G:
            return self.G
        self.G = Sequential()
        dropout = 0.4
        depth = 64+64+64+64
        dim = 7
        # In: 100
        # Out: dim x dim x depth
        self.G.add(Dense(dim*dim*depth, input_dim=100))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))
        self.G.add(Reshape((dim, dim, depth)))
        self.G.add(Dropout(dropout))

        # In: dim x dim x depth
        # Out: 2*dim x 2*dim x depth/2
        self.G.add(UpSampling2D())
        self.G.add(Conv2DTranspose(int(depth/2), 5, padding='same'))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))

        self.G.add(UpSampling2D())
        self.G.add(Conv2DTranspose(int(depth/4), 5, padding='same'))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))

        self.G.add(Conv2DTranspose(int(depth/8), 5, padding='same'))
        self.G.add(BatchNormalization(momentum=0.9))
        self.G.add(Activation('relu'))

        # Out: 28 x 28 x 1 grayscale image [0.0,1.0] per pix
        self.G.add(Conv2DTranspose(1, 5, padding='same'))
        self.G.add(Activation('sigmoid'))
        self.G.summary()
        return self.G

    def discriminator_model(self):
        if self.DM:
            return self.DM
        optimizer = RMSprop(lr=0.0002, decay=6e-8)
        self.DM = Sequential()
        self.DM.add(self.discriminator())
        self.DM.compile(loss='binary_crossentropy', optimizer=optimizer,\
            metrics=['accuracy'])
        return self.DM

    def adversarial_model(self):
        if self.AM:
            return self.AM
        optimizer = RMSprop(lr=0.0001, decay=3e-8)
        self.AM = Sequential()
        self.AM.add(self.generator())
        self.AM.add(self.discriminator())
        self.AM.compile(loss='binary_crossentropy', optimizer=optimizer,\
            metrics=['accuracy'])
        return self.AM

class MNIST_DCGAN(object):
    def __init__(self):
        self.img_rows = 28
        self.img_cols = 28
        self.channel = 1

        self.x_train = input_data.read_data_sets("mnist",\
            one_hot=True).train.images
        self.x_train = self.x_train.reshape(-1, self.img_rows,\
            self.img_cols, 1).astype(np.float32)

        self.DCGAN = DCGAN()
        self.discriminator =  self.DCGAN.discriminator_model()
        self.adversarial = self.DCGAN.adversarial_model()
        self.generator = self.DCGAN.generator()

    def train(self, train_steps=2000, batch_size=256, save_interval=0):
        noise_input = None
        if save_interval>0:
            noise_input = np.random.uniform(-1.0, 1.0, size=[16, 100])
        for i in range(train_steps):
            images_train = self.x_train[np.random.randint(0,
                self.x_train.shape[0], size=batch_size), :, :, :]
            noise = np.random.uniform(-1.0, 1.0, size=[batch_size, 100])
            images_fake = self.generator.predict(noise)
            x = np.concatenate((images_train, images_fake))
            y = np.ones([2*batch_size, 1])
            y[batch_size:, :] = 0
            d_loss = self.discriminator.train_on_batch(x, y)

            y = np.ones([batch_size, 1])
            noise = np.random.uniform(-1.0, 1.0, size=[batch_size, 100])
            a_loss = self.adversarial.train_on_batch(noise, y)
            log_mesg = "%d: [D loss: %f, acc: %f]" % (i, d_loss[0], d_loss[1])
            log_mesg = "%s  [A loss: %f, acc: %f]" % (log_mesg, a_loss[0], a_loss[1])
            print(log_mesg)
            if save_interval>0:
                if (i+1)%save_interval==0:
                    self.plot_images(save2file=True, samples=noise_input.shape[0],\
                        noise=noise_input, step=(i+1))

    def plot_images(self, save2file=False, fake=True, samples=16, noise=None, step=0):
        filename = 'mnist.png'
        if fake:
            if noise is None:
                noise = np.random.uniform(-1.0, 1.0, size=[samples, 100])
            else:
                filename = "mnist_%d.png" % step
            images = self.generator.predict(noise)
        else:
            i = np.random.randint(0, self.x_train.shape[0], samples)
            images = self.x_train[i, :, :, :]

        plt.figure(figsize=(10,10))
        for i in range(images.shape[0]):
            plt.subplot(4, 4, i+1)
            image = images[i, :, :, :]
            image = np.reshape(image, [self.img_rows, self.img_cols])
            plt.imshow(image, cmap='gray')
            plt.axis('off')
        plt.tight_layout()
        if save2file:
            plt.savefig(filename)
            plt.close('all')
        else:
            plt.show()

if __name__ == '__main__':
    mnist_dcgan = MNIST_DCGAN()
    timer = ElapsedTimer()
    mnist_dcgan.train(train_steps=10000, batch_size=256, save_interval=500)
    timer.elapsed_time()
    mnist_dcgan.plot_images(fake=True)
    mnist_dcgan.plot_images(fake=False, save2file=True)


Try to look at it from a broader perspective, what you're essentially doing is wanting to build a sort of multi-class classification model based on your clusters and the topics LDA assigned.
It's like wanting a clustering model and a classification model to do the same thing, which would ultimately depend upon the dimensional space the data points are in and their respective vectors.
I assume you must be using some sort of tweet to vector conversion method to represent a tweet, like bag of words or glove doc2vec or something. Now making clusters on this vector space and hoping this does the same segregation as LDA does to allot topic is quite farfetched. I don't think so, your vector space will be so much in line with the topics LDA produces for your clusters to reflect the same. Also clustering puts data points in same clusters based on certain attributes, but determining why a cluster is made based on these attributes if the data dimensions are is very difficult.


How do I use keras NN to classify data after training?

Once you have a trained model, you can pass new samples to it by using the predict method of the model, which will get you the probabilities of all classes. You than pick the class with the highest probability for each sample as the predicted class:

y_prob = model.predict(new_data)   # Get class probability vector for each sample
y_class = y_prob.argmax(axis=-1)   # Set prediction to class with highest probability

If you are using a Sequential() model, you can also use the predict_classes method, which will get you the same answer:

y_class = model.predict_classes(new_data)


in NLP academic paper, would it be okay to refer the "token embeddings" as "tokens"?

At least to me, it would sound strange, as I would understand tokens as the discrete textual units they are, not their assigned vectors.

I would suggest that you don't try to force nonstandard simplifications. Just say what you want, in a technically accurate way, preferably using short sentences to avoid making it difficult for the reader to follow your discourse.


Error: keras merge LSTM layers in sum mode

The error says what's the problem: the method expects a Tensors, but you are giving a Sequential model object.

Use functional model (from keras.models import Model), not Sequential.

Then, merge the models with:

merged_models = Model(inputs=[first_model_input, second_model_input], outputs=[first_model_output, second_model_output])

or whatever your input looks like.


Classifying one particular class of documents from the rest

Your problem is indeed a typical one-class classification problem, and as far as I know one-class SVM is usually a good option for that.

I think you should investigate what causes the poor performance:

Evaluating with accuracy is probably not informative enough, you would need to find out at least whether the errors tend to be mostly false positive or false negatives, thus using precision/recall.
You could look at what is happening at the level of features: I would expect some words specific to sports to be assigned a strong weight by SVM for instance. It could also be a problem with the dimensionality being too high, maybe you need to remove stop words or filter out rare words, etc.


Reinforcement Learning models are often measured relative to each other and sometimes relative to optimal behavior.

Regret is a common measure of performance for Reinforcement Learning systems.

Reinforcement Learning model performance is stochastic. Thus, is better to run the same model many times to estimate the distribution for performance. From that distribution, bounds can be estimated. Those bounds are similar to confidence intervals for scalar parameter estimates.


Number of features of the model must match the input. Model n_features is `N` and input n_features is `X`.

You are supposed to pass numpy arrays and not lists as arguments to the DecisionTree, since your input was a list it gets trained as 70 features (1D list) and your test had list of 30 elements and the classifier sees it as 30 features.

Nonetheless, you need to reshape your input numpy array and pass it as a matrix

meaning: X_train.values.reshape(-1, 1) instead of X_train (it should be a numpy array not a list)

this is the entire gist:

data=pd.read_csv("ndata.csv")

X_train, X_test, y_train, y_test = train_test_split(data.dis, data.gen, test_size=0.30, random_state=42)

from sklearn import tree

c = tree.DecisionTreeClassifier()
c.fit(X_train.values.reshape(-1, 1), y_train)

accu_train = np.sum(c.predict(X_train.values.reshape(-1, 1)) == y_train)/y_train_size
accu_test = np.sum(c.predict(X_test.values.reshape(-1, 1)) == y_test)/y_test_size

print("Accuracy on Train: ", accu_train)
print("Accuracy on Test: ", accu_test)

I'm getting the following output:

Accuracy on Train:  0.8857142857142857
Accuracy on Test:  0.7333333333333333

Thanks for sharing the dataset. It was helpful for testing.


What is the difference between a data-driven model and an empirical model?

They are the same thing, but it is important you understand differences in process between the statistics school of thought and the data mining mining school of thought. In data mining we impute and approximate relationships, where as some relationships are certain and have precise computations. Obviously, it is always better to compute something precisely than to approximate a relationship.

An example in finance: Bond prices have precise computations - they follow a certain relationship. The are simply discounted cashflows. Other types of securities might require inference because the absolute relationship is unknown, likely because either the cashflow or the interest rate is not known.

Empirical means based on evidence or observations. You are testing something. It might be possible to generate data per a simulation - this is common with genetic models - that is not observed in real life. In that instance, your data driven model would not be empirical.

  em·pir·i·cal
  /əmˈpirik(ə)l/
  | adjective |
  based on, concerned with, or verifiable by observation or experience rather > than theory or pure logic.
  "they provided considerable empirical evidence to support their argument"

Data mining versus statistics:

Normally what you try to do in statistics (as opposed to data science / data mining) is you come up with the definate mathematical model first, the you test it empirically. If the relationship is statistically significant, than it is thought to be a good model.

In data mining you seek out the inferred relationships first and have no manner of testing them.


How to anonymize (de-identify) data in Python?

I don't have access to input dataset, I created sample on my own and tried your code with little modification and it worked

input dataset:
housing      lastname   firstname
64403818    AA  AB
30893205    AC  AD
89883627    AE  AF
90302087    AG  AH

After I executed, input dataset appended with uid and transaction_date

transaction_date    uuid                                     housing    lastname
10/31/2019 20:35    809b4505-2269-48b0-8833-e7502fc2738a    64403818    AA
10/31/2019 20:35    7de91a91-0b58-4703-b62b-4278efe22b05    30893205    AC
10/31/2019 20:35    d6b8cfbd-a9c2-4ffd-b336-0a23547445ea    89883627    AE
10/31/2019 20:35    11db6b3a-9679-4422-b754-4c1b23aa4801    90302087    AG

firstname
AB
AD
AF
AH

and the output dataset becomes
transaction_date    uuid                                    housing
10/31/2019 20:35    809b4505-2269-48b0-8833-e7502fc2738a    64403818
10/31/2019 20:35    7de91a91-0b58-4703-b62b-4278efe22b05    30893205
10/31/2019 20:35    d6b8cfbd-a9c2-4ffd-b336-0a23547445ea    89883627
10/31/2019 20:35    11db6b3a-9679-4422-b754-4c1b23aa4801    90302087

import pandas as pd
import uuid as u
from datetime import datetime
import datetime as dt
 # generate a pseudo-identifier sequesnce using python random number generator library uudi.

def uudi_generator(length):
    uudi_list= list()
    i=0
    while i < length:
        uudi_list.append(u.uuid4())
        i+=1
    return uudi_list

#import original originaL dataset
dataset = pd.read_csv('C:\\mylocation\\input_credit_data.csv', index_col=False)

# pseudo identifier
sLength = len(dataset['housing'])
dataset.insert(0, 'uuid', pd.Series(uudi_generator(sLength), index=dataset.index))

# Transaction record attached to the original
dataset.insert(0, 'transaction_date', pd.Series([datetime.now()]*sLength, index=dataset.index))

#transcation record is attached to originaL data file
dataset.to_csv('C:\\mylocation\\input_credit_data.csv',index=False)

#delete identifiabLe record from dataset
del dataset['firstname']
del dataset['lastname']

# export  de-identified dataset as csv to be shared with the user
dataset.to_csv('C:\\mylocation\\output_bankcredit-data.csv',index=False)


Problem with two layers: how to predict?

This problem is best addressed with discrete-event, stochastic simulation modeling.  The prediction you get from such a model is a probability distribution on the surgery being canceled.

It's easy to code up such a model given your problem statement.  You need:

A boolean function for each kind of surgery, which, given the cast of characters and skill sets as input, determines True/False if the surgery can proceed.
A probability distribution for the appearance of each kind of character, when scheduled.
An object which is initialized with the schedule.

The idea is to do this check (surgery/surgery canceled) many times, thereby growing a set of outcomes for each policy you wish to explore.

If you don't want to roll your own, there are packages for many languages such as SimPy for Python and Simmer for R.

The nice thing about discrete event simulation is that you don't need to make any simplifying assumptions about probabilities, and the business logic can be arbitrarily complex.  For example, if the surgeon and one of the nurses are having an affair, the chance that one cancels on the same day as the other can be increased.


List of Correlations to Correlation Matrix

Well, if this is the whole table, I don't see how a matrix is useful, since one of the variables you're looking at doesn't change...you can use something like this to show the correlation to each category of variable 2 https://seaborn.pydata.org/generated/seaborn.countplot.html


Extracting name, date and total from a set of heterogeneous receipts

There is already an ML engine that does these extractions, here is a general process layout:

Hre is the orginal paper describing the architecture, features, approaches etc. read it up, instead of me copying it here. cloudscan


How to convert Scikit Learn logistic regression model to TensorFlow

Sure, here is a skeleton in TF 2.0.

import tensorflow as tf

weights = tf.Variable(tf.random.normal(shape=(784, 10), dtype=tf.float64))
biases  = tf.Variable(tf.random.normal(shape=(10,), dtype=tf.float64))

def logistic_regression(x):
    lr = tf.add(tf.matmul(x, weights), biases)
    return lr

def cross_entropy(y_true, y_pred):
    y_true = tf.one_hot(y_true, 10)
    loss = tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)
    return tf.reduce_mean(loss)

def grad(x, y):
    with tf.GradientTape() as tape:
        y_pred = logistic_regression(x)
        loss_val = cross_entropy(y, y_pred)
    return tape.gradient(loss_val, [weights, biases])


Soft and hardware for manipulating multi-million record data sets

Use vowpal wabbit for building your regression or classification models. It'll train at the speed at which you can read your data

Use pandas and dask for plotting graphs etc.
You can use an AWS i2.8xlarge or equivalent machine which will cost you ~ USD 6k for an year


Is there a way to replace existing values with NaN

Randomly replace values in a numpy array

# The dataset
data = pd.read_csv('iris.data')
mat = data.iloc[:,:4].as_matrix()

Set the number of values to replace. For example 20%:

# Edit: changed len(mat) for mat.size
prop = int(mat.size * 0.2)

Randomly choose indices of the numpy array:

i = [random.choice(range(mat.shape[0])) for _ in range(prop)]
j = [random.choice(range(mat.shape[1])) for _ in range(prop)]

Change values with NaN

mat[i,j] = np.NaN

Dropout for any array dimension

Another way to do that with an array of more than 2 dimensions would be to use the numpy.put() function:

import numpy as np
import random
from sklearn import datasets

data = datasets.load_iris()['data']

def dropout(a, percent):
    # create a copy
    mat = a.copy()
    # number of values to replace
    prop = int(mat.size * percent)
    # indices to mask
    mask = random.sample(range(mat.size), prop)
    # replace with NaN
    np.put(mat, mask, [np.NaN]*len(mask))
    return mat

This function returns a modified array:

modified = dropout(data, 0.2)

We can verify that the correct number of values have been modified:

np.sum(np.isnan(modified))/float(data.size)

[out]:

0.2


Simply set, 'parallel' means running concurrently on distinct resources (CPUs), while 'distributed' means running across distinct computers, involving issues related to networks.

Parallel computing using for instance OpenMP is not distributed, while parallel computing with Message Passing is often distributed.

Being in a 'distributed but not parallel' setting would mean under-using resources so it is seldom encountered but it is conceptually possible.


Backpropogation through Maxpool and relu

Why is backpropagation through maxpool and relu needed?

Any differentiable function (max pool and relu) through which inputs pass will have a gradient.. The nonlinear functions are still functions, so the chain rule still applies here...and hence we have the Gradients flowing through them..This gradient will have to be backpropagated...


How can I read input node and output node from .h5 file in keras model?

I assume that you have the Keras .h5 file ready. Now, we need to load the model from that file using,

model = keras.models.load_model( 'model.h5' )

model represents the Keras model. Now, model.input will correspond to the input tensor.

input = model.input
output = model.output

Also, you get the output in a similar way. The input and output tensors will have their names. This can be observed by printing the tensor using Python's print.


Full Doc2Vec Implementation/Repdoduction in TensorFlow?

Implementations
These are all full scripts using Tensorflow, but just using Tensorflow does not ensure quality.
Bad
WangZ's implementation looks complete from a brief glance, although it only offers Adagrad and Stochastic Gradient Descent, two of the least reliable gradient descents. You really want minibatch gradient descent, which will predictably converge close to the local minimum where you initialize your system (think: optimization problems from calculus class). Beyond that, there's nothing really notable about his or her implementation.
Better
Jonathan's implementation is a bit better than WangZ's but worse than the Machine Learning Cookbook solution. He also uses Gensim instead of TensorFlow, which is not bad except from an engineering standpoint (it doesn't scale quite as well). He uses mini-batch gradient descent, but with hierarchical softmax instead of NCE. This is not going to make a huge difference except perhaps increasing training time.
Best
The Machine Learning Cookbook implementation is really just incredible. It follows all of the recommendations from Google's original papers on the topic, even improving on the gradient descent (this uses mini-batch gradient descent). Notably, this also uses NCE (Noise Contrastive Estimation), which I don't typically see, instead of hierarchical softmax to obtain the error for gradient descent, which Google showed produces slightly better results and works faster. A lot of thought clearly went into this, and whoever was behind it knew what s(he) was doing. I would go with this one if I were you.
PV-DM vs PV-DBOW
PV stands for Paragraph Vectors, the model you are talking about. It is also know as doc2vec. It comes in two flavors, CBOW (Continuous Bag of Words), sometimes also called DBOW (Distributed Bag of Words), and DM (Distributed Memory).
These are both distributed models in that each "neuron" contributes to the training task, but nothing has any meaning without the other "neurons" to give it context.
To paraphrase Google's documentation: The CBOW/DBOW model predicts the target word from a bag of context words (e.g. "mat" from "the cat sits on the") while the DM model predicts a skip-gram from a single word (e.g. "sits .. mat" from "cat").
In practice, research shows that the skip-gram/DM model works better if you have a lot of data and the CBOW/DBOW model works better with less. That said, this technique (and deep learning in general) isn't particularly effective without lots of training data.
IMDB
Any of these implementations can be trained on comment data from IMDB. For example, the Machine Learning Cookbook implementation is hard-coded to pull data from a similar dataset . To change where it gets the data from, you would modify the load_movie_data() function in text_helpers.py (same directory in the repository as your link, but a different file) to pull from your preferred source. For instance, you might want to modify it to pull data from the local file-system instead.


Can KMeans clustering be used on word2vec output?

It's totally fine to cluster word2vec output to know semantically similar words. KMeans is an option, you might also want to checkout some approximate neighbor scheme such as Locality Sensitive Hashing.


Neural network accuracy for simple classification

I will add my 2 cents at the end of this answer. However, this is how it can be done using a neural network. Firstly, yes, you should expect to need more data to train even a simple neural network because their are more parameters that need tuning. Think of them like little faucets that you need to tune in order to get the right output volume based on an input. If you have millions of these faucets you an imagine that this is an arduous process.

You will need some of the following imports

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

But, in your case you know what your data should be so you can simulate it. I do this as follows and make a training and testing set.

import numpy as np

n = 10
m = 2

x_train = np.zeros((n, m))
y_train = np.zeros((n,))
for i in range(n):
    label = np.random.randint(0,m)
    y_train[i] = label
    x_train[i, label] = 1

x_test = np.zeros((n//3, m))
y_test = np.zeros((n//3,))
for i in range(n//3):
    label = np.random.randint(0,m)
    y_test[i] = label
    x_test[i, label] = 1

Now we will have a training set which contains $n$ instances and a testing set with a third as many. $m$ is the number of possible inputs. For cat vs. dog this would be $m=2$. You will be using your more general case where $m=10$. Each entry in the matrix $x$ has the vector with one-hot encoded vector where the index in accordance with the label is 1.

We need to reshape the data for it to fit with the Keras structure.

# The known number of output classes.
num_classes = m

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], m,)
x_test_reshaped = x_test.reshape(x_test.shape[0], m,)
input_shape = (m,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

We then build our model

model = Sequential()
model.add(Dense(64, activation='relu', input_shape = input_shape))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

We then train our model

epochs = 4
batch_size = 128
# Fit the model weights.
history = model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

  Epoch 4/4 10/10 [==============================] - 0s 251us/step -
  loss: 0.6247 - acc: 0.6000 - val_loss: 0.5311 - val_acc: 1.0000

Voila, now you have perfect classification with this network. You can play around with the model and see a summary of the model using.

model.summary()

For $m = 10$

Due to the higher complexity that this set has you will need to increase the number of instances in your training set. I will also 2 layers to our model and make them less wide. Furthermore, we will add more epochs so we will train longer.

import numpy as np

n = 1000
m = 10

x_train = np.zeros((n, m))
y_train = np.zeros((n,))
for i in range(n):
    label = np.random.randint(0,m)
    y_train[i] = label
    x_train[i, label] = 1

x_test = np.zeros((n//3, m))
y_test = np.zeros((n//3,))
for i in range(n//3):
    label = np.random.randint(0,m)
    y_test[i] = label
    x_test[i, label] = 1

# The known number of output classes.
num_classes = m

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], m,)
x_test_reshaped = x_test.reshape(x_test.shape[0], m,)

input_shape = (m,)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

model = Sequential()
model.add(Dense(32, activation='relu', input_shape = input_shape))
model.add(Dense(32, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test_reshaped, y_test_binary))

  Epoch 10/10 1000/1000 [==============================] - 0s 49us/step
  - loss: 1.5977 - acc: 1.0000 - val_loss: 1.5235 - val_acc: 1.0000

My suggestions

I would not use a NN for such a case. Most of the frameworks allow you to add information throughout your model. Such that if you have images you can run a CNN over them, then when you are ready to convert your layers to a densely connected layer you can in additional information, such as vectorized text.

You can thus use a random forests approach or something even simpler to get your 100% classification even faster. Then you can feed the output of this model to your deep learning framework which has already "extracted the features" from the images and concatenate these additional features to that tensor. Then you will pass this larger tensor through the subsequent Dense layers to get your final output.


Frankly, even 50 images will not be sufficient if you are going to create and use a CNN model. If you think you want more images for you model training, then go for data augmentation. It is a process of transforming an image by a small amount (be it height, width, rotation etc or any combination of these). In this way, an image and its augmented image will differ slightly. You can find relevant article here-

https://medium.com/nanonets/how-to-use-deep-learning-when-you-have-limited-data-part-2-data-augmentation-c26971dc8ced

To answer the part that should there be same number of images in each class, there should be approximately same number. This problem is a general problem while working on classification task and there are several ways to deal with it, including simulating the data (augmentation).

I would suggest that first create a separate test set, then on the remaining train set, use data augmentation and finally create the model.

EDIT

Using a pretrained convnet is also an option, as stated in a deep learning book-

  A common and highly effective approach to deep learning on small image datasets is to use a pretrained network. A pretrained network is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. If this original dataset is large enough and general enough, then the spatial hierarchy of features learned by the pretrained network can effectively act as a generic model of the visual world, and hence its features can prove useful for many different computer vision problems, even though these new problems may involve completely different classes than those of the original task. For instance, you might train a network on ImageNet (where classes are mostly animals and everyday objects) and then repurpose this trained network for something as remote as identifying furniture items in images.


How does a FC layer work in a typical CNN

Yes, essentially a typical CNN consists of two parts:

The convolution and pooling layers, whose goals are to extract features from the images. These are the first layers in the network.
The final layer(s), which are usually Fully Connected NNs, whose goal is to classify those features.

The latter do have a typical equation (i.e. $f(W^T \cdot X + b)$), where $f$ is an activation function. Usually in the context of CNNs, $f$ is a ReLU, except for the activation function of the final layer, which is selected according to the nature of the problem. The most common cases are:

Sigmoid activation functions work for binary classification problems.
Softmax activation functions work practically for both binary and multi-class classification problem.
For regression problems, the final layer has no activation.

One final note I'd like to make is that before entering the first FC layer, the output of the previous layer is flattened. By this I mean that the (typically 3) dimensions of that tensor are layed out into one large dimension.
For example a tensor with a shape of $(5, 5, 32)$, when flattened would become $(5 \cdot 5 \cdot 32) = (800)$.


Can we use a neural network to perform arithmetic operation between 2 numbers?

It is an easy task. Make some training data, two inputs and one output. this is a regression task. You can train a simple MLP or you can employ RNNs such as LSTM for this task. Take a look at here.


Boostrap parameter in random forest regressor?

Refer this answer on stack

  The benefit of random forests comes from its creating a large variety of trees by sampling both observations and features. Bootstrap = False is telling it to sample observations with or without replacement - it should still sample when it's False, just without replacement... You tell it what share of features you want to sample by setting max_features..


How to implement accurate counts or sums when you have different numbers of days of the week?

I would add a column that is a 3 if it's a weekday and a 4 if it's not using an apply, something like this:

df['divide_by'] = df.apply(lambda x: 3 if x['Day_int']<5 else 4, axis=1)

Assuming days are Monday 0 to Sunday 6. Then you can add the column as follows:

df['adj_total'] = df.apply(lambda x: x['Total']/x['divide_by'], axis=1)

Then you can remove the divide_by column and you have the result


Extract and normalize date from string for the proper text mining

The best option is dateparser, whose goal is to parse natural language dates with support for Dutch. There is another package recurrent which does natural language parsing of dates focusing on recurring events.


It has the same images since ILSVRC 2012. The test set images have never been released.


Have you heard about the intersection graph? You can try to draw players as points, connections (team mates) as edges and teams as transparent coloured blobs on top.

As for your original question, I cannot understand your goal. I think your formulation is not well defined / incomplete. Suppose you have teams A [1,2] B [2,3] and C [1,3]. What do you want to display? Do you want to list the parts of the Venn diagram? I think that for more than 3 sets this can become more cumbersome than the bipartite graph itself = simple listing of team compositions.


How can I change the transparency of a histogram plot in Seaborn using Pairgrid?

With a pandas dataframe, the following should do it without seaborn:

train_df.groupby('Embarked')['Fare'].plot.hist(alpha=0.5)

If you have to use seaborn you can use FacetGrid without the col and row argument:

g = sns.FacetGrid(train_df, hue='Embarked')
g = g.map(plt.hist, 'Fare', alpha=0.5)
g = g.add_legend()


How valuable is a categorical feature that has a predominant category over all other ones?

Feature importance is an empirical question. Train a model with the feature in it. Train another model without the feature in it. Then see which model does better when predicting new data, aka the test dataset. The difference between model performance on train and test datasets is one way to measure overfitting.


You can test with higher values of Dropout (0.5,0.7,0.9) and/or try L1/L2 Regularization to combat overfitting  : Keras Regularizers.

Update:
You can play with a combination of l1/l2 regularization and dropout for your convolutional and FC layers. Start with low values of lambda (0.001) and increase it thereon. A common practice is to have dropout only in the FC layers, see if it helps your problem.

Also, from the looks of your loss curve, your model probably hasn't converged.You can train it until your validation loss starts going up/becomes stable.


Cleaning data with two fields mixed in the same column?

I suggest adding a company name for each corresponding month. See the attached picture.
The formula for the first column determined if it is for a month or for the company name. Assuming that all your months are in the three-letter format and there is no company named 'May' or 'Sep', the formula for the cell B2 would be
=SUMPRODUCT(--(A2={"Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"}))>0

The formula for C2 is
=INDEX($A$2:$A$13,MATCH(2,1 /($B$2:B2=FALSE))+1)

Please refer to this page (https://www.get-digital-help.com/index-match-last-value/) for explanations for this formula.
Finally, you can filter the third column in my example to keep months for the specific company only. They will be in the order needed.


SHAP Explanations in case of repeated train/test split

I'd propose doing a similar thing that you do with your accuracy and Kappa metrics — calculate SHAP values for all 500 splits, and take the average of these $n\_samples \times n\_features \times 500$ matrixes in the third dimension to get $n\_samples \times n\_features$ matrixes, which you can use to create your desired plots.


What Does the Normalization Factor Mean in the AdaBoost Algorithm?

The normalization factor  is used to reduce any probability function to a probability density function with total probability of one. See Wikipedia.

Say your unnormalized value is [0.1, 0.2, 0.3, 0.2]. You normalize it by dividing it by $z_t=0.1+0.2+0.3+0.2=0.8$, therefore get the normalized value [0.125, 0.25, 0.375, 0.25] which sums up to 1..


Default value of learning rate in adam optimizer - Keras

Learning rate is a very important hyperparameter, and often requires some experimentation.  There are some good Related questions here, make sure to check those out.

With too large a learning rate, you might bounce around an optimum, or you might start off by sling-shooting out to a part of the parameter space where the gradients vanish.  (With too small, you might take too long to converge to an optimum, or you might find a poor local optimum.  These effects are lessened by adams momentum effect.)
That said, the best learning rate generally depends on the problem.  Presumably the default was chosen pretty well for general use, but your finding isn't surprising.
There is a parameter decay for simple decay, or you can set up a callback for step decays.


Reducing the effect of down voters with rating system

You should look into other estimators of location.

What you want is a robust estimator, with a high break-down point.

The extreme approach would be the median.

But you may get more numerically interesting results with a trimmed mean.

You define a threshold, say 2%. Then you remove the top 2% of votes, and the bottom 2% of votes, and take the mean only of the remaining entries. An app with 98% 5 stars will still get a 5.0

But to prevent manipulation, I would look into other signals. Such as clustered votes from a single region, for example.


Sequence to sequence RNN model, maximum number of training size

so I found a solution by dividing input training data into small batches and it did the trick. Here is the code:

divideby=1000
for j in range(divideby):
    start=j*len(input_texts)/divideby
    end=(j+1)*len(input_texts)/divideby if j<divideby+1 else -1

    encoder_input_data = np.zeros(
        (end-start, max_encoder_seq_length, num_encoder_tokens),
        dtype='uint8')  # size_of_training_samples, max_length_word_measured_in_characters,number_of_unique_chars,
    decoder_input_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')
    decoder_target_data = np.zeros(
        (end-start, max_decoder_seq_length, num_decoder_tokens),
        dtype='uint8')

    for i, (input_text, target_text) in enumerate(zip(input_texts[start:end], target_texts[start:end])):
        for t, char in enumerate(input_text.split(splitby)):
            encoder_input_data[i, t, input_token_index[char]] = 1.
        for t, char in enumerate(target_text.split(splitby)):
            # decoder_target_data is ahead of decoder_input_data by one timestep
            decoder_input_data[i, t, target_token_index[char]] = 1.
            if t > 0:
                # decoder_target_data will be ahead by one timestep
                # and will not include the start character.
                decoder_target_data[i, t - 1, target_token_index[char]] = 1.

    model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
              batch_size=batch_size,
              epochs=epochs,
              validation_split=0.2)
```


What is the difference between BERT architecture and vanilla Transformer architecture

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder
BERT ends with the final representation of the words after the encoder is done processing it.
In Transformer, the above is used in the decoder. That piece of architecture is not there in BERT


Calculate regression coefficients for individuals (low sample size regression)?

@Ben Norris found out that the relaimpo packages has a hard minimum number of observations, so if I wanted to pursue this path I have to up my sample size.

As I only have the data that I have, I pursued a "hacky" solution which I am going to describe for completionists sake. The steps were as follows:

Assign each individual to one of k groups randomly, so that n/k > 4 (with n being the total sample size)
Repeat this step i couple of times,so that each individual is assigned to igroups
Split the total data set into a list of  k dfs along the groups
Train a regression model with relaimpo for each data set
Average the resulting coefficients over all groups an individual was a part of to get approximate "individual coefficients"

This is a rather unscientific process but seemed better to me than simply "duplicating" individuals answers until the minimum sample size was met.

Here is the code I used:

library(dplyr)
library(magrittr)
library(relaimpo)

#create groups
df %>%
  add_count(i_TAN) %>%
group_by(i_TAN) %>%
  mutate(
    g1 = as.integer(runif(n, 1, 51)),
    g2 = as.integer(runif(n, 1, 51)),
    g3 = as.integer(runif(n, 1, 51)),
    g4 = as.integer(runif(n, 1, 51))
  ) %>%
  {.} -> df

# Create all dfs
df %>%
  select(i_TAN,g1,g2,g3,g4) %>%
  gather(nam,group,-i_TAN) %>%
  distinct() %>%
  select(-nam) %>%
  left_join(df, by = "i_TAN") %>%
  select(1:35) %$%
  split(.,group) %>%
  {.} -> list_of_df

lapply(list_of_df, function(x) { x["group"] <- NULL; x }) %>%
  lapply(function(x) { x["i_TAN"] <- NULL; x }) %>%
  {.} -> list_of_modelDF

# Fit all models
lapply(list_of_modelDF,function(x){lm(Consideration~.,data = x)}) -> list_of_reg

lapply(list_of_reg,function(x){relaimpo::calc.relimp(object = x, type = c("lmg"), rela = TRUE)}) -> list_of_relaimpo
```


Is regression or ANOVA the right approach for determining RRP based on a set of RRPs associated with categorical variables

ANOVA is a linear regression...a regression on categorical variables.
Regressions can have any mix of continuous and categorical variables. Some, like ANOVA and ANCOVA, have special names that you may want to use when you're communicating with people who have less training in statistics and don't know how to unify them with linear regression.


It depends on the further analysis that you want to perform. Increasing amount of occurrences can significantly effect the properties of your time series data, because you have only 10% of the data available.

In general, this is an interpolation question. There are fundamentally different statistical models that you can use to find the missing data. The Pandas dataframe.interpolate() is a good way to start.


Keras model giving error when fields of unseen test data and train data are not same

As others before me pointed out you should have exactly the same variables in your test data as in your training data.

In case of one-hot encoding if you have unseen categories in your test data your model doesn't know how to handle them it was not trained on those variables. In that case during data preparation you shall create all the variables that you had during training with the value of 0 and you don't create new variable for the unseen category.

I think your confusion and the differing number of variables come from the function that you use to do the one-hot encoding for you. Probably you run them on the two datasets separately and it will only create the variables that it founds in the specific datasets.
You can overcome on it by using label encoder or onehotencoder transformer from scikit-learn that will save inside its obeject the original state and in every transformation it will recreate exactly the same structure.

UPDATE to use sklearn onehotencoder:

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit(train_categorical_data)
encoded_train=encoder.transform(train_categorical_data)
encoded_test=encoder.transform(test_categorical_data)

You can save the encoder to use it later. See more about it in the official documentation.


As long as your operations are all compatible with pytorch tensors and Autograd then yes your network will be trained end-to-end.

A good rule of thumb is to ask yourself : I'm I using tensors and pytorch operators end-to-end? If yes (and unless stated otherwise in the docs), you should be pretty safe.


Downsampling audio files for use in Machine Learning

Resampling of audio is a standard process and there are many implementations available. In Python you can use librosa, or you can write a script that uses ffmpeg or similar.

If you want to reuse an already trained model this is critical, as the neural network will have learned features based on 16kHz input. It might work OK if you re-train the model with your data using 48kHz. It will be slower to train and predict, but performance can be similar.
However your would need to update the audio_sr parameter of the audio feature extraction. Otherwise the length of window in milliseconds would be wrong, probably no longer matching what is typical for speech, resulting in degraded performance.


How to find a similarity value between cars

I suggest you use a weighted per-attribute similarity, for instance let a and b a pair of tuples representing the attributes of some car A and another car B. For example:

a = (69000, 2012, base)
b = (71000, 2013, base)

For the km driven I would use Skm = 1 / (1 + |km(a) - km(b)|), so:

Skm = 1 / (1 + |69000 - 70000|) = 1 / 1001 = 0.0009

For the case of the year the same function could be use, then:

Syear = 1 / (1 + |2012 - 2013|) = 1 / 2 = 0.5

Last, for the trim of the car I would use a function that returns 0 if the categories were different 1 otherwise.

Strim = (base == base) = 1

Finally,

S = Wkm * Skm + Wyear * Syear + Wtrim * Strim

Where Wkm + Wyear + Wtrim = 1.0 and I would adjust the values according to what you think is decent accurate similarity ratio. Setting the values to be Wkm = 0.6, Wyear = 0.2, Wtrim = 0.2  the similarity S(a, b) = 0.30054. This similarity has the nice property that tuples with the same attributes yield a value of 1.

A good practice is to normalize the values of each attribute so you don't assign low values to any attribute for instance you could divide the columns of the km and year attributes for the maximum value, therefore for the pair of tuples in the example you will have the following transformed tuples:

a' = (0.985, 0.99, base)
b' = (1, 1, base)

Now the similarity using the same weights is S(a, b) = 0.9. Alternatively you could use a similarity matrix for the trim of the car, this is base could be more similar to mid thant to top. For example:

cat  | base | mid | top |
-------------------------
base |  1   | 0.5 | 0.2 |
-------------------------
mid  |  0.5 |  1  | 0.7 |
-------------------------
top  |  0.2 | 0.7 |  1  |

The values of the matrix have to be adjusted to fit your needs. For a more in depth comparison between similarity functions between categorical attributes (trim) see this for a comparison of different similarity measures for continuous attributes (year, km) see this.

Again to see which similarity is the best will depend on the data and on your perception, I would select a random set of pairs of cars and manually assign a similarity. Then split this set into a train set and test set. Use the train set to adjust the parameters of the different possible similarities, in the case of my proposal Wkm, Wyear and Wtrim. Then once your satisfied evaluate on the test set.


How to get the number of syllables in a word?

You can try another Python library called Pyphen. It's easy to use and supports a lot of languages.

import pyphen
dic = pyphen.Pyphen(lang='en')
print dic.inserted('Rohit')
>>'Ro-hit'


OK WHAT I understood from your understanding is below:-

1.What do I set η to?
Answer:**It is learning rate. As usual everyone uses **η=0.01. Learning rate is crucial because if you set bigger values when updating weights the it will reach to a local minima and never reach to global minima. Now if you set it low then it will gradually went to global minima. This iris dataset problem is so trivial that it takes no time to reach global minima but for a complex dataset you will need another factor such as decay rate. After a while, when training you will find the accuracy isn't increasing then the neural net suppose that it reached global minima now if you set decay rate to learning rate like 0.01* decay rate then learning rate will be much less and then it will TRULY converges to global minima as the gradient will be minimized in a small amount.

2.Z is the label for your dataset.You should have used one hot encoding. I think rather than writting everything here i would suggest following links.
Please go to there and then pick up a paper and pen.do the math.Bang.

How Neural Network learns

And obviously check 3Blue1Brown youtube channel's playlist on Neural Net.

Welcome to Deep Learning.Best of luck.


Data normalisation and recommendation based on skillset

Root words

The problem here is that language has many different ways of distorting words to give slightly more information without changing the root of the word. Your database which contains the needed courses for a specific profession should ideally be tokenized into a numerical value. Or if that is not possible you can get the stem of a word as follows

from nltk.stem.snowball import SnowballStemmer
stemmer = SnowballStemmer("english")
print(stemmer.stem("Accountant"))
print(stemmer.stem("Accounting"))

  account account

Recommendations

There's a lot of resources on how to do this, especially using deep learning in recen tyears. You can look at some of the publications from Netflix, Amazon, SoundCloud. All these groups have very powerful recommendation systems. In essence what you want to do is determine some user metrics such as click through sequence, search history. Then build a database with these features and a target such as the next item that is played. From this you can train your model to predict a set of items the user is likely to click on next given his feature set.


How does tensor product/multiplication work in TensorFlow?

You may want to read the documentation.

output[..., i, j] = sum_k (a[..., i, k] * b[..., k, j]), for all indices i, j.

For instance, in your example

$~~88=1\times12+2\times14+3\times16,~~~94=1\times13+2\times15+3\times17$
$214=4\times12+5\times14+6\times16,~229=4\times13+5\times15+6\times17$


Why does reducing the n_estimators in RandomForestClassifier improve accuracy?

If you are talking about testing accuracy in this case (ie you are comparing results on data you didn't train with) - it's possible that adding more estimators is overfitting on your training set and is therefore performing poorly on your holdout set. If this is the case I would recommend approaching the problem with a more basic method such as LogisticRegression - as it is less likely to overfit when compared to ensemble methods.
As for finding the best parameters - try sklearn's RandomizedSearchCV to fine-tune your hyperparameters to maximize performance.


I am working on a project, and I've been trying to come up with a custom loss/score function. Would really appreciate some help

Instead of computing $A$, make your algorithm compute $10\times A.$
Then use your favourite loss function for continuous variables. The most common one is the squared $L^2$-norm $l(10\times A, B) = (10\times A-B)^2.$

Here is a plot of $l$ that I made with Wolfram Alpha:.


Predicting most likely application to be opened

How do you plan to predict the exact app?  Are you thinking of a user level model (you can't predict that a user will open app X if she doesn't have it installed in the first place) Or maybe you want to predict apps at a segment level( instead of predicting individual apps)
A neural network should work well if you have enough data since it has good "capacity"
The time series part comes from the fact that the probability of opening app depends on what app was opened before it. This is called "structured prediction" and is quite common in language models etc.
The output will be a distribution over the apps and you can use a soft max layer for the final output


Learning rate and memory of Least Mean Square algorithm

This should be the weight update formula your model uses.

$W_{n+1} = W_n - \mu\nabla\epsilon[n]$,

$\mu$ is the learning rate, and we multiply it with the change we want to do. Obviously, the smaller the learning rate you have, the less update you make, and thus the "longer" the model converges and also the "longer" the model remembers its previous states.

Think like this, if you set the learning rate to 0, you learn nothing and your parameters are never updated.


How should I choose n_features in FeatureHasher in sklearn?

As mentioned in its documentation, it is advisable to use a power of 2 as the number of features; otherwise, the features will not be mapped evenly to the columns. Also, it is suggested to leave the number of features as its default value of 2 ** 20 for a real-world setting. Select a lower value such as 2 ** 18 when memory or downstream model size is an issue Ref.
Consider in mind that, as also stated in the documentation, small numbers of features are likely to cause hash collisions, but large numbers will cause larger coefficient dimensions in linear learners Ref
My overall suggestion is to use a power of 2 as the number of features. If you have a small number of features, you can treat the number of features (e.g., 30 as you mentioned in the comment below) as a hyperparameter and find the optimal value using cross-validation. For example, you can test different powers of 2 such as 2, 4, 8, 16, etc depending on the size of your data and use the cross-validation to find the optimal value. That is the best solution. But please note that the hashing method is efficient when the number of input features is very large. In your case, I would go with the other methods available in the technical literature.


Dealing with categorical variables in regression problems which method to use?

This solution would be performant only if your values has an order. Some models use as learning function the distance between points, and if you use your method, a student in Math and a student in English (0 and 2 making a 2 distance) will have more distance than a student in Math and a student in Science (0 and 1 making a 1 distance).
Using this method involves a bias, so you'll have to go another way. One well known method is One Hot Encoding, which will create 3 binary-variables Column1_Math, Column1_Science, Column1_English, with values 0 or 1 (for example, if column 1 is Math, then you'll have Column1_Math = 1, Column1_Science = 0, Column1_English = 0). This way, you avoid biaising your model.
I already explained other ways I know to deal with your issue in this answer that I highly suggest you to take a look at


Am I supposed to be using Mixed Effects?

I always thought that the mixed effects model was for logistic regressions when my dependent variable was categorical

This is not true at all. We use mixed effects, in particular random intercepts when we have clustered data, such as repeated measures on individuals, or sampling within clusters such as schools, or regions, or countries. It is certainly not the only approach - instead we can use Generalised Esitmating Equations, or just fit fixed effects for the grouping variable, but when there are a large number of groups, and we can justifiably consider their "effect" to be normally distributed and uncorrelated with the residual error or the fixed effects, a mixed effects model is often very attractive.


How should I approach such classification problem where the input is an array of integers?

Your information is not discriminative enough

Why? Coefficients of polynomials dont give (alteast partially) discriminative information about roots of the polynomial. In other words different coefficients could give same roots.

It does not matter how complex your network is it cant catch what is not there to be catched in the first place.


Are there deduplication algorithms that do not work on a metric space?

One option is fingerprinting. If two objects have the same fingerprint, they are probably the same object. Depending the technique used, the fingerprint can not tell about approximate duplicates.


PCA shows overlapping boundaries, then why SVM performs best

I assume you applied SVM to your initial data and use PCA only for visualization. I this case:

I guess your projection via the PCA is not showing the real picture.

You should check first how much of your data is explained with the first two principal components of the PCA. Your projection might change to much the structure of your data so that its not seperable anymore. In case the projection to the selected principal components is not changing your data too much, it could be that you maintain the seperability.

Finally, note:
If the projection is linearly seperable, then so is your data.
If the projection is not linearly seperable, you cannot conclude about seperability of your data.


Difference between isna() and isnull() in pandas

Pandas isna() vs isnull().

I'm assuming you are referring to pandas.DataFrame.isna() vs pandas.DataFrame.isnull(). Not to confuse with pandas.isnull(), which in contrast to the two above isn't a method of the DataFrame class.

These two DataFrame methods do exactly the same thing! Even their docs are identical. You can even confirm this in pandas' code.

But why have two methods with different names do the same thing?

This is because pandas' DataFrames are based on R's DataFrames. In R na and null are two separate things. Read this post for more information.

However, in python, pandas is built on top of numpy, which has neither na nor null values. Instead numpy has NaN values (which stands for "Not a Number"). Consequently, pandas also uses NaN values.

In short

To detect NaN values numpy uses np.isnan().
To detect NaN values pandas uses either .isna() or .isnull().
The NaN values are inherited from the fact that pandas is built on top of numpy, while the two functions' names originate from R's DataFrames, whose structure and functionality pandas tried to mimic.


LSTM neural network for music generation

These are some suggestions you can try:

Just use 1 LSTM layer instead of 2, there might be less problem of vanishing gradient for shallower networks. In fact the dense layer of 4 is optional, since return_sequences = True.
Find a way to normalize the input in the range 0 to 1.
Try different optimizers like SGD or Adam.
Reduce your learning rate, the default learning rate might be too large for an input dimension of 4.
The loss should not be categorical_crossentropy, since this is a regression problem. The loss should mean squared error.

In order of importance 4 > 5 > 3 > 1 > 2 (i.e point 4 is most important)


How can I create a negation of the sentence?

You could look in to Style Transfer. This is an active field of research.


What should I start with first? Python for beginners or applied data science?

TLDR: No, it is not required for someone with your background to take "Python for everyone" beforehand.
I have taken the "Applied Data Science with Python Specialization" by UofM and it provided a very soft start in terms of Python for Data Science. Also, see the description of the first course of the specialization "Introduction to Data Science in Python":

This course will introduce the learner to the basics of the python programming environment, including fundamental python programming techniques such as lambdas, reading and manipulating csv files, and the numpy library. The course will introduce data manipulation and cleaning techniques using the popular python pandas data science library and introduce the abstraction of the Series and DataFrame as the central data structures for data analysis, along with tutorials on how to use functions such as groupby, merge, and pivot tables effectively. By the end of this course, students will be able to take tabular data, clean it, manipulate it, and run basic inferential statistical analyses.

As they say, the course walks you through the important libraries, such as Numpy and Pandas, and covers some key methods. Moreover, the second course of the specialization "Applied Plotting, Charting & Data Representation in Python" provides an intro to Matplotlib. So this one is covered too.
Finally, if you would like to go for competitions I highly recommend the course "How to Win a Data Science Competition: Learn from Top Kagglers" (as a next step). It is more advanced than the UofM specialization and provides great insights in the mechanics of and approaches for competitions.


Improving Validation Loss and Accuracy for CNN

Your data set is very small, so you definitely should try your luck at transfer learning, if it is an option.

Other than that, you probably should have a dropout layer after the dense-128 layer. Also, it is probably a good idea to remove dropouts after pooling layers. Instead, you can try using SpatialDropout after convolutional layers.


How to use Embedding() with 3D tensor in Keras?

I'm not entirely sure if this is the cleanest solution but I stitched everything together. Each of the 10 word positions get their own input but that shouldn't be too much of a problem. The idea is to make an Embedding layer and use it multiple times. First we will generate some data:

n_samples = 1000
time_series_length = 50
news_words = 10
news_embedding_dim = 16
word_cardinality = 50

x_time_series = np.random.rand(n_samples, time_series_length, 1)
x_news_words = np.random.choice(np.arange(50), replace=True, size=(n_samples, time_series_length, news_words))
x_news_words = [x_news_words[:, :, i] for i in range(news_words)]
y = np.random.randint(2, size=(n_samples))

Now we will define the layers:

## Input of normal time series
time_series_input = Input(shape=(50, 1, ), name='time_series')

## For every word we have it's own input
news_word_inputs = [Input(shape=(50, ), name='news_word_' + str(i + 1)) for i in range(news_words)]

## Shared embedding layer
news_word_embedding = Embedding(word_cardinality, news_embedding_dim, input_length=time_series_length)

## Repeat this for every word position
news_words_embeddings = [news_word_embedding(inp) for inp in news_word_inputs]

## Concatenate the time series input and the embedding outputs
concatenated_inputs = concatenate([time_series_input] + news_words_embeddings, axis=-1)

## Feed into LSTM
lstm = LSTM(16)(concatenated_inputs)

## Output, in this case single classification
output = Dense(1, activation='sigmoid')(lstm)

After compiling the model we can just fit it like this:

model.fit([x_time_series] + x_news_words, y)

EDIT:

After what you mentioned in the comments, you can add a dense layer that summarizes the news, and adds that to your time series (stock prices):

## Summarize the news:
news_words_concat = concatenate(news_words_embeddings, axis=-1)
news_words_transformation = TimeDistributed(Dense(combined_news_embedding))(news_words_concat)

## New concat
concatenated_inputs = concatenate([time_series_input, news_words_transformation], axis=-1)


You have to see things for what they are. First, you have to figure out what PCA does. It's not clustering, but it can be used to realize that there could be clusters in the data.
PCA is not always the answer. It depends on the data. There are other dimensionality reduction techniques such as autoencoders, non-negative matrix factorization and so on...

For the movie example, it does not make sense to use PCA. The iris database contains another type of data, where it does make sense. For the movie example, you might want to have a look at singular value decomposition and collaborative filtering.

One of the reasons for doing PCA could be for clustering. If you have many dimensions, and the intrinsic dimensionality is low you will have the curse of dimensionality. Reducing the dimensionality by PCA and doing clustering on the embeddings solves, in this case, the curse of dimensionality.


I think there is no need to use ML for this problem. We can solve this by a simple lookup table/map table. The thing is we have to update the lookup table whenever we encountered a new category.

To apply ML to this problem. You should have the data of a set of preprocessed, processed responses like [Epilex 300mg(preprocessed), Epilex 300(processed)]. If you have this data, you can apply the decision tree(without pruning) model to predict the processed text, given an unprocessed text. Remember you will get predictions with only the categories within the trained categories. What you can do is, if a new category has occurred you can manually add that response to train data. And train your data again. If next time that category is encountered it will predict correctly. Like this you can improve your model prediction power. After some time you can able to predict every response correctly. The screen shorts of sample code is attached here.

To have actual code consider this Link.


How to Monitor ML Classification Models in production?

Yes, PSI and CSI are useful when we want to understand if there is any change/shift in the population type.
But to understand multi collinearity, VIF is what you should look into.
Ref: https://etav.github.io/python/vif_factor_python.html
and if you think you are left with only few variables, then you need to create few additional variables (interaction variables) and where there is a chance that they may not be correlated with others.


How to use a multiple linear regression model built from normalized data

So, the question asks:

How to normalise incoming (individual observations)
How to get the real value predictions and not the normalised values.

When we do normalisation using the Sci-kit learn module, instead of using the very handy fit_transform() method in the scaler, you could instead perform a .fit() over your original observations and then apply the .transform() to the newly-observed values. Obviously, in this case you need separate scalers foreach feature, as they are distributed differently from one another.

Again in Sci-kit learn, there is an inverse_transform() method which reverts the normalised value back to the original scale.


You should take out the image embeddings of the uniform (under different conditions) and the employee and match the embeddings. You can also remove the face of employee after detecting it using MTCNN.


What is a good hardware setup for using Python across multiple users

Is installing Python locally a good practice?  Yes, if you are going to develop in Python, it is always a good idea to have a local environment where you can break things safely.

Is there value in setting up a Python "server"?  Yes, but before doing so, be sure to be able to share your code with your colleagues using a version control system.  My reasoning would be that, before you move things to a server, you can move a great deal forward by being able to test several different versions in the local environment mentioned above.  Examples of VCS are git, svn, and for the deep nerds, darcs.

Furthermore, a "Python server" where you can deploy your software once it is integrated into a releasable version is something usually called "staging server".  There is a whole philosophy in software engineering — Continuous Integration — that advocates staging whatever you have in VCS daily or even on each change.  In the end, this means that some automated program, running on the staging server, checks out your code, sees that it compiles, runs all defined tests and maybe outputs a package with a version number.  Examples of such programs are Jenkins, Buildbot (this one is Python-specific), and Travis (for cloud-hosted projects).

What are the hardware requirements for such a box? None, as far as I can tell.  Whenever it runs out of disk space, you will have to clean up.  Having more CPU speed and memory will make concurrent builds easier, but there is no real minimum.

Do I need to be concerned about any specific packages or conflicts between projects?  Yes, this has been identified as a problem, not only in Python, but in many other systems (see Dependency hell).  The established practice is to keep projects isolated from each other as far as their dependencies are concerned.  This means, avoid installing dependencies on the system Python interpreter, even locally; always define a virtual environment and install dependencies there.  Many of the aforementioned CI servers will do that for you anyway.


In general, you cannot know which is better until you try both with lots of data and evaluate them with a strong statistical test.
I know you won't like this answer but this is how neural networks work.
Hope this helped


Neural Network backprop formula - Matrix dimensions won't match?

It's important to note that you are storing weights towards previous layer in columns. So my example will be specifically for that case.

If you are using a different notation (weights from neuron towards previous layer are in rows of your matrix) -  swap the words "rows" and "columns" below:

For your case, any incoming gradient (from the layers above or from the Error aka Cost function) must have dimension 2.  It must be a vector. Otherwise it will not physically "match" your output nodes.

Section A

The gradient wrt to your weights (hidden-to-output layer) will be a 3x2 matrix. You can subtract this matrix from your weights, to correct them. Don't forget about learning rate.

To build this matrix, copy the incoming gradient (vec of dimension 2, already passed through the activation function) into every row of a new, empty matrix (3x2), then component-wise multiply its every column by the fwdprop output of your 3-node hidden layer. This is your GradForWeights matrix

Section B

The gradient wrt hidden nodes will be a 3d vector.

To make it, use dot product between your Weights (3x2) and the incoming gradient vector (2d).  The resulting vector will be of dimension 3, and can also be used as the "incoming gradient vector" for the layer below (of course, after you pass it through the activation function of that sub-layer).


How to find relation between N components and predict the value of any one component using the predicted relation?

I understand your task can be formulated as an imputation task, see the link below for further details. Also before you proceed look up the concepts of missing data.

In order to fill in missing data you’ll have to build N distinct models using available observations. Each of N models will have N-1 inputs.

Note that because more than one component can be missing your models should be able to handle the missing data. According to my knowledge neural networks cannot handle the missing data out of the box. Tree based models, such that random forests or GBMs are more suitable for this kind of tasks.

http://www.stat.columbia.edu/~gelman/arm/missing.pdf


the relationship between the number of filters/kernels and the number of feature maps

In the linked tutorial, each kernel is 2-dimensional, and applied to a single channel/feature map from its input. To construct a feature map, the output from $N_{input\_channels}$ convolutions are added together - this is important as it allows to build feature maps that have interactions between feature maps in the previous layer.

From the diagram, the first input layer has 1 channel (a greyscale image), so each kernel in layer 1 will generate a feature map. However, once you have 64 channels in layer 2, then to produce each feature map in layer 3 will require 64 kernels added together. If you want 256 feature maps in layer 3, and you expect all 64 inputs to affect each one, then you usually need 64 * 256 = 16384 kernels. The value 4096 is coming from some other aspect of the architecture not shown in the diagram, such as dividing the feature map into groups so that each output layer only processes a fraction of the input layers.

There are a few notation and presentation differences between presentations and tutorials on convolutional networks, depending on the source. This is one of them. Other sources may arrange the kernels into a 4D structure: $N_{input\_channels} \times N_{output\_channels} \times K_{width} \times K_{height}$ to make the relationship between kernels and input/output more explicit.

Another way to show the same thing is to treat the kernels as 3D, and ensure that the kernel depth $K_{depth}$ is the same as the number of input channels $N_{input\_channels}$ - this is same thing mathematically as summing up multiple separate convolutions, and dimensions might be $N_{output\_channels} \times K_{width} \times K_{height} \times K_{depth}$


K-means: What are some good ways to choose an efficient set of initial centroids?

An approach that yields more consistent results is K-means++.  This approach acknowledges that there is probably a better choice of initial centroid locations than simple random assignment. Specifically, K-means tends to perform better when centroids are seeded in such a way that doesn't clump them together in space.

In short, the method is as follows:

Choose one of your data points at random as an initial centroid.
Calculate $D(x)$, the distance between your initial centroid and all other data points, $x$.
Choose your next centroid from the remaining datapoints with probability proportional to $D(x)^2$
Repeat until all centroids have been assigned.

Note: $D(x)$ should be updated as more centroids are added.  It should be set to be the distance between a data point and the nearest centroid.

You may also be interested to read this paper that proposes the method and describes its overall expected performance.


nazz's answer doesn't work in all cases and is not a standard way of doing the scaling you try to perform (there are an infinite number of possible ways to scale to [-1,1] ). I assume you want to scale each column separately:

1) you should divide by the absolute maximum:

arr = arr - arr.mean(axis=0)
arr = arr / np.abs(arr).max(axis=0)

2) But if the maximum of one column is 0 (which happens when the column if full of zeros) you'll get an error (you can't divide by 0).

arr = arr - arr.mean(axis=0)
safe_max = np.abs(arr).max(axis=0)
safe_max[safe_max==0] = 1
arr = arr / safe_max

Still, this is not the standard way to do this. You're trying to do some "Feature Scaling" see here

Then the formula is:

import numpy as np

def scale(X, x_min, x_max):
    nom = (X-X.min(axis=0))*(x_max-x_min)
    denom = X.max(axis=0) - X.min(axis=0)
    denom[denom==0] = 1
    return x_min + nom/denom

X = np.array([
    [ 0,  1],
    [ 2,  3],
    [ 4,  5],
    [ 6,  7],
    [ 8,  9],
    [10, 11],
    [12, 13],
    [14, 15]
])
X_scaled = scale(X, -1, 1)
print(X_scaled)

Result:

[[-1.         -1.        ]
 [-0.71428571 -0.71428571]
 [-0.42857143 -0.42857143]
 [-0.14285714 -0.14285714]
 [ 0.14285714  0.14285714]
 [ 0.42857143  0.42857143]
 [ 0.71428571  0.71428571]
 [ 1.          1.        ]]

If you want to scale the entire matrix (not column wise), then remove the axis=0 and change the lines denom[denom==0] = 1 for denom = denom + (denom is 0).


Counting the occurrence of each string in a pandas dataframe column

Something Equivalent to this should work:

the function(pandas.Series.str.count) in this link will even make it easier...

You can use split by regex :

#using your sample
df[['class1', 'class2', 'class3',..]] = df['genre'].str.split('|', expand=True)

Alternative to the above method (but iterating the dataframe)

  l = list(df[index-key].values)
  l = (",".join(l))
  l.split('|')

And if need remove column genre add drop:

df = df.drop('genre', axis=1)

And then you can use value_counts()

But this assumes that you have same length of each genre or apply a check first and proceed accordingly..

Or

You can also try using Counter from the collections module(after splitting them, just update each key's value)

example Counter dict

>>> from collections import Counter
>>> Counter(['apple','red','apple','red','red','pear'])
Counter({'red': 3, 'apple': 2, 'pear': 1})


Transformer model: Why are word embeddings scaled before adding positional encodings?

This is specified in the original Transformer paper, at the end of section 3.4:

Transcription:

3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension model. We also use the usual learned linear transformation and softmax function to convert the decoder output to predicted next-token probabilities. In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [24]. In the embedding layers, we multiply those weights by √model

This aspect is not justified by the authors, either on the paper or anywhere else. It was specifically asked as an issue in the original implementation by Google with no response.
Other implementations of the Transformer have also wondered if this was actually needed (see this, this and this).
Some hypothesithed arguments (source) are:

It is for the sharing weight between the decoder embedding and the decoder pre-softmax linear weights.
It is not actually needed.
It is to make the positional encoding relatively smaller. This means the original meaning in the embedding vector won’t be lost when we add them together.

For reference, there are other StackExchange questions discussing this (see this and this).


how to access weights of individual Neurons in the output layers in MLPs?

I'm guessing you want something like this:

model.layers[-1].get_weights()


Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

For data load, Postgre outperforms MongoDB.
MongoDB is almost always faster when returning query counts.
PostgreSQL is almost always faster for queries using indexes.

Check out this website  and this one too for more info. They have very detailed explanations.


Bigdata cluster compatible distributed predictive model

Spark does have a pretty good Python API, check out this tutorial.

For traditional Hadoop stack, take a look at mrjob, it lets you write MapReduce jobs in Python and run them on several platforms.


Minimum number of trees for Random Forest classifier

This is not necessarily an answer to your question. Just general thoughts about cross-validating the number of decision trees within a random forest.

I see a lot of people in kaggle and stackexchange cross-validating the number of trees in a random forest. I have also asked a couple of colleagues and they tell me it is important to cross-validate them to avoid overfitting.

This never made sense to me. Since each decision tree is trained independently, adding more decision trees should just make your ensemble more and more robust.

(This is different from gradient boosting trees, which are a particular case of ada boosting, and therefore there is potential for overfitting since each decision tree is trained to weight residuals more heavily.)

I did a simple experiment:

from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier
from sklearn.grid_search import GridSearchCV
import numpy as np
import matplotlib.pyplot as plt
plt.ioff()

df = load_digits()
X = df['data']
y = df['target']

cv = GridSearchCV(
    RandomForestClassifier(max_depth=4),
    {'n_estimators': np.linspace(10, 1000, 20, dtype=int)},
    'accuracy',
    n_jobs=-1,
    refit=False,
    cv=50,
    verbose=1)
cv.fit(X, y)
scores = np.asarray([s[1] for s in cv.grid_scores_])
trees = np.asarray([s[0]['n_estimators'] for s in cv.grid_scores_])
o = np.argsort(trees)
scores = scores[o]
trees = trees[o]
plt.clf()
plt.plot(trees, scores)
plt.xlabel('n_estimators')
plt.ylabel('accuracy')
plt.savefig('trees.png')
plt.show()

I am not saying your are committing this fallacy about thinking more trees can cause overfitting. You clearly are not since you have asked for a lower bound. This is just something that has been bugging me for awhile, and I think it is important to keep in mind.

(Addendum: Elements of Statistical Learning discusses this at page 596, and is in agreement on this with me. «It is certainly true that increasing B [B=number of trees] does not cause the random forest sequence to overfit». The author does make the observation that «this limit can overfit the data». In other words, since other hyperparameters may lead to overfitting, creating a robust model does not rescue you from overfit. You have to pay attention when cross-validating your other hyperparameters.)

To answer your question, adding decision trees will always be beneficial to your ensemble. It will always make it more and more robust. But, of course, it is dubious whether the marginal 0.00000001 reduction in variance is worth the computational time.

Your question therefore, as I understand, is whether you can somehow calculate or estimate the amount of decision trees to reduce the error variance to below a certain threshold.

I very much doubt it. We do not have clear answers for many broad questions in data mining, much less specific questions like that. As Leo Breiman (the author of random forests) wrote, there are two cultures in statistical modeling, and random forests is the type of model that he says  has few assumptions, but is also very data-specific. That is why, he says, we cannot resort to hypothesis testing, we have to go with brute-force cross-validation.


What's an appropriate datastore for variable length sequence data for PyTorch consumption?

What I've opted for at the moment is packing all of the samples into a single HDF5 table buffer, and keeping a separate table with metadata that tracks each individual sequence's buffer position and length. This works, but I won't be marking this answer as correct because I'm not satisfied with it. This storage method is very poorly suited to editing, and it's vulnerable to loss if a bug were to cause the tables to become out of sync.


Splitting a neural network in 2 microservices

Yes, you can split the model into two parts after training.  Not sure, what would be the advantage, but it is possible.  High level steps :

Define and train the model
Save trained model
Create two copies of model (Layers 1 to 8 and new input layer + 9 to 16)
Transfer weights to these copies of model

After this, service 1 can run first few layers and provide the output to service 2.

Example code to split the model (Code uses Keras as wrapper over TensorFlow):

import keras
from keras.models import Model, load_model
from keras.layers import Input, Dense
from keras.optimizers import RMSprop
import numpy as np

# Create original model and save it
inputs = Input((1,))
dense_1 = Dense(10, activation='relu')(inputs)
dense_2 = Dense(10, activation='relu')(dense_1)
dense_3 = Dense(10, activation='relu')(dense_2)
outputs = Dense(10)(dense_3)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer=RMSprop(), loss='mse')
model.save('test.h5')

# Load the model and make modifications to it
loaded_model = load_model('test.h5')
loaded_model.layers.pop()
loaded_model.layers.pop()

# Create your new model with the two layers removed and transfer weights
new_model = Model(inputs=inputs, outputs=dense_1)
new_model.compile(optimizer=RMSprop(), loss='mse')
new_model.set_weights(loaded_model.get_weights())

new_model.summary()
new_model.save('test_complete.h5')

Source : https://github.com/keras-team/keras/issues/8772


Is it possible to get an ROC curve using Relu activation?

YES
The typical way to do a ROC curve is to have probability outputs, set a threshold, and calculate the sensitivity and specificity.
y | prob
----------
0 | 0.2
0 | 0.3
1 | 0.7
1 | 0.8

At a threshold of 0, we classify everyting as $1$, so sensitivity of $1$ and specificity of $0$. At a threshold of $0.25$, we get $0,1,1,1$, so sensitivity of $1$ and specificity of $0.5$. Et cetera...
Plot those $(1, 1)$, $(0.5, 1)$ points to start building your ROC curve.
This idea of setting a threshold and calculating the sensitivity and specificity does not assume probability outputs, so let's do the same with non-probability outputs.
y | output
----------
0 | 2
0 | 3
1 | 7
1 | 8

At a threshold of 0, we classify everyting as $1$, so sensitivity of $1$ and specificity of $0$. At a threshold of $2.5$, we get $0,1,1,1$, so sensitivity of $1$ and specificity of $0.5$. Et cetera...


What statistical model should I use to analyze the likelihood that a single event influenced longitudinal data

For the record, I think this is the type of question that's perfect for the data science Stack Exchange. I hope we get a bunch of real world examples of data problems and several perspectives on how best to solve them.

I would encourage you not to use p-values as they can be pretty misleading (1, 2). My approach hinges on you being able to summarize traffic on a given page before and after some intervention. What you care about is the difference in the rate before and after the intervention. That is, how does the number of hits per day change? Below, I explain a first stab approach with some simulated example data. I will then explain one potential pitfall (and what I would do about it).

First, let's think about one page before and after an intervention. Pretend the intervention increases hits per day by roughly 15%:

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

def simulate_data(true_diff=0):
    #First choose a number of days between [1, 1000] before the intervention
    num_before = np.random.randint(1, 1001)

    #Next choose a number of days between [1, 1000] after the intervention
    num_after = np.random.randint(1, 1001)

    #Next choose a rate for before the intervention. How many views per day on average?
    rate_before = np.random.randint(50, 151)

    #The intervention causes a `true_diff` increase on average (but is also random)
    rate_after = np.random.normal(1 + true_diff, .1) * rate_before

    #Simulate viewers per day:
    vpd_before = np.random.poisson(rate_before, size=num_before)
    vpd_after = np.random.poisson(rate_after, size=num_after)

    return vpd_before, vpd_after

vpd_before, vpd_after = simulate_data(.15)

plt.hist(vpd_before, histtype="step", bins=20, normed=True, lw=2)
plt.hist(vpd_after, histtype="step", bins=20, normed=True, lw=2)
plt.legend(("before", "after"))
plt.title("Views per day before and after intervention")
plt.xlabel("Views per day")
plt.ylabel("Frequency")
plt.show()

We can clearly see that the intervention increased the number of hits per day, on average. But in order to quantify the difference in rates, we should use one company's intervention for multiple pages. Since the underlying rate will be different for each page, we should compute the percent change in rate (again, the rate here is hits per day).

Now, let's pretend we have data for n = 100 pages, each of which received an intervention from the same company. To get the percent difference we take (mean(hits per day before) - mean(hits per day after)) / mean(hits per day before):

n = 100

pct_diff = np.zeros(n)

for i in xrange(n):
    vpd_before, vpd_after = simulate_data(.15)
    # % difference. Note: this is the thing we want to infer
    pct_diff[i] = (vpd_after.mean() - vpd_before.mean()) / vpd_before.mean()

plt.hist(pct_diff)
plt.title("Distribution of percent change")
plt.xlabel("Percent change")
plt.ylabel("Frequency")
plt.show()

Now we have the distribution of our parameter of interest! We can query this result in different ways. For example, we might want to know the mode, or (approximation of) the most likely value for this percent change:

def mode_continuous(x, num_bins=None):
    if num_bins is None:
        counts, bins = np.histogram(x)
    else:
        counts, bins = np.histogram(x, bins=num_bins)

    ndx = np.argmax(counts)
    return bins[ndx:(ndx+1)].mean()

mode_continuous(pct_diff, 20)

When I ran this I got 0.126, which is not bad, considering our true percent change is 15. We can also see the number of positive changes, which approximates the probability that a given company's intervention improves hits per day:

(pct_diff > 0).mean()

Here, my result is 0.93, so we could say there's a pretty good chance that this company is effective.

Finally, a potential pitfall: Each page probably has some underlying trend that you should probably account for. That is, even without the intervention, hits per day may increase. To account for this, I would estimate a simple linear regression where the outcome variable is hits per day and the independent variable is day (start at day=0 and simply increment for all the days in your sample). Then subtract the estimate, y_hat, from each number of hits per day to de-trend your data. Then you can do the above procedure and be confident that a positive percent difference is not due to the underlying trend. Of course, the trend may not be linear, so use discretion! Good luck!


Adding a column to a dataframe in pandas using another Column

Iterrow passes a copy of the row, not the reference. This should fix your problem:

df.loc[index,'Keywords'] = str(plot)

However, I would recommend using apply, imho it is more elegant. And it is alot faster.

That would looks something like this

def string_to_keywords(string):
    plot = re.sub('[^a-zA-Z]'," ", string)
    plot = plot.lower()
    plot = plot.split()
    return " ".join([i for i in plot if not i in set(stopwords.words('english'))])

df["Keywords"] = df["Plot"].apply(string_to_keywords)


Format of data in SQL for machine learning

Disclaimer: I'm not at all expert about deploying big ML systems in production. This answer is only based on my experience with many different ML problems and datasets

My humble advice would be not to try to design the format of the data before having a quite precise idea of what kind of ML process is going to be applied. There is no "one size fits all" in ML and there's a real risk that by starting with the format of the data, you will end up with something which turns out to be completely inappropriate for the task.

Start with local experiments instead:

Use a small subset of the data at first.
Design some simple problems of the same kind as the real ones you plan to do eventually.
Vary as many aspects as possible of the experiments: preprocessing, learning algorithms, parameters, size of the data, etc. Move progressively to more realistic tasks and amount of data.
Evaluate the advantages/disadvantage of the different methods/setups, then select a range of target setups
Finally design everything including the data format based on these target setups.

Following this logic during the experimental stage you can just export your data in any format convenient for whatever framework you're testing. it's only at the end of the experimental stage that you design the production system, e.g. SQL server etc.


You can use two methods: one that takes a formula argument or if you are not using formula you can use model.matrix.

You can do the following:

# Using the same data from the question:
data <- data.table(A = c(0, 2, 4),
                   B = c(1, 3, 5))

> data

   A B
1: 0 1
2: 2 3
3: 4 5

If only the raw data and the interactions are required

formula = y ~ .^2

model.matrix(formula, data=data)
  (Intercept) A B A:B
1           1 0 1   0
2           1 2 3   6
3           1 4 5  20

If raw data, squared columns, interactions and intercept are needed, just like the question

You can use the following formula for all variables:

formula = y ~ .^2 + poly(var, 2, raw=TRUE)[, 2] ... etc

In addition you can paste the variables names automatically. Formula based on this post

formula <- as.formula(paste(' ~ .^2 + ',paste('poly(',colnames(data),',2, raw=TRUE)[, 2]',collapse = ' + ')))

> formula

~.^2 + poly(A, 2, raw = TRUE)[, 2] + poly(B, 2, raw = TRUE)[, 2]

> model.matrix(formula, data=data)

  (Intercept) A B poly(A, 2, raw = TRUE)[, 2] poly(B, 2, raw = TRUE)[, 2] A:B
1           1 0 1                           0                           1   0
2           1 2 3                           4                           9   6
3           1 4 5                          16                          25  20

The resulting dataframe is identical to the array in the question.

If squared columns, interactions and intercept are needed.

> formula <- as.formula(paste(' ~ A:B + ',paste('poly(',colnames(data),',2, raw=TRUE)[, 2]',collapse = ' + ')))

> formula
~A:B + poly(A, 2, raw = TRUE)[, 2] + poly(B, 2, raw = TRUE)[, 2]

>  model.matrix(formula, data=data)
  (Intercept) poly(A, 2, raw = TRUE)[, 2] poly(B, 2, raw = TRUE)[, 2] A:B
1           1                           0                           1   0
2           1                           4                           9   6
3           1                          16                          25  20


How to choose the good number dimension of autoencoder?

Treat it as a hyperparameter, and do a hyperparameter search over it. This requires having a good metric for your model performance. Which usually means having a validation set with labels. Training can still be unsupervised.


What does coefficient mean for a binary independent variable in multiple linear regression?

Your question is a little bit confusing but I will try to answer it:

You are asking if the coefficient sign equals the correlation sign? You want to know if a positive sign in the regression leads to a positive correlation between the $Y$ and the binary $X$s.

Depends.

This is true if you are running a regression with only one THAT regressor.

The equation $Retweets=\beta_0+\beta_1*truth.label$ complies with this.

The model $Retweets=\beta_0+\beta_1*truth.label.1+\beta_2*truth.label.2$ does not necesarily complies with the statement.

Why? When you have two or more variables in a model, what they do to each other is to complement between themselves, i.e. one finds what the other variable misses (in terms of errors). So you can't state anything about coefficients and correlations.

This is true, does not matter the type of variable: Binary, Integer, Continuos.


I'm not certain on your first question; the paper does seem poorly written in this part.  They seem to be arguing that taking just $E(h(x))/n$ or $E(h(x))/\log n$ are inappropriate.  The former tends to zero regardless of $x$, but the latter should be reasonable.  Indeed, since $c(n)\sim 2\log n$, it's not far off from what they end up doing.  (Exponentiation pushes the scores into the range $[0,1]$ which is nice, and using the more exact $c(n)$ calibrates things better for "(c) if all the instances return $s\approx 0.5$, then the entire sample does not really have any distinct anomaly.")

The rest of the confusion seems to be stemming from what set we're averaging over.  In the isolation forest paper, the averaging is of the depths of (the leaf containing) $x$ over multiple trees in the isolation forest.  In the other reference, the average is of the height of all binary trees on $n$ nodes.