Can someone explain what batch size is doing in convolutional NNs?

Batch size determines how often the weights are updated during training. The smaller the batch size, the more frequent the updates. On the other hand, when the batch size is small, updates are made without evaluating a large portion of the data at hand so at times, the gradient may be moving in a direction other than minimum if that particular batch is not a good representative of the entire dataset. So in a sense, a large batch size helps you get to a lower loss faster compared to a smaller one - it takes relatively larger steps with little noise. In practice, one rarely uses the entire dataset because it takes too much time per iteration. Of course for smaller dataset, this is not a problem.

Here you can find a good discussion of this issue by Andrew Ng.

Having said that, I also vaguely remember a paper discouraging from using too large batches (more than 1024). Smaller batches might wiggle the loss left and right too much before reaching a minimum, but they will get closer to it. And also some discussion about training with very large batches not performing as well on validation sets. So there is a trade-off and the optimal value should be picked by trial and error.


Binary classification algorithm where the input variables are arrays

A very simple approach is to build a detector with a sliding window on which you compute standard deviation. For each slide compute the mean and you can test if the next point after the window is far from mean more than a number of standard deviations. This should work because if the window is not very large it will adapt to the local mean and also if the functions are relatively constant on windows and spikes are large, as they look from your graphs


Segmenting pandas dataframe with lists as elements

Are you concerned that a column might have too many values and that one hot encoding will produce features with less predictive power? In that case you could select a handful of the features for more prevalent class values and keep them as is, while combining the other features for rarer class values into one feature.


Who wrote the formula for gini importance/sklearn's feature importance score?

There is a paper that covers "The origins of the Gini index". Gini index was detailed by Leo Breiman et al. in "Classification and regression trees" book in 1984. Leo Breiman also wrote a seminal paper on Random Forests in 2001 which includes the notion of feature importance.


I think you are looking for K-prototype algorithm


Output prediction from a sequence of data

Have a look on conditional random field. Conditional random fields  are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is their conditional nature, resulting in the relaxation of the independence assumptions. Additionally, CRFs avoid the label bias problem. It is often used for labeling or parsing of sequential data.


Why is finite precision a problem in machine learning?

Finite precision is decimal representation of a number which has been rounded or truncated. There many cases where this may be necessary or appropriate. For example 1/3 and the transcendental numbers $e$ and $\pi$ all have infinite decimal representations. In the programming language C, a double value is 8 bit and precise to approximately 16 digits. See here.

http://www.learncpp.com/cpp-tutorial/25-floating-point-numbers/

To concretely represent one of these numbers on a (finite) computer there must be some sort of compromise. We could write 1/3 to 9 digits as .333333333 which is less than 1/3.

These compromises are compounded with arithmetic operations. Unstable algorithms are prone to arithmetic errors. This is why SVD is often used to compute PCA (instability of the covariance matrix).

http://www.sandia.gov/~smartin/presentations/SMartin_Stability.pdf

https://en.wikipedia.org/wiki/Numerical_stability

In the naive bayes classifier you will often see multiplication transformed into a sum of logarithms, which is less prone to rounding errors.

https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes


Naive Bayes / SVM classifiation - min. number of records (Python)

My suspicion is that you did not put random_state parameter on train_test_split. Don't worry it is quite common error, it just happen that your dataset is small and often with small dataset variability between result will be heavily amplified. Having small amount of data is OK, and sometimes we have to work with what we have, as long as you know how to handle it, just don't expect your model to generalize very well.

Your model relies on bag of words representation I presume. With small data what could go wrong is having a lot of Out of Vocabulary words (words in test-set does not available in training set), to handle this you can apply smoothing e.g. Add-one smoothing.


What is the proper train data format in LSTM?

[sentence_len, embedding_size] to [sentence_len, step_len, embedding_size] (step_len means present a word with it surround words).

Use step_len=1, just the current word. after squeezing the tensor, its shape will be  [batch_size,sentence_len,embedding_size].

  Previous sentence's last word may be affected by latter sentence's first few words.

Use padding, append the last 3 to 5 words of previous sentences at the start of the current sentences, do not backpropagate for the errors of those padded words, by masking their loss. as for the case of first sentences of paragraphs, padded with a simple <pad> token.

  doesn't need BiLSTM anymore? I think it still helps here.

Using BiLSTM is a must. if you want to limit step_len to be 1, by the way, you are computing the conditional probability of Y being true (correct/not correct) conditioned on the whole context (sentence). as you are feeding the sentence at once, the loss will be the summation over all single losses.

Side note:
To batch the training examples, you also need to pad the end of sentences with the three to five words of the next ones. use <pad> for sentences ending the paragraph, don't permit their contributions to the model (adjusting the weight), mask your padded examples to have zero loss. because their labels are just noise.


How do you generalize from stochastic to batch learning using gradient descent?

Do you average them or something ?

Yes! The gradient of the loss for the minibatch is just a simple average of the gradients of the individual examples in the minibatch. See Section 8.1.3 of the Deep Learning Book:

  Hence, we can obtain an unbiased estimator of the exact gradient of the generalization error by sampling a minibatch of examples $\{x(1),...x(m)\}$
  with corresponding targets $y(i)$ from the data-generating
  distribution $p_{data}$, then computing the gradient of the loss with respect
  to the parameters for that minibatch:
  $$\hat{g} = \frac{1}{m} ∇_\theta \sum_{i} L(f(x^{(i)}; \theta), y^{(i)}).$$


Volume:
Simply stated, big data is to big to work on one computer. This is a relative definition, as what can't work on today's computer will easily work on computers in the future.
 - One Google search uses the computing power of the entire Apollo space mission.
 - Excel used to hold up to 65k rows in a single spreadsheet. Now it holds over a million.

Velocity:
Data is coming in extremely fast. Traditional scientific research methods of a few hundred cases could take weeks, months or even years to analyze and publish.
 - Iris flower data set
 - Statistical Programming Language R
 - Twitter Firehose (6,000 tweets per second)

Variety:
Big Data that is contained in one specific data type or does not fit well within the format of a relational database. This data often comes in the form of unstructured text.
 - Estimated 80% of all enterprise data is unstructured
 - Open Data(Government)
 - noSql Databases

Iris Data Set: https://en.wikipedia.org/wiki/Iris_flower_data_set
Open Data: https://www.data.gov/open-gov/


Big problem and very good question!

I used spacy in the past, which has a German module. I guess stemming is not supported, but lemmatization.

Looking at the output below, I don't think that spacy will solve your problem to be honest. However, I just wanted to let you know about this option.

Spacy Lemmatization:

#pip install spacy
#python -m spacy download de

import spacy
nlp = spacy.load('de_core_news_sm')

mywords = "Das ist schon sehr schön mit den Expertinnen und Experten"

for t in nlp.tokenizer(mywords):
    print("Tokenized: %s | Lemma: %s" %(t, t.lemma_))

Result:

Tokenized: Das | Lemma: der
Tokenized: ist | Lemma: sein
Tokenized: schon | Lemma: schon
Tokenized: sehr | Lemma: sehr
Tokenized: schön | Lemma: schön
Tokenized: mit | Lemma: mit
Tokenized: den | Lemma: der
Tokenized: Expertinnen | Lemma: Expertinnen
Tokenized: und | Lemma: und
Tokenized: Experten | Lemma: Experte


Inception Score (IS) and Fréchet Inception Distance (FID), which one is better for GAN evaluation?

Here is the original paper proposing FID.
Here is an excerpt from
Jason Brownlee's https://machinelearningmastery.com/how-to-implement-the-frechet-inception-distance-fid-from-scratch/ along with a couple of quotes from the paper:
The inception score estimates the quality of a collection of synthetic images based on how well the top-performing image classification model Inception v3 classifies them as one of 1,000 known objects. The scores combine both the confidence of the conditional class predictions for each synthetic image (quality) and the integral of the marginal probability of the predicted classes (diversity).
The inception score does not capture how synthetic images compare to real images. The goal in developing the FID score was to evaluate synthetic images based on the statistics of a collection of synthetic images compared to the statistics of a collection of real images from the target domain.

For the evaluation of the performance of GANs at image generation, we introduce the “Frechet Inception Distance” (FID) which captures the
similarity of generated images to real ones better than the Inception
Score.

Like the inception score, the FID score uses the inception v3 model. Specifically, the coding layer of the model (the last pooling layer prior to the output classification of images) is used to capture computer-vision-specific features of an input image. These activations are calculated for a collection of real and generated images.
The activations are summarized as a multivariate Gaussian by calculating the mean and covariance of the images. These statistics are then calculated for the activations across the collection of real and generated images.
The distance between these two distributions is then calculated using the Frechet distance, also called the Wasserstein-2 distance.

The difference of two Gaussians (synthetic and real-world images) is measured by the Frechet distance also known as Wasserstein-2
distance.


What are some data sources with strong positive and negative correlations in feature and sample space?

A couple of spatio-temporal datasets are likely to exhibit this characteristic:

Transportation demand across different locations of a city over different hours of a day. While residential locations have high demand originating there in the morning hours, and low demand in the evening, the reverse is true of work locations. The demand pattern of these two location types will be negatively correlated, whereas that of two different residential locations (or two locations of the same type) will be positively correlated. From the point of view of hours, correlation between two morning hours will be positive, whereas between a morning and an evening hour will be negative.
Temperature pattern across days, for different cities across the world. Here, data for cities in the northern and southern hemisphere will be negatively correlated.


Public datasets that show "cyclical" behavior

The search term you are looking for is limit cycle. You can find some examples (legged-locomotion, digital control systems, etc.) here: https://www.sciencedirect.com/topics/engineering/limit-cycle.


How to train or approach the image datasets with different resolution in Deep Learning

You don't need to change their size. You can zero pad the images when you feed them into the Network. Zero padding creates a "frame of zeros" around each image, so that they all take equal shape.

conv layers in Keras / TensorFlow 2.0 come with already built-in zero padding arguments.


There has been some research in building very specialized deep learning models to use complex numbers using complex algebra, but these tools are not yet user-friendly. So unless you want to go down the rabbit hole and try to understand their implementations and then try to write your own implementation, i would avoid this.

In the past, I worked with MRI data which is a 3D complex-valued matrix. I inspired myself from the work done in color images, where 3 channels are used to represent red (R), green (G) and blue (B). I this separated the real and imaginary part, and treated them as separate channels into the deep learning model all as real numbers.

1D example:

From a complex vector, we get 2 channels where the imaginary part can then be treated as a real number.

$[1+1j, 2+2j, 3+3j] = [1, 2, 3], [1, 2, 3]$


How to combine different kernels for Gaussian process in GPyTorch?

From official documentation: For example, to compose two kernels via addition, you can either add the kernel modules directly:

self.covar_module = ScaleKernel(RBFKernel() + WhiteNoiseKernel())

Or you can add the outputs of the kernel in the forward method:

covar_x = self.rbf_kernel_module(x) + self.white_noise_module(x)

The class for GPRegressionModel would then look like this:

class GPRegressionModel(gpytorch.models.ExactGP):
def __init__(self, train_x, train_y, likelihood):
    super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)

    self.mean_module = gpytorch.means.ConstantMean()
    self.covar_module = ScaleKernel(RBFKernel() + WhiteNoiseKernel())

def forward(self, x):
    mean_x = self.mean_module(x)
    covar_x = self.covar_module(x)
    return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)


Can a decision in a node of a decision tree be based on comparison between 2 columns of the dataset?

Yes, but not in any implementation that I am aware of.

The idea is mentioned in Elements of Statistical Learning, near the end of section 9.2.4 under the heading "Linear Combination Splits."  But this is not implemented in the popular CART or Quinlan-family of decision tree algorithms*, it is not done in sklearn's trees, and I do not know of any other python or R package that use it.

Some R packages do a more limited version, where a split can be made on two features, but these splits are of the form "$x_1>\alpha\text{ and }x_2\leq\beta$" as opposed to direct comparisons of the variables.  See https://stats.stackexchange.com/questions/4356/does-rpart-use-multivariate-splits-by-default

An obvious problem is computational requirements: just checking over all pairs of features is now quadratic, and allowing arbitrary linear combinations of two features is potentially much much larger.  On the other hand, if you want to restrict to direct comparisons $x_1\geq x_2$ (with no coefficients), that should be tractable (if substantially slower than CART).  The Elements authors suggests Hierarchical Mixtures of Experts model instead if incorporating linear combinations is desired.

Oh, one more comment.  If you really want splits like $x_1\geq x_2$, you could just generate all the features $x_i-x_j$; then a more common implementation of decision trees will be able to make your splits, when considering these new features.  (Probably there will be some side effects, and still the computational problem arises: you've added $\binom{m}{2}$ features.)

* I've found a comment that suggests that CART does support multi-feature ("surrogate") splits?:
https://stackoverflow.com/a/9996741/10495893


I want to create an additional feature(column) based on some manipulation of values from existing features

Welcome to the community!

The code below is a starter. You can go on by naming the column and adding it to the original DataFrame:

import pandas as pd

data = pd.DataFrame({'x':[1,2,3], 'y':[10,20,30],'z':[100,200,300]})
print(data)
def my_fun(data,expression,variables):
    for v in variables:
        expression = expression.replace(v,'data.'+str(v))
    return eval(expression)

my_fun(data,'2*x+y',['x','y'])

output

   x   y    z
0  1  10  100
1  2  20  200
2  3  30  300

0    12
1    24
2    36
dtype: int64

There are two ways in general:

Use eval function to evaluate/execute expression in string form as I did.
Use symbolic libraries, most commonly used is SymPy, to use symbolics directly as variables.

Hope it helps. Good Luck!


What is the best approach for detecting the defect?

It is a possible way, but this class of problems falls under outlier detection in machine learning.
The defining feature is that one can only train for usual events not rare events, there are few or even no data at all, so a different approach needs to be taken.
Since usual neural network training requires adequate samples from each class, a workaround is needed. For example autoencoder networks are an option for outlier detection


Can also write a geometric mean function by a little help  from Numpy.
import numpy as np

def geometric_mean(i):
    x = np.array(i)
    return x.prod()**(1.0/len(x))


That's a great question, because on its face it seems like the weight should be $\beta$ alone, and, it should be in front of recall. The answer is in the text from which that reference is taken, on page 133: http://www.dcs.gla.ac.uk/Keith/pdf/Chapter7.pdf

The definition is designed to make the metric indifferent to a change in precision or recall when $P/R = \beta$. That is, $F_\beta$ increases by the same amount when either precision or recall increases, at the point where precision is already $\beta$ times bigger than recall.

The definition does indeed weight recall more highly as you can verify. Honestly on re-reading the text above, I was confused, because I don't see how it makes sense to think of "equilibrium" as the point where precision is much bigger, if recall matters more.

I plugged in the formula to Wolfram Alpha, and:

Hm. These are only equal if $R/P = \beta$! I think the paper may have misstated this then, or else I've really missed something. It's a formula whose value changes at the same rate with respect to precision or recall, when recall is already $\beta$ times larger, and in that sense it corresponds to treating recall as $\beta$ time more important.


Understanding a Neural Network with Keras (preferably), TensorFlow or PyTorch

From Keras Issue #41:

  One simple way to do it, is to use the weights of your model to build a new model that's truncated at the layer you want to read. Then you can run the ._predict(X_batch) method to get the activations for a batch of inputs.


Interpreting Loss in neural network: Neural network train loss gradually tappers and validation loss never reaches a minima

this is a case of overfitting: your model performs very good just after few epochs but only in the training set, while on the validation set it improves too slowly.

There are several tricks that you can use:

1) a simpler network: you can lower the dimensions of the embeddig layers, as example. I also see some of the embeddings you are using can be avoided: the case of the month_emb, which I suppose stands for "month", and with have a dimensionality of 12. Is much better to use months as a simple number, or a categorical feature.

2) got more data: this is the hardest and often impossible way to proceed, but often is the best one.

3) reduce the number of inputs, by transforming/joining some of them.


Predicting next number in a sequence - data analysis

Interesting problem. The pandas autocorrelation plot suggests that the data is random.

How much do you know about the source?  Is it believable that the sequence is, in fact, random?

Have you gone through the data and plotted a histogram of the integer appearance counts? Do they appear uniformly or are some more frequent than others?

One thing I think you should try is to reformulate your LSTM model. I don't think this is a regression problem, even though the target is a single integer. It's a classification problem with 70 classes. Try treating it that way and using a categorical crossentropy loss function. In such a scenario, ~1.5% accuracy would represent random guessing. Can the model do better than that?


GP is also known as genetic programming. It is an algorithm which has been inspired by natural selection (survival of the fittest) to find an ideal algorithm to perform some task. These algorithms create individuals at each generation which have an encoded behaviour as a set of genes. The individuals which best perform the task at each generation are selected to be permutated slightly and progress into the next generation. This causes the solution space in later generations to narrow around some local minimum.

Example

Let's look at a simple example to see how this can be used for finding the minimum of a 2D function from -100 to 100.

This consists in 4 crucial steps: initialization, evaluation, selection and combination.

Initialization

Each individual in the population is encoded by some genes. In our case the genes represent our $[x, y]$ values. We will then set our search range to [0, 1000] for this specific problem. Usually you will know what is naturally possible based on your problem. For example, you should know the range of possible soil densities in nature. We will create 100 individuals in our population.

Evaluation of the fitness

This step simply asks you to put the $[x,y]$ values into your function and get its result. Pretty standard stuff.

Selection

There are many ways with which you can select parents. I will always keep the alpha male. The best individual in the population, he will be cloned to the next. Then I will use tournament selection. We will repeat the following until the next generation population is full. Pick four parents at random, take the best individual from the first two and the best from the last two. These will be the two parents which will gives us our next offspring.

Combination

From the two parents we will build the new genome for the child using the binary values of the $[x,y]$ values of the parents. The resulting binary value for each codon in the genome of the child is selected from the two parent genes by uniform random.

import numpy as np

class Genetic(object):

    def __init__(self, f, pop_size = 1000, n_variables = 2):
        self.f = f
        self.minim = -100
        self.maxim = 100
        self.pop_size = pop_size
        self.n_variables = n_variables
        self.population = self.initializePopulation()
        self.evaluatePopulation()

    def initializePopulation(self):
        return [np.random.randint(self.minim, self.maxim, size=(self.n_variables))
                           for i in range(self.pop_size)]

    def evaluatePopulation(self):
        return [self.f(i[0], i[1]) for i in self.population]

    def nextGen(self):
        results = self.evaluatePopulation()
        children = [self.population[np.argmin(results)]]

        while len(children) < self.pop_size:
            # Tournament selection
            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p1 = self.population[randA]
            else: p1 = self.population[randB]

            randA, randB = np.random.randint(0, self.pop_size), \
                           np.random.randint(0, self.pop_size)
            if results[randA] < results[randB]: p2 = self.population[randA]
            else: p2 = self.population[randB]

            signs = []
            for i in zip(p1, p2):
                if i[0] < 0 and i[1] < 0: signs.append(-1)
                elif i[0] >= 0 and i[1] >= 0: signs.append(1)
                else: signs.append(np.random.choice([-1,1]))

            # Convert values to binary
            p1 = [format(abs(i), '010b') for i in p1]
            p2 = [format(abs(i), '010b') for i in p2]

            # Recombination
            child = []
            for i, j in zip(p1, p2):
                for k, l in zip(i, j):
                    if k == l: child.append(k)
                    else: child.append(str(np.random.randint(min(k, l),
                                                             max(k,l))))

            child = ''.join(child)
            g1 = child[0:len(child)//2]
            g2 = child[len(child)//2:len(child)]
            children.append(np.asarray([signs[0]*int(g1, 2),
                                        signs[1]*int(g2, 2)]))
        self.population = children

    def run(self):
        ix = 0
        while ix < 1000:
            ix += 1
            self.nextGen()
        return self.population[0]

To use this algorithm you can define some functions and then run the following.

f = lambda x, y: (x-7)**2 + y**2
gen = Genetic(f)
minim = gen.run()

print('Minimum found      X =', minim[0], ', Y =', minim[1])


Depending on what language you are using you can just print the model. In R it looks like this:

summary(my.model)

The output will look like this, or similar:

##
## Call:
## lm(formula = dist ~ speed.c, data = cars)
##
## Residuals:
##     Min      1Q  Median      3Q     Max
## -29.069  -9.525  -2.272   9.215  43.201
##
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)
## (Intercept)  42.9800     2.1750  19.761  < 2e-16 ***
## YearsExp     3.9324     0.4155   9.464 1.49e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 15.38 on 48 degrees of freedom
## Multiple R-squared:  0.6511, Adjusted R-squared:  0.6438
## F-statistic: 89.57 on 1 and 48 DF,  p-value: 1.49e-12

Your betas are under the "Estimate Std." Column. Your beta0 is (Intercept) and your beta1 is YearsExp (or whatever your variable is) etc... If you have more than one variable there will be more in this column for you to see.

After you get the betas you can write a function to apply your model to new data like this:

y-hat <- 42.9800 + YearsExp*3.9.324

Model summary copypastad w/ variable name edit from: https://feliperego.github.io/blog/2015/10/23/Interpreting-Model-Output-In-R


Your description corresponds to the general problem of grammatical inference (or grammar induction): as opposed to statistical models, the goal is to learn the exact language represented as a formal grammar. Importantly this is symbolic learning, i.e. there is no concept of minimizing error because errors are not acceptable at all.
As far as I know the field is rather theoretical, as opposed to statistical ML: one formally proves that some classes of languages can or cannot be learnt under specific constraints. A type of learning framework which used to be studied in this context (not sure if it's still active) is language identification in the limit. A quite different framework is Probably approximately correct learning (PAC), but in this is more general and not strictly about grammatical inference (to the best of my knowledge).
So I'm not aware of any real dataset: in general arbitrary datasets don't have the kind of formal constraints which are relevant for this kind of problem. But it's possible that some parts of the literature contain at least artificial datasets.


i need a characters /letters dataset for matlab

You can find car license plat Here  or you can collect them from this web site but it will take you  a while.


What is disadvantage of using CART in regression problems?

Regression and classification trees are nearly identical in how they function. I'm not aware of any specific downside to regression trees specifically. However, trees in general do have the downside of being susceptible to overfitting:

  The main downside of decision trees is that even with the use of pre-pruning, they
  tend to overfit and provide poor generalization performance. Therefore, in most
  applications,...ensemble methods...are usually used in place of a single decision tree.

Muller & Guido, Introduction to Machine Learning with Python, 2017.


The offset parameter is sometimes called "bias" in classification tasks, and its intuitive understanding doesn't have to do with what kind of kernel is used. It is basically used to compensate for feature vectors that are not centered around 0.

I will try to intuitively explain what the bias does with a toy example. Let's say you have some feature vector x who's parameters are always negative. The set of weights w you use in your SVM (let's say linear, just for clarity) will perhaps transform the features into the range [0, 1] - so they will always be negative. But, those elements that belong to class 1 fall in the range [0, 0.5], and the ones that are from class 2 fall into the range [0.5, 1].

To classify into a class, the SVM uses 0 as the threshold breakpoint - if greater than 0, it is an element of class 1, if less than 0, it is an element of class -1. But in this case, all the elements will be classified into class 2. However, with a bias of 0.5 (in this linear case), they will be classified correctly.

This geometric interpretation doesn't quite work for more complex kernels, but the idea is the same: the bias term attempts to compensate for features that are not centered around zero. In practice, if the features are centered around zero the bias term isn't always needed. To get around the bias issue, you can either:

center your features and forget about the bias term (doesn't always work)
augment your feature data to include a 1 at the beginning. i.e.:

[feat1 feat2 feat3] --> [feat1 feat2 feat3 1]

for all of your feature vectors. Then the bias term will just be learned as another SVM weight parameter.
Learn your weights, then calculate the bias based on the optimal prediction rule:

That is, given all the support vectors $S$, the learned weights $\alpha_i$, the training class $y^{(i)}$, training features $x^{(i)}$, and features-to-use-for-prediction $x$, the predicted class will be $y(x)$. The optimal $w_0$ or bias will then be calculated as the average distance between the correct and calculated predicted class:

Hope that helps. I think the easiest thing is to center your features ;)


Feature importance difference in two similar machine learning models

In 2nd case, you are not comparing apple to apple.
Let's say we have 4 Features and all are equally good [Also, No interaction].
Case I -
We created two Models using 2 Features each
Model I - F1/F2
Model II- F3/F4
Comparing these two Models as Features will give you an Idea about F1/F2-combined compared to F3/F4-combined. This is your Case - I.
Case II -
If you will compare F1/F2-combined, F3(alone), and F4(alone), definitely F1/F2 combined will have high importance.
 The output probability of a Model using two Features is basically holding the information of both the Features together.


The code below should achieve the desired results:

import pandas as pd
import numpy as np
import re

df2 = pd.DataFrame([[1, 'plugs: $3.00'],
                    [4, np.NaN],
                    [7, 'quarts: $3.00']],
                    columns=['name', 'price'])

df2

    name          price
 0     1   plugs: $3.00
 1     4            NaN
 2     7  quarts: $3.00

def price(x):
    rprice = re.search('(plugs:|quarts:)\s*\$([\d\.]*)', x)
    if rprice == None:
        return ('','0')
    else:
        return rprice.groups()

df2.fillna("", inplace=True)
df2['price'].map(lambda x: price(x))
df2['Plugs'] = df2['price'].map(lambda  x: float(price(x)[1]) if price(x)[0] == 'plugs:' else 0)
df2['Quart'] = df2['price'].map(lambda  x: float(price(x)[1]) if price(x)[0] == 'quarts:' else 0)

df2 (with new columns below)

   name          price  Plugs  Quart
0     1   plugs: $3.00    3.0    0.0
1     4                   0.0    0.0
2     7  quarts: $3.00    0.0    3.0

Some Notes:
I used regular expressions to extract the type and cost, and I replaced the NA with blank text.


Your images are probably too large for using just three convolutions. So the number of parameters is huge during flattening (and the net cannot learn global features of the images, but this is a different problem).

Try to

use more convolution and pooling operations
see if strided convolutions help
check the number of parameters using the summary method of Keras


You have a typo in your formula
#derivative of fun1.
fun1.tag <- function(theta){
  y <- 0
  for(i in 1:length(x)){
    y <- y + 2*(theta^2+(x[i]^2)-20*x[i]-1)/((1+(x[i]-theta)^2)^2)
  }
  return(y)
}

There is y +  missing inside the loop.


What to do when we can't trust our human classifiers?

First, neural networks are good in dealing with"label noise". I'm currently on mobile/vacation, so remind me to search the paper on Friday.

Second, the more important question is how to get a good ground truth. Without a good ground truth you can't evaluate your models, no matter how good they might be.

I see the ways:

(1) have multiple experts label the stuff. Then you can make the ground truth a probability, not a simple label. If 9 experts say it is cancer and 1 says it is not, you would label it with 90%

(2) wait. If you can access the patients data, it will likely be more obvious in a year (especially if it was not treated)

(3) other diagnostic methods: I'm not a medical doctor, but I'm pretty sure there are invasive methods to diagnose cancer which are reliable


Can I train a CNN to detect the number of objects without localizing them first?

There are different approaches to achieve your goals:

1.) Provide only number of objects in an image in order to train an object detector.
This is called weak supervision or weak labeling.

Some works that utilize this approach:

https://arxiv.org/pdf/1711.05282.pdf : Their approach is to train an object detector on image proposals, together with object counts.
https://hal.inria.fr/hal-02393688/file/1912.00384.pdf : Object detectors can be trained by providing only one categery label per image

2.) There are several works focusing on counting objects:

For example the following works focus on counting arbitrary objects:

https://arxiv.org/pdf/1903.02494.pdf
https://arxiv.org/pdf/1604.03505.pdf

For density estimations there are several works counting people:

https://arxiv.org/abs/1903.00853

Depending on the setup, it could also be used for your application,


Training NLP with multiple text input features

Concatenating the whole question and its answers in a RNN could be an option to try, but then always use a reserved special token (or various) to mark where the questions start. E.g. you could concatenate like:

Question text <1> answer 1 <2> answer 2 <3> answer 3 <4> answer 4

where <1>, <2>... are the special tokens so that the model, with enough examples, may be able to understand its meaning.

The "enough examples" is worth stressing, specially as this type of model may require a large complexity (in terms of number of parameters) to make it work, and hence you will need a large enough data set, possibly of the order of the 10k or larger. You can also test some data augmentation by mixing the order of the answers in the input, and of course changing the correct label.


How to implement an LSTM RNN with multiple input features

It is becouse you convert your data to list data = data.values.tolist() But model input expects an array but not a list of arrays. So you need to convert your input list to the array.


the error occurred while selecting feature using recursive feature elimination in sklearn

Recursive feature elimination works by considering co-efficient value of feature column (in case of linear models) or variable importance exposed, by say, random forest classifier.

For example, in case of a linear model, it will eliminate feature with smallest (absolute) value of co-efficient (i.e. less impact on outcome).

In case of SVMs with non-linear kernels, the classification is happening in a transformed space of features and not original features. So it is not possible to assign importance measure to original features. So RFE doesn't work with RBF kernel here.

Related question on variable weight for non-linear kernel


Is it good to remove outliers from the dataset?

It mostly depends on what you are trying to achieve with your model.
Sometimes the information carried by outliers is indeed negligible if not of interest (say, for example, that those high values are caused my data collection/input errors), and may affect your model performance. In some other cases though, outliers carry a lot meaning and you might want your model to be aware of their existence/possibility. In other scenarios, the outliers are what you actually care about (see Anomaly/Novelty detection e.g.).

Long story short, if these outliers are really such (i.e. they appear with a very low frequency and very likely are bad/random/corrupted measurements) and they do not correspond to potential events/failures that your model should be aware of, you can safely remove them. In all other cases you should evaluate case by case what those outliers represent.


What skills do I need to become a data scientist? And how to show them?

So you're still on the Basics and William's answer is pretty good, I will list here a bit of stuff to learn, and where to.

1 - You need the basics, that is already much more than you expected it to be:

Linear Algebra: knowing the best way of inverting a matrix might be useful for a computer scientist, but you're not aiming for that. You need to understand concepts and their meaning and effects such as:

Matrix Rank (For example, this could tell you, by an Autocorrelation matrix, that your data is still not enough for things like least squares.)
Meaning of Vector Spaces and basic linear transformations such as base change
Meaning of eigenvalues and eigenvectors

Calculus: also, focus on the meaning and understanding, computers can do most of the operations, even analytically

Derivatives and Integrals
Optimization

Signals and Systems: that might be a little bit biased (since I am a signal processing researcher) of me but, learning how to model certain phenomena and how they behave may help you to solve problems, that's basically applied Linear Algebra and Calculus (most things are). Aim for really basic, Signal processing is one of the research areas most affected by Data Science/Machine Learning, to the point people were making surveys to change the name of the community in IEEE.

The above can be found on many, many books and intros and won't be difficult for you to search for them.

Statistics: Machine learning derives from statistics and this is essential. Actually, you can learn calculus from a statistic point of view instead of function optimization. Links in the subjects names are free courses to Udacity or Udemy

You will need to learn Descriptive Statistics, which will allow you to understand data.
And learning Inferential Statistics.

It is important to learn what you can do with classic statistics to avoid wasting computers in problems that could be solved easily. It is a good practice to model things in the simplest way possible and escalate to more complex model if needed.

Machine Learning: Machine learning models are your everyday tools as a Data Scientist:

You can start with a crash course on Google Developers and Intro to Machine Learning in Udacity.
Then, you can go deeper... NO! Not Deep Learning, deeper into Machine Learning, also on Udacity.
But you might want to take a more formal course such as Stanfords at some point, just to fill some voids.

Machine Learning Tasklist: You stopped at linear regression? So here are the most popular models waiting for you:

Linear and Logistic Regression: pretty basic
Decision Tree: basic but highly interpretable
Support Vector Machines: simple but powerful (and I love Kernel methods, shoot me for that)
Naive Bayes
k Nearest Neighbors
K-Means Clustering
Random Forests and Gradient Boosting (Ensembles): these are really powerful and might be interpretable if you don't let them grow without care

Also, you might need some dimensionality reduction tools such as:

Principal Component Analysis
Linear Discriminant Analysis

Then you can go for more complex Neural Networks:

Self-organizing-maps, Feed Forward, RNNs, CNN's and so on...

Note: CNN's are usual in Computer Vision applications but the only thing it asks of data is that it has been organized in a way that allows for meaningful correlations between data that is near each other. Example: a process with multiple sensors in a time series might benefit from a CNN.

Data Analysis: a Data Scientist must have a personal relationship with data. For any good relation, you will need to understand your loved one (but the relationship with data is usually toxic, hahaha). Udacity has a nice Intro to Data Analysis, also for free.
Learn to express yourself:

make a youtube channel, present small tutorials, and classes to the community.
Try to answer Stack Exchange questions and help others, that will build your respect in the community and goodwill for when it is your time to ask. Also, this is a good way to practice expressing your ideas in a text.
Write a blog, it is a good way to have a notebook and also gaining attention from the community.

You can check on hands-on books such as Data Smart and Data Science from Scratch. Data Smart is about getting Insight from information and that's mostly your job as a D-Scientist.

2 - Build Respect

Try creating packages and libraries and making them available on GitHub and sharing your relevant solutions.
Win Kaggle competitions, many companies take Kaggle seriously as .... and having a good score will get you nice positions. You don't need to win in first place by the way. Also, competitions are usually good examples of real-world problems that will get you that so needed experience that you can't get while you don't have a Data Scientist role.
Also, some competitions pay really well.
Explore Kaggle, share algorithms, read and try to improve on others and search for datasets that might be of your interest.
Making datasets is a bit exhaustive but might be a way of making money while you're not ready.

3 - Get programming skills

Not only learning frameworks but understand how things work and basics of problem-solving must be put to the test every day. Also, making everything from scratch is fun and is good for learning, but when you do your work you will need code with high maturity, checked hundreds of times all over the world.

You'll need some tools, Python is a great language for data science (since the community is active and it is free, Matlab has a lot of nice tools and marvelous documentation but it is really expensive and quite a bit slow)

Some top libraries:

NumPy is the most fundamental package, understand it well
Pandas for wrangling with data
Seaborn, Bokeh, Plotly, and Matplotlib for plotting stuff and helping you making good reports
SciKit-Learn this is usually the fastest way to test a machine learning algorithm
Theano is similar to NumPy but was constructed with Machine Learning in mind
Keras, this is a library for building Neural Networks really fast, it uses Theano or TensorFlow as backends
TensorFlow, PyTorch, and other deep learning related stuff.

Also, you might want to get some knowledge of JavaScript and libraries for acquiring data on the web.

4 - Go Deeper

You may never need deep learning, depending on the area you're going to apply but this is that good nuclear weapon you hope to never have to use but someday you may:

Understand start with Udemy
There is a nanodegree on Udacity

Also, remember DL is computationally intensive and you want to avoid needing it (since these are expensive)

5 - Finally: Career

Learning never stops and you will never stop learning new concepts, every single day.
Courses are long, take them at your own pace. Try getting the basics of learning how to use, then go back and learn it for real
Try to get a few certificates and posting them on your LinkedIn. Make a few projects and create online articles on your blog and on LinkedIn and Kaggle.
Try choosing something you can relate to while searching for work. DS covers a wide range of subjects and trying to get insight from things you understand is easier than trying to get insight from things that sound like random noise for you
Build a network of collaborators, help your colleagues and try building a vast network ranging from medicine to linguistics, they might tell you what you're doing wrong while looking at data not related to your field of expertise.
Finally, and this took a bit longer than I anticipated but: don't give up. This is a long journey but is absurdly rewarding, both financially and personally. And try not working alone, create a small group of people to work with and make some projects.
This is my longest answer to a Stack Exchange question.


How to interpretate multiple histograms corresponding to each feature in multiple linear regression for relationship?

A histogram shows the distribution of values in a feature/variable. For instance in CHAS you see that there are two values (0,1), which means binary encoding (aka one-hot or dummy).

Histograms are helpful to understand the general structure of data and to detect things such as outlier or censoring. They are also good to get a quick idea of skewness in the data/distribution. Look at „age“ for example. You can see concentration of values at the top of the x-values. This likely is due to houses being older than 100 years are recorded as „100“ in the data. A similar pattern occurs for the price.

However, wrt „significance“ and correlation, histograms do not tell you a lot. For correlation, look at correlation measures. For significance (I guess you mean importance), you need to do sequential testing of features or you employ Lasso/Ridge regression for a more systematic approach.


Well, I think your question does not have a "correct" answer since you are looking for a piece of advice on ML model deployment.

As you mentioned, you have a complete back-end in java, and you should not drop it.
Instead, (1) you can take advantage of a micro-services oriented approach. With micro-services, you can communicate different pieces of code using HTTP. You may have some of your back-end controllers in java, and some kind of ML-managers to wrap prediction routines with web routes.

I think it's a good idea for you to learn python, and also python web frameworks (flask is a very simple one, you can google "ML deployment flask" or something like that, and there is plenty of tutorials). Also, (2) you can convince your data scientists to write their micro-services for you because it is simple to do. Then, you can get their code and adapt to better look like a web-service.

Another way to take advantage of your java back-end is to (3) try to translate the python/R ML code to Spark MLlib, which has many powerful methods for you to write your java code. However, you will eventually have to code some operations that your data scientists made in python/R, which is not available in MLlib.

It depends, but I hope that it helps!


Unless you are trying to do this as a learning exercise, just use Spark which has ML libraries made for distributed computing. See here


What if outliers still exist after variable transformation?

No right way in all cases. I have dealt with outliers from statistics and business problem view.

Are the outliers in a segment the business is expanding into and expect more people in these "outlier" areas? In this case these are not outliers from the business view and probably should be kept.
Are the outliers in a segment the business is retreating from and expect fewer people in these areas. Possibly want to get rid of these records.
Are these outlier records just outlier in this feature or in others? If they are just far from the mean in 1 feature, might want to keep them.
Discretizing - I am not a fan since the model is losing information. But can add an indicator variable and try the model. Might want to try multiple bucket approaches but I still think the model should see the real numbers.
I built models where the model would always get these outliers correct or incorrect regardless of if they were in the model training or not. So including these was a moot point. Want to make sure they are in a validation set to check. Make an outlier validation set.
Is the business treating these people different regardless of the model. For example, if this is a marketing model, the business might be targeting high income people regardless of what the model says. So including these in the model may penalize lower income people. Try building the model with and without.
I am sure I am missing other techniques that I have done. Outliers do not mean wrong and they do not mean right. They can be looked at from a purely statistical view. But in building a model since we should know something about the business problem and have access to subject matter experts, we can probably have a better informed decision. And always test. Might need to build multiple models.


Handling categorical features in Factorization Machines algorithm - Feature Hashing vs. One-Hot encoding

I decided to expand a bit on my comment and make a full answer.

So the reason why somebody may say that performing the hashing trick can destroy interactions is because it may map different features to the same bucket. But usually it's not a big deal.

Note that DictVectorizer doesn't perform the hashing trick:

  When feature values are strings, this transformer will do a binary one-hot (aka one-of-K) coding: one boolean-valued feature is constructed for each of the possible string values that the feature can take on. For instance, a feature “f” that can take on the values “ham” and “spam” will become two features in the output, one signifying “f=ham”, the other “f=spam”.

To do it, you need to use a different vectorizer: HashingVectorizer


k-means clustering data with large number of meaningless values

One thing you could do is apply some dimensional reduction algorithm (such as PCA) so you can get the columns with high variance, then run k-means on that data set.

However, I suggest against using k-means in sparse matrices like yours. Anony-Mousse's answer here explains it well.


A Bernoulli distribution is a probability distribution where a random variable takes the value 1 with probability p and the value 0 with probability 1-p. This distribution maps very well onto logistic regression because the random variable is the target variable and the model is estimating the probability of 0 or 1 given the specific values of the features.
The Gaussian distribution is not as well suited because it is a continuous probability distribution that is goes from negative infinity to positive infinity. The result would be predictions outside the bounds of the problem space.
The cost function is something different. The cost function is how errors in prediction are penalized.


Can somebody explain me the the following parameters of Keras LSTM layer

units : According to the official docs, it defines the output dimensionality. In simple words, the number of LSTM units which will be used.

  units: Positive integer, dimensionality of the output space.

Units are nothing but the LSTM cells which will be used to process the inputs.

stateful :

According to the docs :

  stateful: Boolean (default False). If True, the last state for each sample at index i in a batch will be used as initial state for the sample of index i in the following batch.

If this argument is set to False then, the states or the memory of the LSTM cell will get reset at every sample which is passed through it. It follows this manner:

Take a sample from the batch.
The sample will pass through the LSTM and produce a state vector ( conditional memory ).
The output is given
If stateful=True then, the state vector will be used as the initial state for the 2nd sample.
If stateful=False then, a new state vector will be used for the 2nd sample.

Hence, the memory should migrate from one sample to another is decided by the stateful argument.

unroll :

According to the docs,

  unroll: Boolean (default False). If True, the network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences.

It can thought as :

  RNNs, once unfolded in time, can be seen as very deep feedforward networks in which all the layers share the same weights. ( Deep Learning, Nature, 2015 )

You can read an excellent article here.

The unrolling will the make the LSTM, a deep feed forward network which has shared weights. You can see and understand through this figure.


Adding the input layer - units with a decimal

Round up or down to a whole number. Keras documentation specifies that units should be a positive integer, and I'm not sure what a fractional unit would even mean. Does this work when you try to run it?


Looking for binary class datasets with high class imbalance, that also have intra-class imbalance in the minority class

The imblearn.datasets package (documentation is here) has a function called fetch_datasets() which is described as:

fetch_datasets allows to fetch 27 datasets which are imbalanced and binarized

I do not know them in sufficient depth to know whether they meet all four of the criteria you've listed, but these may be a good first place to start with.


How do I force specified coefficients in a Linear Regression model to be positive?

Sorry, but on the surface, this sounds like a terrible idea to me: if linear regression gives you negative coefficients for some explanatory variables that you think should be positive, then it means that either your data is "wrong" (typically noisy or too small) or your intuition is misguided.

I can't see any good reason why one would use a data-driven approach if the goal is to manually force the model in a particular way. This is the equivalent of breaking the thermometer to hide the fever.

I'd suggest the following instead:

In general an unexpected outcome is arguably a good thing, in the sense that it tells us something we didn't know about the data. That's a cue to investigate what happens in the data. Linear regression is simple enough to analyze: one can look at the correlation, plot the relation between the variables etc.
If there's really something suspicious going on with some variables, maybe some errors in the data which make them behave in a way they shouldn't, then it's much better to discard them altogether from the model rather than fixing their coefficient, because this way the model won't rely on them at all.


Higher variance in PCA can mean, that data structure is less informative?

It is not the best use of Principal Component Analysis (PCA) to compare information loss for different data structures.

It would be more appropriate to use Information Theory. Information Theory is the study of data compression. It provides many tools to make quantitative comparisons across different encoding schemes.


Where can I get labels for small ImageNet?

I am affraid this dataset does not include labels,

You can use datasets that are described here:

https://patrykchrabaszcz.github.io/Imagenet32/

Those files will have the same format as CIFAR dataset

And they will contain labels according to the map file that is available in original Imagenet devkit.


Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib

If I were you, I would pick anyone of the frameworks I am comfortable with and implement the use-case.
Spark-Streaming + MLlib should work and would be my choice since its user base is on the rise and it is one of the most popular project under the Apache Umbrella with good enterprise business plan. Both Cloudera and Hortonworks provide enterprise level support.
Now, in theory Spark-Streaming lacks behind Storm in stream processing, but the framework is cool in a way that it provides you the option to do streaming, common map and reduce, graph processing and SQL under the same framework. So once you have the pipeline to convert your data to RDD you are good for most of the common jobs related to Data Analysis. It's written from scratch in Scala which is a very powerful language and provides huge scalability in a distributed setup when handling concurrency.
Hope this helps, feel free to reach out to me with any questions you have.


Feature selection, is it possible to combine wrapper and embedded methods?

Can I use Lasso as the estimator in a "wrapper" feature selection method?

It will certainly work, but the embedded feature selection of Lasso is unlikely to actually come into play in your example.  The forward sequential feature selection builds the model for each candidate feature to be added to the selected feature set, starting from no features.  When you only build up to three features then, the only way Lasso's feature selection will have an effect is when it determines that fewer than three variables should be kept, or if it somehow decides that one of the already-selected features should have zero coefficient.  The latter would be very surprising, and the former only makes sense if your data is very simple (only two variables contribute anything significant to the predictions).

So, here Lasso is probably only really contributing as regularization, shrinking coefficients but probably not zeroing any of them out.

How should I do feature selection for a neural network?

Since Lasso is a linear model assuming feature independence, you may be handicapping the net's ability to find and use nonlinear relationships or feature interactions.

I doubt there's a "right" answer here (no free lunch sort of thing).  Here are some ideas though.

Don't do feature selection.  Count on the neural network, perhaps with heavy regularization methods (dropouts, L1 penalty especially on the first layer, etc.), to sort out what's important.
Do hyperparameter tuning inside the wrapper.  This is very computationally expensive, but may be the most performant approach, and might be doable for smallish nets.
Fix the architecture inside the wrapper.  You say "every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned," but this might not be a problem.  Often enough, if a net has more neurons than it needs, several may learn nearly the same feature.  (You might try to turn this into a compromised version of (2), by applying some network pruning instead of full hyperparameter tuning.)
Use another model inside the wrapper: one that can capture nonlinearities and feature interactions unlike a linear model, say a tree-based model.  This still may fail to attribute importance to features that a network would pick up on, but it should be closer than a linear model.


There are two possible reasons for this result:

Low number of training examples
Using dense layers without batch normalization

You have a relatively deep network and your training set size is small. In such cases if you run your model too many times you will certainly overfit the training data. The reason is that whenever you have a powerful model that can learn complicated functions and if you provide low number of training examples, it has capability to fit the data and not learn it. Suppose that you have a fitting problem in calculus and you have 4 points and you also have polynomials with degree of four. In such cases you can fit the data exactly without any error but the point is that you are fitting it, not learning it. In your case your model is powerful and it tries to fit the data, not to learn it. The cure is to provide enough training data.

The reason that your model can not learn is that you are stacking dense layers without batch normalization. each time you update the weights, the output of the deep layers change and this cause the covariat shift. To avoid this problem use batch normalization.

As a solution for you I recommend you to putting just two hidden layers and manipulate the number of units in those layers and provide more training examples. Your task can easily be learned by a two hidden-layer-network.

After struggling for about one day finally I want to express my opinion about the problem. Depending on the problems, they may be solved using machine-learning or other techniques. Machine learning problems are those which you can not define an appropriate function to map the input to output or it may be so much hard to do so. Whenever you have the function, it can be used as the hypothesis, the final goal of machine learning algorithms. I tried hard and put so much time on this code and I get the same result. Nothing is going to be learned at all. I have been trying for about one day and still no progress has been seen. To explain the reason, the data is so much hard to be learned! for imagining how difficult it is, I recommend you to write numbers from one to ten in a straight line and put a line between consecutive numbers. The numbers are endless, so you will have no generalization because the boundaries that are going to be found will work just for the two neighbor numbers. This means that if you use the current features, you can not separate, learn, your data. I tried to do, somehow, feature engineering and used the following code to solve the problem:

import keras
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers.normalization import BatchNormalization
from keras import regularizers
from keras.optimizers import Adam

def gen(x):
    if (x % 2 == 0):
        return 0; # represents 2
    else:
        return 1; # represents 4

a = []
for i in range(1,100001):
    temp = np.random.randint(0, 10000000)
    a.append([temp, temp ** 2, temp ** 3, temp ** 4, gen(temp)])

a = np.array(a)

x = a[:, 0: a.shape[1] - 1]
y = a[:, a.shape[1] - 1:]

mean_of_x = np.mean(x, axis = 0, keepdims = True)
std_of_x = np.std(np.float64(x), axis = 0, keepdims = True)
x = (x - mean_of_x) / std_of_x

n_classes = 2
y = keras.utils.to_categorical(y, 2)

percentage = 95 / 100
limit = int(percentage * x.shape[0])

x_train = x[: limit, :]
y_train = y[: limit, :]

x_test = x[limit: , :]
y_test = y[limit: , :]

x_train.shape

model = Sequential()
model.add(Dense(1000, activation='relu', input_shape=(x_train.shape[1],)))
model.add(BatchNormalization())
# model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(BatchNormalization())
# model.add(Dropout(0.5))

model.add(Dense(n_classes, activation='softmax'))

model.summary()

model.compile(loss = 'categorical_crossentropy', optimizer = keras.optimizers.Adam(lr = 0.0001, decay = 1.5), metrics=['accuracy'])

model.fit(x_train, y_train, batch_size = 128, epochs = 2000, verbose = 1, validation_data=(x_test, y_test), shuffle = True,
          class_weight = {0: 10, 1: 1})

As you can see, the above code uses high order polynomial. Surprisingly, no progress was seen here too. this model has the mentioned problem for the previous feature in higher dimensions. That's why the learning does not happen here too.

the point here is that although you can not learn using the current feature, number itself, or its high order polynomials, you already have a solution for the problem. Instead of passing the number itself to the learning problem, pass the modulus two of the current number. This feature is so much easy to be learned. you may need just one unit.


Have you tried to do a correlation test between the output and the features used to model it?

I usually use Pearson's Correlation to determine the features with low correlation factors. Then I remove them from the training.

In here, I have DOC_DS used to model DOC_US, I have them both in a data frame df with my other features. From the trails I did on my models, the features with less correlation affects the model accuracy later on.

In Python Environment:

print(np.array_str(np.corrcoef(df.DOC_DS,df.DOC_US)))


How to get windspeed when temperature is maximum for each city?

First, you should find maximum temperature for each city, then merge it with the main data to find the relative windspeed

city_and_max_temp = data.groupby('city').agg('max')[['temperature']]
data.merge( right=city_and_max_temp ,how='inner')[['city','windspeed']]


Machine learning algorithms for Geometrical objects shapes identification

This is quite doable.
Just make sure of the following:

Any test data should be one of the shapes you have trained on.
Make sure you have balanced data. (You can introduce bias if required)
To make it more challenging you can create or find data like google street view images for numbers.
Segmentation and Classification, both are possible.

VGG should do just fine, you could also use resnet. The overall model can be fit into an inventory pipeline where you sort the inventory through shapes!


How to select features based on feature importance using SelectFromModel?

The error is related to this line:

 selection = SelectFromModel(rg_cv, threshold=thresh, prefit=True)

It should be modified as follows:

print("Tuned rf best params: {}".format(rg_cv.best_params_))
be=rg_cv.best_estimator_
best=be.fit(X_train,y_train)
thresholds = np.sort(rg_cv.best_estimator_.named_steps["clf"].feature_importances_)
for thresh in thresholds:
    # select features using threshold
    selection = SelectFromModel(be.steps [1][1], threshold=thresh, prefit=True)


How does back propagation works through layers like maxpooling and padding?

Max pooling will cancel the effect of not pooled values to the gradients.
Padded values either have no effect.

Nice thing about convolution is, that it is basically reducable to a matrix multiplication and the backpropagation is simply the transposed of it. So you have already your backward pass stored in the foward pass.


Where is the error in the following code?

First of all, since you only seem to want to know about the Python error, your question is probably better suited for Stack Overflow.

Secondly, this is not fun to answer with the information you've provided.  This is certainly not a minimal working example (MWE) - I can't copy your code as is, and run it myself.  I'm left to guess many things:

Are you importing matplotlib?
Did you rename matplotlib.pyplot as plt?
Are you importing numpy and did you rename it np?
Does your traceback literally end in KeyError is 0 or is it KeyError: 0 or KeyError: False?  Can you show the full traceback?  The traceback for a MWE is typically not long.

In the positional argument (movieDF[movieDF['%s'%genre]==1].year) on your problematic line, you ask for a key (movieDF['%s'%genre]==1) that looks like it should evaluate to a Boolean.  Is this never 0?  I'm forced to grasp at straws because you haven't provided a sample dataset; I shouldn't have to download from IMDb myself.

Edit:

Just to troubleshoot, can you place x = movieDF[movieDF['%s'%genre]==1].year on a line by itself and tell me what x[0] is?  What type would that x be?  Is it a dictionary, where the .hist() method is expecting a list or other array?  This documentation makes it sound like the method will not accept a dictionary...what type is movieDF.year?


Batch Normalization and Input Normalization in CNN

Its is basically not really important to rescale your input to [0,1]. Your input data should simply be in the same range. So [0,255] would be also a legit range.
BN should be different to interpret. It helps to keep your activations through the whole network on the same level. It is especially helpfull when we talk about deeper archtitectures.
Actually when you spend some extra time/iterations, your network should also converge without BN.


problems during training a MLP type of network

When the training loss is lower than the validation loss, the model is said to overfit the training data, i.e. it has learned so much from the training data that it only adjusts well to it and it can't generalize to new data. This phenomena is regarded as the variance of the model. The bias of the model is the difference between the training loss and and the loss you've previoussly selected as the minimum loss reachable, or the desired one.

However, this analysts is usually done over other well known metrics, such as precision and recall. You first calculate these metrics on your training data, and then, on the evaluation data. Then you perform the analysis taking the same considerations.

In order to reduce the variance/overfitting, there are common techniques:

Increse the training data by adding more instances, if available
If no more instances are available, perform data augmentation to increase the training dataset
Use regularization, for example, dropout.
Shorten the network. If there are lots of layers, the network may be learning too specific features from the training data

As mentioned before, I'd perform the analysis using other metrics rather than the loss.


Temporal train test split for recommender systems

What I have previously done, and what have worked well in my domain (retail), is to first get a sense of the distribution of your user/rating frequencies. More specifically, say you want a test-set consisting of 10% of the observations, then figure out the threshold (n) where, if you take all users with at least n interactions, and their latest interaction(s), you end up with 10%-ish datapoints.
In your case, perhaps some fraction of the users have at least 5 interactions (ratings), then take the latest interaction(s) these users have done, and use that as your test-set (use more or less than 5 to change the size of your test-set). This way, there exists no users in your test-set that are not a part of your training-set, although, the users with very few interactions will not be represented in the test-set. This train/test-skew could also pose a problem, but not being able to provide recommendations for these users, I have found, pose an even larger problem.


Implementation of Association Rules in Javascript

I don't know how efficient they are, but I did find some implementations:

https://github.com/dmarges/apriori
https://github.com/seratch/apriori.js


The key question is how we can allow the public to make useful queries to a dataset without revealing private information.

The field of Differential Privacy deals with answering just that.

The key concept is to track a user's queries and the information revealed and add uncertainty to the answers to guarantee privacy.

For example, a broad query like "what is the rate of cancer in Europe?" could have an answer of with "1% +/- .1%", whereas "what is the rate of cancer for the population of men aged 72 living at my neighbours's address?" would give an answer of "50% +/- 50%": we do not want to reveal whether a specific individual has cancer or not.

Differential privacy gives users an information budget and tracks the queries, results and information revealed. The more you ask, or the closer you get to personal information the less specific the answer. At one point the information budget is exhausted and can query no more. You end up with nothing more than "your neighbour has cancer with probability p=0.5 +/- 0.5".

This also guards against constructing a clever sequence of queries to pin down the answer like in the children's game of "20 questions". For example, you might try to ask questions like "what is the rate of men of cancer for men of age 72 in this city?", "what is the rate of cancer for men of height 182 cm at age 72?", "what is the height distribution of men with my cancer in my street?" etc. until you can deduce whether your 182 cm tall, 72 year old, male neighbour has cancer or not.

Differential Privacy guarantees that such strategies will never work (under certain assumptions).

Moving Averages

With this background, your example of only publishing the moving average seems quite reasonable. Differential Privacy can be used to calculate how long or short the moving average needs to be on your dataset in order to provide the best possible information while preserving privacy.


Have a look at the Common Voice project from Mozilla. They have an open source dataset of multiple languages, Spanish currently has 324 hours of validated speech data (out of 579 total). You can download the dataset for Spanish and other languages at https://commonvoice.mozilla.org/en/datasets.


How to seperate text lines from txt files in python?

You can read the file line by line, discard blank lines, and wrap the lines that are left by a list to get the same as your desired result

with open(file_name, 'r') as f:
    print([[x.strip()] for x in f.readlines() if x.strip() != ''])

By doing [x.strip()] you clear the whitespace at the beginning and end of the file and then wrap it in a list, and the condition x.strip() != '' makes sure that the line actually contains content and is not just a blank line.


Proper order of optimizing hyperparameters in ANN

My initial suggestion is to look in the literature for networks which have been implemented to do solve similar problems and start from there.
Then you should just try a bunch of different values for the parameters and see what works in your case.

See also this Cross Validated question: https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw


W is just the coefficients that we estimate to determine the contribution of each variable to the model. We attempt to estimate the log odds ratio by multiplying coefficients by the input variables that we have.

If we didn't have W, by default it would just be the 1 matrix multiplied by X, or each variable output would be assumed to affect the response equally (ignoring the scale differences between predictors).


What can be done to increase the accuracy of a biological dataset?

There are many possible explanations.
Maybe just mistakes in the code or maybe there is just no better separation possible.
When usign SVM the features are extremely important and you should most of your time invest in choosing/designing them.
The statement "I have applied Deep learning" ist just to vague and therefore I can not give any tip.
Third when dealing with unbalanced data you have many possible actions.
I recommend reading:
Oversampling and undersampling in data analysis


Time Series - Values recorded every 10 minutes - Fill missing values

One option would be to interpolate the missing values. There are various methods depending on your use case (ie. linear, cubic, etc.)

https://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html

You could also drop the missing values df.dropna() and just use the known values you have for thresholding.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html


Where can I download historical market capitalization and daily turnover data for stocks?

Quant SE is better place for questions related to getting financial data:

What data sources are available online
Market capitalization data | Quantitative Finance


Why do we use the F1 score instead of mutual information?

The reason why one uses any particular evaluation measure should be based on the semantics of the task. Corollary: there's no unique evaluation measure which is perfect for every task.
Obviously there are also technical constraints to take into account, but imho considering only the technical aspect of performance optimization is the most common mistake made in ML applications.

For instance OP mentions non-symetry of the F1-score: in some tasks it makes complete sense to use a non-symmetric measure, for example any task where the goal is to extract specific elements, e.g. named entity recognition. Moreover the question of symmetry makes sense only for binary classification, and many tasks involve more than 2 possible classes. In such cases it's common to use micro or macro F-score, but again it depends what one wants to measure. It's also important to keep in mind that a single metric is always a simplification of the performance, often there are many other relevant aspects. For instance, mistakes for a particular class might be more costly than for another, like in the case of medical prediction systems where a false negative means that a patient doesn't get a potentially life-saving treatment (note that generic F-score can be weighted to account for that).

The interpretability of the performance measure is also very important. In many applications one needs to estimate some kind of average error rate that a non-expert can understand, otherwise the ML system is just a magic black box to its users and this can cause serious ethical and practical issues.


Kaggle has a "Wine Reviews" dataset with over 130k reviews. There are even several tutorials based on this dataset I believe. I would check those notebooks before starting any new project.

https://www.kaggle.com/zynicide/wine-reviews


Multi-label classification based on single-label dataset

I would train it using the loss appropriate for multi-label (sigmoid activation/binary cross-entropy loss instead of softmax activation/categorial cross-entropy loss).
The model will give you probabilities per label, so it's up to you to decide how to interpret them in your application.

Having said that, some multi-label problems have mostly single labels, much fewer double labels, and almost no labels beyond that. For problems like this, you could try a single-label model and see if it works better than a multi-label one. Or have an ensemble :)


Under what conditions should an autoencoder be chosen over kernel PCA?

I think you can interpret autoencoders as essentially performing kernel PCA, but with an extremely complex kernel. Like, really, stupidly complex.

The kinds of things you can accomplish with autoencoders are spectacular. For example, consider BicycleGAN.

If kernel PCA is a step above PCA, autoencoders are miles away. If you have an application where PCA makes sense but isn't working out, try kernel PCA. If you have an application where condensing your input into an embedding of some kind would be valuable but PCA is unlikely to be very helpful, use an autoencoder.


Linear Regression Coefficient Calculation

While I am not sure if you need the calculations done within the class specifically, there is a relatively more simple way to extract the intercept and slope coefficients using the linear_model from sklearn and pandas if it is of use to you.

Suppose we have the following variables:

y: [-0.006,-0.001,0.015,0.017,-0.0019,-0.005]
x1: [-0.018,-0.008,0.011,0.017,-0.008,-0.002]
x2: [-0.04,-0.003,0.012,0.011,-0.004,-0.009]
x3: [-0.06,-0.007,0.3,0.09,-0.005,-0.006]

Now, let's run a linear regression using sklearn:

from pandas import DataFrame
from sklearn import linear_model
import statsmodels.api as sm

dataset = {'y': [-0.006,-0.001,0.015,0.017,-0.0019,-0.005],
                'x1': [-0.018,-0.008,0.011,0.017,-0.008,-0.002],
                'x2': [-0.04,-0.003,0.012,0.011,-0.004,-0.009],
                'x3': [-0.06,-0.007,0.3,0.09,-0.005,-0.006]
                }

df = DataFrame(dataset,columns=['y','x1','x2','x3'])

X = df[['x1','x2','x3']]
Y = df['y']

# Regression Model
regr = linear_model.LinearRegression()
regr.fit(X, Y)

print('Intercept: \n', regr.intercept_)
print('Coefficients: \n', regr.coef_)

Once we do this, our intercept and slope coefficients are then printed:

>>> print('Intercept: \n', regr.intercept_)
Intercept:
 0.0022491408670789535
>>> print('Coefficients: \n', regr.coef_)
Coefficients:
 [ 0.62742415 -0.06618899  0.02384715]

Hope you find this of use if you are simply looking to extract the intercept and slope coefficients.


Error in using sklearn's GridSearchCV on Word2Vec

Do:
from sklearn.metrics import accuracy_score, make_scorer

s_model = GridSearchCV(s_obj,parameters,cv=2, scoring=make_scorer(accuracy_score))


IS there any way we can add range in hyperparameter tuning of Decision Tree?


Pandas DataFrame: Aggregating multi-level groups by matching keys

Sorry, I realised all I needed to do was to add another level of grouping, but the aggregation is the same;

stockBinGroups = data.groupby(['stock', 'binNum'], sort=False)
binAveVol = stockBinGroups.aggregate({'volume': np.mean}).reset_index()
binAveVol.head()

    stock  binNum         volume
0  stock0       2  174095.238095
1  stock0       3  100428.571429
2  stock0       4   79880.952381
3  stock0       5   73642.857143
4  stock0       6   69761.904762


What is the point of tensors in CNNs? Why not simply reshape the data into matrices?

Tensors come pretty natural in convolutionals networks.

Local pixel information matters: if $e$ is a pixel in your example above, it's important to know that $a$ through $i$ are its neighbors. This information gets lost when you reshape an image into a vector. Look how a convolutional layer works.

Training usually is done in batches, which is another dimension as far as a neural network is concerned. You don't want to mix different images into a matrix, they are totally independent.
Finally, there is depth channel. Initially it's one of R, G, B channels of the input image, and then each channel corresponds to a filter applied to the previous conv layer. Once again, filters are independent, it doesn't make sense to mix them up until the final layer.

So, in total there are 4-rank tensors going through the conv net. It's not only more intuitive (each dimension has a meaning), but also results in higher accuracy, because it employs all meaningful information from the images.


How to predict a group of events using machine learning?

Welcome to the site!

Assumptions before answering your question:

The Target variable is a categorical(each stage is considered as a separate category).
You have the past data of the customers whose application got rejected(if hasn't reached the last stage)/successful(reach last stage) at the respective level. For Example, customer A applied and got his application rejected at stage-2, No Legal).
You have some demographics of the customer to predict the stage at which the application might get rejected.
You have good data(no outliers, well engineered features, good understanding on data etc).

Now if the above assumptions are met, then you can apply Machine Learning algorithms which are used for doing multi-class classification.

Applicable Algorithms:

Neural Networks
Random Forest
K nearest Neighbor
Decision Tress
Support Vector Machines etc.

Finally, you can apply Ensemble model by combining results of the 2 or more models.

For measuring the accuracy you can use Confusion Matrix(By diving the data into Test: 30% of data and Train: 70% of data).


Which tool should I use for combining this large dataset?

I would suggest trying to do it in batches. The underlying issue could well still be memory related in some way, as the merge method makes copies of its input and so is not memory efficient at all.

As an example, you could read in 10 files, create the desired output, as you have done already. Repeat this for files 10-20, then 20-30 then 30-40. The finally four the four files that you have created. It is a bit of an annoyance, but sometimes these little workarounds get the job done.

[EDIT]

Another option might be to use the more involved memory management during reading, option via the chunksize argument of pd.read_csv(). This will read parts of the file into memory in chunks, as the name suggests. If you do this in a loop, it should put an upper limit on the memory usage. For example (untested):

chunksize = 50e6        # 50 Mb

for single_file in list_of_file_paths:
    for i, chunk in enumerate(pd.read_csv(single_file, chunksize=chunksize):
        if i == 0:
            result = chunk
        else:
            result = pd.merge((result, chunk), copy=False, how= ...)

You may need to do something with the indivdual chunks before merging them.

Additionally, note that I set the copy argument in merge to False, which might help - the documentation is a little vague as to how it saves memory.


No, minimizing the classification loss does not translate to minimizing regression error and does not yield better regression performance.

This is a core tenant of Measure Theory. Regression measures outcome variables on ratio, interval, or ordinal scales. Classification measures outcome variables on nominal scales. You can always go from more complex scales (i.e., ratio, interval, or ordinal) to simpler scales (i.e., nominal). There are no guarantees about going back. This is technically called a non-injective mapping.

One way to think of it is that you are picking a downsampling scheme to map the data from regression to classification, and there are many possible mappings back from classification to regression. There are no guarantees on which mapping was used. Once downsampled, the data has to stay in the lower sampling space.


JSON and XML are not tools. They are data formats. For example, to find out more about JSON you can look up the RFC description that specifies how to format JSON files:

https://tools.ietf.org/html/rfc7159

It explains how to tag the data. Then there are tools that validate JSON, and XML files to see if they conform to the requirements, or there are tools that convert JSON into another format or read it into memory into some object or class.


How to correctly pass Word2Vec vectors as input to an LSTM

That means your word 'enquiringly' is not in your word embedding vocabulary vocab.

For out-of-vocabulary(OOV) words, there is usually a embedding vector dedicated to them. Try to find that special symbol in the vocab and use that corresponding embedding vector.


Is this correlation between distance matrices?

You are basically right. You want to check the degree of dependence of one variable with respect to another one. No matter how you generate each variable, if you want to know how dependent it is from the other one, you normally use correlation to run that evaluation.

Very useful options you should consider to analyze the potential dependence between X and Y are:

Correlation: to measure both the strength and direction of the linear relationship between two variables.
Covariance: to assess the direction of the linear relationship between variables (not the strength).
Pearson's correlation: to obtain a single metric representing the degree (strength) of dependence between the two variables
Spearman's correlation: to evaluate a potential non-linear dependence between the two variables.

For more information, please, check this post here.


Choosing the right parameters for SARSA and Q-Learning & Comparing Models

As you are building policies in simulation, and can avoid the need to use approximate methods (the state space is small enough to fit in a table in memory), then your goal is to converge on the optimal policy. With correct setup, all the methods you are comparing are guaranteed to converge for your problem, provided the assumptions of reinforcement learning are not broken.

  With that being said, I cannot measure the performance based on the number of actions needed to reach the terminal state, as it will always be reached with the same number of actions.

This is just the reward scheme for maze solvers. For those the goal is to solve the maze quickly.

You have a reward scheme. The best policy maximises the expected reward. That is the only measurement that a completed RL output (your policy) cares about. As all your chosen learners should learn the optimal policy, you cannot compare them on this matter. So you maybe need to look at efficiency - how quickly the algorithm converges to optimal policy.

  The state-space is quite large actually, which is one of my concerns about how many epochs (episodes) I should have, its a 250*8*7*3000 space-action (3-state combination, 250,8,7 and 3000 possible actions per state).

I would characterise this as a small state space. It will fit into memory as a table. Looping through the 40 million state/action pairs should be pretty fast unless the simulation step is expensive for some reason.

In fact, if you have a simple model for transitions and rewards at each step, I would encourage you to use Value Iteration (a dynamic programming method) instead of the sampling based approaches in SARSA or Q-Learning. To do so, you would need to be able to define code for state transitions and rewards given current state and action. Sometimes though, the simulation is much easier to code than the transitions/rewards model. If that's the case for you, stick with SARSA or Q-Learning.

As there are no consequences to you for bad decisions and low rewards during training stages - learning offline in simulations - then Q-Learning may be preferable as it learns the optimal policy whilst exploring. Compared to  SARSA you have to be concerned about how to reduce $\epsilon$ so as to converge on the optimal policy. For Q-Learning you can leave $epsilon$ at a relatively high value (e.g. 0.1) and still learn whilst refining estimates of alternative actions.

However, I am a little concerned about this:

  the end-state reward depends on the actions selected in the other 249 days

This is a problem, since it breaks the assumptions of the Markov Decision Process that all the algorithms are based upon. The reward at any step should only be based on current state/action. By doing this, you may invalidate the policy. It definitely would defeat dynamic programming (may prevent it converging) unless you included action history in the state (which would definitely make your state space large). However, it may still work for sampling methods with a high value of $\lambda$, because these are more robust when faced with environments which are not strictly MDPs.

  Any suggestions on how to compare the models?

I suggest on CPU time required to learn the optimal model, assuming you intend to repeat this process on different data routinely. The way to find out is to experiment, as convergence will depend on properties of the problem, how fast your simulation code runs, and in your case you are stretching one of the assumptions behind the design of the RL solvers which could make strongly bootstrapping methods (like SARSA(0)) either fail to converge or converge very slowly.

If convergence takes too long, set a cutoff at some time limit (e.g. an hour) and find out expected reward from initial conditions when following the policy learned so far. You could Monte Carlo sample that for a fair comparison (and if results are stochastic even with a deterministic policy, then give plenty of time for 1000s of runs to get a good estimate).


Using pre-trained word2vec vs re-training a word2vec model

As always, "is A better than B" always depends on what you consider better? Is it accuracy, is it speed etc.

The dumb but correct answer to your question is: "try both and see which one is better".

Performance of techniques is always to some extent dependent on the data. What you need to remember with word vectors, is that they are learned based on a certain context. If the context that was used by Google's model is similar to yours, you might be better off using their model. But if it's different, you might run into some problems.

Just imagine the following case. You have 4 words: King, Man, Queen, Woman. Which pairs of two words would you create? Depending on the context, you could make a case for several

King/Man and Queen/Woman because of gender
Man/Woman and King/Queen because of the use of the word etc.


What Regresson Algorithm Would Fit This Data?

This answer is based on additional information provided in a comment:

  what is x, y?
  Balls 1,2,3 = Y, Date = X

Lotteries and other games (such as Casino games) are designed so that one is (or should be) unable to predict what happens. For instance in roulette, you could win a lot of money by applying the St Petersburg paradox. However, casinos restrict the sequence of playing so that you have no chance to do this.

WRT lotteries, the data generating process is supposed to be 100% random. This means that your X (the date) does not carry structural information about the outcome (y). Consequently, there is no way to make a reasonable prediction of what the numbers of next week would be.


Remove rows that are too much alike not to be duplicates

Take a look into record linkage methods. They typically are used to find the same entity between two datasets but can also be used to link a dataset to itself and find duplicate records (it’s called deduplication in the literature)


Do I have to standardize my new polynomial features?

It's better to do the square computing on the values of the non standardized features, and then standardize these new features.

The point of standardizing features is to put features in a similar scale, e.g. 0-1. If you didn't standardize these new features after the computing, although your new features are between 0-1, your new features will be one or more magnitude smaller than your old features.


How to know if classification model is predicting 1 or 0

You can find the model predictions here:
model1.predict(X_test)

array([0, 0, 0, ..., 0, 0, 0], dtype=int64)

In a binary (good/bad or 1/0) model, predict_proba is the probability of 0 or 1 , the value that is calculated by your model, which is used along with a cutoff (which is 0.5 in the sklearn implementation and cannot be changed) to determine if the final prediction is 0 or 1
model1.predict_proba(X_test)[:,0] # probability the answer is 0

array([0.94009529, 0.96378774, 0.98951049, ..., 0.67607543, 0.97599932,
   0.82838031])

model1.predict_proba(X_test)[:,1] # probability the answer is 1

array([0.05990471, 0.03621226, 0.01048951, ..., 0.32392457, 0.02400068,
   0.17161969])

since this is a binary model, both should add up to one.
The accuracy score is how many predictions were correct.
(correct_pred_0 + correct_pred_1)/total_predictions
If you remove the [:,1], you get the entire array:
model1.predict_proba(X_test)

        #col 1       col 2

array([[0.94009529, 0.05990471],
       [0.96378774, 0.03621226],
       [0.98951049, 0.01048951],
       ...,
       [0.67607543, 0.32392457],
       [0.97599932, 0.02400068],
       [0.82838031, 0.17161969]])

First column is the probability for 0 and the second column is the probability for 1
you can check the order of the classes using this:
model1.classes_,

for binomial, the default is like this:
array([0, 1], dtype=int64)


Best way to store large data set using R from Twitter?

Find a way to make your program write to disk periodically. Keep count of the number of tweets you grab and save after that number is high. I don't write R but psuedocode might look like:

$tweets = get_tweets();
$count = 0;
$tweet_array = array();
for each ($tweets as $tweet) {
  $tweet_array += $tweet;
  $count++;
  if ($count > 10000) {
    append_to_file($tweet_array, 'file_name.txt');
    clear_array($tweet_array);
  }
}


As described on the linked page, 128 is the dimensionality of each vector (i.e. the number of input channels), and 10 is the number of timesteps.
8 is the resulting number of timesteps after applying the filter of size 3 to the initial 10 timesteps.
In order to make it clearer, let's visualize the 1D convolution (source):

The missing parts in the picture are the batch size (4) and the output time length (8).


Can I plug models like Linear Regression into a CNN feature map result?

What you're explaining is basically almost every CNN model where you basically have a fully connected layer at the end of the convolutions and that is equivalent to having a linear/logistic regression at the end (given there's only one output)

The only difference between all of those is whether there's an activation function or not, and if you are trying to build a classifier then you would definitely want an activation function to map the output values to a probability value between 0 and 1.

Example:

That "Fully Connected" layer is just basically a Linear/Logistic regression (10 of them actually since we have 10 outputs in this example) and a softmax applied to the output values to scale them all between 0/1 and ensure the sum is exactly 1.


What is difference between leave one subject out and leave one out cross validation

Leave one out subject makes it sure that you don't have subject bias.

The fact that you have the same subject in your training and your testing datasets will make the model know more about your subject than it should. With a brand new subject, the model will probably perform poorly because it never trained on the subject before.

There is no such tool available in sklearn, as it's very easy to build your own if the subject id is a column in your dataset.


For 2 tables of size $n$ and $m$, join operation complexity is in $O(n.m)$ in worst case. If $n=m$ we are facing a quadratic operation which is why you observe slowness with the increase of tables size. Parallelize a quadratic problem requier at least to increase quadratically the number of slaves, without taking account shuffling, to keep constant execution time with increase of problem size which is not really a feasable option...


Binary classification as a 2-class classification problem

tl;dr

The two approaches you mention are equally effective.

Why?

First of all, I'll assume you're referring to a neural-network-type model. You should note that Convolutional Neural Networks are discriminative models, meaning that they are trained to find the differences between two classes (e.g. dog and not-dog).

  Would the second approach be less effective than the first?

Just to disprove this claim, consider the following. You have a 2-output classification network; let's name the outputs dog and not-dog. Now imagine, as you say, that the network can only identify dog patterns. All it would have to do was to give a positive weight to those patterns for the dog class and a negative weight for the not-dog class. In this sense the not-dog class would be trained as the opposite of the dog class, which would not make it less effective than a single-output binary classifier.

Even if the not-dog class could not be trained and remained constant, due to the fact that we're using a softmax activation, predictions would be generated just by the relative difference between the dog output and the constant not-dog output. This is exactly like having a single-output binary classifier.