Can an R^2, or coefficient of determination be used on non-linear data?

The $R^2$ calculation for nonlinear regression models can very a lot depending on the software / application / etc. Also, they involve a lot of approximations in order to make any calculation and are not as easily interpretable as the linear case. There are many other methods that will give you a good and very interpretable result directly from your data. For example, some kind of cross-validation like K-fold cross-validation or bootstrap methods give you a good sense of how trustworthy your estimates are directly from your data without relying on a bunch of normal approximations. Lastly, for most applications I would also recommend using some goodness of fit measure that takes into account parsimony like the AIC or BIC.

If it's a linear regression (linear in the parameters), then yes the $R^2$ is fine and is interpretable in the commonly thought of way.

$R^2$ is good for cases where you have no model for the noise in your measurement. A good way to think of it is just as a measure of the fraction of the variance in the measurement that's covered by your fit model. But it's not necessarily what you want for cases where you're concerned you might be overfitting your data (i.e. using a fit function with more parameters than you can justify) or where the error bars on different data points are very different from one another (as happens, for example, when Poisson noise dominates). Chi2, and especially reduced chi2 (i.e. chi2 per degree of freedom) are probably more appropriate for these cases.


Why some people add results from PCA and other dimensional reductions techniques as features

This is feature engineering. You just give the algorithm another look at the data, from another point of view. It often helps to understand better data when you have different point of views.

For example, let's assume you want to learn the US road directions to a simple ML model. You give him all examples from roads 1 to 100 mapping to 0 (if the road is east-west), or 1 (road is north-south), except for the number 47. Then you ask the model to predict 47 and it will answer 0 because it's between 46 and 48 who are labelled 0.

Now if you give the model another point of view (here, the parity of the road number), then it will obviously learn efficiently the road directions.

You can view the results of the PCA as the parity, just another view of the data to understand it better.


Sklearn Random Feature Importances Identical for Predicting Different Response Variables

It seems that your self.clf points to your Method. At the end, you are probably printing the features importance of a unique classifier.
Maybe you should copy it:
from sklearn.base import clone

class NoCVMethod:
    [...]
    self.clf = clone(Method) # only copy the estimator
    # OR
    self.clf = deepcopy(Method) # if you want to also copy the data estimator

See here (or here as you suggested) for more details about copying an sklearn estimator.


How to interpret this 'Zipf' (survival, war casualties) plot?

As you look at the chart, you can see the vertical axis represents some kind of Survival Function and horizontal axis defines War Casualties. As you can see, the higher the magnitude of Was Casualties, the lower the Survival rate/function and vice-versa. The higher the Survival rate the lower casualty count. Each dot represents each recording. Most likely, you need to look into how Survival Function is defined to really get a good understanding.


finding the mean for each of the channels (RGB) across an array of images

Could probably be more elegant, but here's an idea:
mean[:, :, 0:2] = np.mean(imgs[:, :, :, 0:2], axis=0)


Distance between very large discrete probability distributions

Yes, plenty.

Get the book "encyclopedia of distances".

For example, you can use Histogram Intersection distance. Since your data is already normalized, that reduces to Manhattan distance, if I am not mistaken. Yes: this can be appropriate for distributions.


Difference between "reducing batch_size" and "increasing epochs" to decrease loss amount?

The concept of overfitting does not generalize over a specific combination of batch size and epochs. It depends on your data and the architecture of your model

A friend of mine ran into these scenarios with a CPU based image classifier:

1) If I use more epochs ,it may take me a lot of time to come to a desirable
   outcome.

2) If I prefer small batch-sizes over small epochs , It might take less time
   time to compute , but not reach the desirable outcome by that epoch limit.

I used a GPU and my results were different. Using low epochs , and better convolutional architecture , I reached a better accuracy with not so small batch sizes.

I increased epochs, and my accuracy improved until I felt I reached overfitting.
I increased batch-sizes and my accuracy was not increasing at a decent rate.

I had to come to balance at which my model was acceptable.

Its a balance , yes. A balance which I am afraid, the designer needs to take care of. Not always they are inversely proportional. The data set and the architecture makes all the difference in this debate I am afraid.


Your data boils down to something like this structure:

> str(model)
List of 2
 $ data:List of 1
  ..$ : int [1:3, 1:4] 1 2 3 4 5 6 7 8 9 10 ...
  .. ..- attr(*, "dimnames")=List of 2
  .. .. ..$ : NULL
  .. .. ..$ : chr [1:4] "a" "b" "c" "d"
 $ foo : num 1

but you have some other components that we don't need to bother with, and my data doesn't have row names and is a lot smaller.

model is a list of 2 (for me) and 13 (for you) parts.

The $data component is also a "List of 1" component.

So colnames(model$data) is trying to get the colnames of a list, and failing:

> colnames(model$data)
NULL

Which you would have spotted if you'd tried running colnames(model$data) yourself.

You want the colnames of the first element of the list model$data:

> colnames(model$data[[1]])
[1] "a" "b" "c" "d"

and hence:

> sample(colnames(model$data[[1]]),1)
[1] "b"

Its possible that because this is a "kohonen" class object that there are functions that get these data matrices for you. You'll need to read the documentation to figure this out. What I've shown above is digging in the structure to find the data you want.


I suggest you use ANTLR, which is a very powerful parser generator. It has a good GUI for entering your BNF. It has a Python target capability.


You can get the tags for the headline or news content using NLP. Tags are generally noun chunks, entities or the action mentioned in the headline. You need to perform standard steps of text processing i.e remove stop words, tokenize sentence into words, do stemming or lemmatization then parse it to extract information. Most of these are already implemented. Look into spaCy or use NLTK.

You can also look into SVO (Subject, verb, object model) to assign tags. SpaCy provides an easy way to do this, do check it out.


Do Recurrent Neural Networks assume stationarity or just a general kind of sequential dependence?

The distribution is stationary, but it doesn't mean the time series should be stationary.
Let's imagine the recurrent network, that returns the n+1 if it was n on previous step. Distribution is not changing with time, but the time-series always goes up.


Effects of slipperiness in OpenAI FrozenLake Environment

According the documentation in that repo:

However, the ice is slippery, so you won't always move in the direction you intend.

That implies is_slippery increases the noise. Specifically, the transition matrix will be effected. It makes sense that your model takes longer to learn when there is an increase in noise between intended and actual action.
There are appears to be many hard-coded numbers in that example. It appears a person with deep knowledge of that specific environment added constants to account for the increase in noise.


Finding clusters in multidimensional data

In general it does not make much sense to cluster features. In an ideal world for your features to be the best they can be they should actually be independent, thus there should be no relationship between them. Typically when we talk about clustering it is clustering the instances. To attribute some associative labels to a subset of the instances based on the similarity of their feature values.

Many clustering algorithms exist, I would say that the most popular is K-means however spectral clustering and Gaussian mixtures are also frequently used. As always, each algorithm is best suited for a specific type of dataset, it is up to you to choose which is best suited, or you can just try all of them and see which is best.

Here you can find a list of clustering algorithms with their respective usecases. Always use the libraries when you want to implement standard algorithms, they are highly optimized. But for education sake it is good to look at what is happening.

I will describe a homebrew version of the K-means algorithm such that you can understand what is happening under the hood and perhaps you will see why we cluster instances and not features.

The K-means algorithm

import numpy as np
import matplotlib.pyplot as plt

First we will make some artificial data. These will consist of $n$ 2D Gaussian clusters with a given mean and variance. Here $n=5$ and we will have 300 instances per Gaussian distribution.

params = [[[ 0,1],  [ 0,1]],
          [[ 5,1],  [ 5,1]],
          [[-2,5],  [ 2,5]],
          [[ 2,1],  [ 2,1]],
          [[-5,1],  [-5,1]]]

n = 300
dims = len(params[0])

data = []
y = []
for ix, i in enumerate(params):
    inst = np.random.randn(n, dims)
    for dim in range(dims):
        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]
        label = ix + np.zeros(n)

    if len(data) == 0: data = inst
    else: data = np.append( data, inst, axis= 0)
    if len(y) == 0: y = label
    else: y = np.append(y, label)

num_clusters = len(params)

print(y.shape)
print(data.shape)

  (1500,) (1500, 2)

So now we have 1500 instances (records) in 2D space. This can be extended to any number dimensions. 2 is easiest to plot.

plt.scatter(data[:,0], data[:,1])
plt.show()

I will paste the entire algorithm at the bottom of the answer for a quick copy and paste but here I will go through its different parts so you can see how it works. The algorithm goes as follows: first we initialize some centroids within the range of our data. These are the red dots in the image below

def train(self, data, verbose=1):

        shape = data.shape

        ranges = np.zeros((shape[1], 2))
        centroids = np.zeros((shape[1], 2))

        for dim in range(shape[1]):
            ranges[dim, 0] = np.min(data[:,dim])
            ranges[dim, 1] = np.max(data[:,dim])

        if verbose == 1:
            print('Ranges: ')
            print(ranges)

        centroids = np.zeros((self.k, shape[1]))
        for i in range(self.k):
            for dim in range(shape[1]):
                centroids[i, dim] = np.random.uniform(ranges[dim, 0], ranges[dim, 1], 1)

        if verbose == 1:
            print('Centroids: ')
            print(centroids)

            plt.scatter(data[:,0], data[:,1])
            plt.scatter(centroids[:,0], centroids[:,1], c = 'r')
            plt.show()

Then we will calculate the distance from each instance (record) to each centroid.

distances = np.zeros((shape[0],self.k))
for ix, i in enumerate(data):
    for ic, c in enumerate(centroids):
        distances[ix, ic] = np.sqrt(np.sum((i-c)**2))

Then we will attribute each instance as belonging to the centroid it is closest to.

labels = np.argmin(distances, axis = 1)

Now we will update the position of the new centroids by finding the mean position of all instances which are closest to the given centroid in each dimension.

new_centroids = np.zeros((self.k, shape[1]))
for centroid in range(self.k):
    temp = data[labels == centroid]
    if len(temp) == 0:
        return 0
    for dim in range(shape[1]):
        new_centroids[centroid, dim] = np.mean(temp[:,dim])

We then repeat this process until the centroids no longer move significantly. Usually if the difference in position for all the centroids is less than machine epsilon we consider the algorithm to have converged.

if np.linalg.norm(new_centroids - centroids) < np.finfo(float).eps:
    print("DONE!")
    break

For this dataset it took 16 iterations for convergence, this is the final result.

What we see here is that each instance is grouped into a cluster with similar attributes as itself. For example if the dimensions of our data represented height (x-axis) and weight (y-axis), then we can group people into 5 different BMI indices.

Purple: short and skinny
Green: short and fat
Blue: mid height and mid weight
Yellow: Mid height and fat
Turquoise: Tall and mid weight.

Each member in our population belongs to a specific BMI index which is clustered based on two measurable attributes he possesses (features), his height and his weight.

The full K-means algorithm

This is the algorithm

class Kmeans(object):

    def __init__(self, k=1):
        self.k = k

    def train(self, data, verbose=1):

        shape = data.shape

        ranges = np.zeros((shape[1], 2))
        centroids = np.zeros((shape[1], 2))

        for dim in range(shape[1]):
            ranges[dim, 0] = np.min(data[:,dim])
            ranges[dim, 1] = np.max(data[:,dim])

        if verbose == 1:
            print('Ranges: ')
            print(ranges)

        centroids = np.zeros((self.k, shape[1]))
        for i in range(self.k):
            for dim in range(shape[1]):
                centroids[i, dim] = np.random.uniform(ranges[dim, 0], ranges[dim, 1], 1)

        if verbose == 1:
            print('Centroids: ')
            print(centroids)

            plt.scatter(data[:,0], data[:,1])
            plt.scatter(centroids[:,0], centroids[:,1], c = 'r')
            plt.show()

        count = 0
        while count < 100:
            count += 1
            if verbose == 1:
                print('-----------------------------------------------')
                print('Iteration: ', count)

            distances = np.zeros((shape[0],self.k))
            for ix, i in enumerate(data):
                for ic, c in enumerate(centroids):
                    distances[ix, ic] = np.sqrt(np.sum((i-c)**2))

            labels = np.argmin(distances, axis = 1)

            new_centroids = np.zeros((self.k, shape[1]))
            for centroid in range(self.k):
                temp = data[labels == centroid]
                if len(temp) == 0:
                    return 0
                for dim in range(shape[1]):
                    new_centroids[centroid, dim] = np.mean(temp[:,dim])

            if verbose == 1:
                plt.scatter(data[:,0], data[:,1], c = labels)
                plt.scatter(new_centroids[:,0], new_centroids[:,1], c = 'r')
                plt.show()

            if np.linalg.norm(new_centroids - centroids) < np.finfo(float).eps:
                print("DONE!")
                break

            centroids = new_centroids
        self.centroids = centroids
        self.labels = labels
        if verbose == 1:
            print(labels)
            print(centroids)
        return 1

    def getAverageDistance(self, data):

        dists = np.zeros((len(self.centroids),))
        for ix, centroid in enumerate(self.centroids):
            temp = data[self.labels == ix]
            dist = 0
            for i in temp:
                dist += np.linalg.norm(i - centroid)
            dists[ix] = dist/len(temp)
        return dists

    def getLabels(self):
        return self.labels

To use the algorithm use the following, where data is artificial data we made above but can also be any numpy matrix where the records are the rows and the features are the columns.

kmeans = Kmeans(5)
kmeans.train(data)


It is generally not best practice to trust ML techniques to do the work for you. A data scientist knowns on what principles the methods that he or she applies work, and then stills tries multiple in the same setting, to see which methods best apply to the domain at hand. How well certain measures work can be a clue to the structure of your data.

Right off the bat it is important to understand that Weka offers 3 kinds of feature selection methods: Filters, a Wrapper and PCA. Filters apply criteria (mostly based on a measure of how it divides a set) for feature suitedness (variations on how well they divide the search space) , wrappers use a learning algorithm for their evaluation and PCA doesn't so much select features as that it recombines features to maximize variance in the dimensions.

That being said: I suggest you start looking at InformationGainEval a method that uses information gain to select attributes. It is quite popular and easy to understand. You can find a good explantation here:
https://stackoverflow.com/questions/33982943/how-the-selection-happens-in-infogainattributeeval-in-weka-feature-selection. Than work your way through the others (the documentation is accessible from the GUI).


Fine tuning CNN with imbalance data gives good results - not sure why

Are you getting good results on your training or your test set? If it's only the first then you are overfitting, but if it's the second it will not have to do with the fact that your whole training set fits in your batch, the gradients still have a strong pull towards the dominating class(es). But having the pretrained weights will help a lot if the classes are relatively easy to seperate. It will also help if your dataset is similar to the one it was pretrained on, because the intermediate layers that do feature extractions have learned valuable filters for your problem instance.


Why keep vocabulary and posting list separate in a search engine

It has many reasons to be (performance, design, storage, compression, evaluation of data structures). The principal reason is that all structures are verified in the practice, but you can make you own data structure and show a new mode to do.

Even the google has a paper for verify that he works fine, if you has your own data structure and design, I suggest you choose a database with the correct size for your experiment, search for the precision and recall values for your own collection and make it.

Other reasons that you can see is that the information has different requirements about performance, compression, storage and hardware.

The cost of gigabyte for main memory is very different between RAM and HD, when you have a lot of servers your strategy of cost reduction is a reason of improve low cost storage and no high hardware requirements.

It's clear when you have a collection with Terabytes of data, many cultures or many countries. (google has a paper about his cluster).

Think simple, a collection with 50GB is a small collection, but how much cost a server with 60GB of RAM and how much cost a server with 50GB of HD?

The answer is clear and the it's reflects in the data structures (B-tree is good for secondary memory and simple hastables are good and fast in main memory).

But you decide.

A old good reference is the chapter 5 of Managing Gigabytes (https://www.amazon.com/Managing-Gigabytes-Compressing-Multimedia-Information/dp/1558605703/ref=sr_1_1?s=books&ie=UTF8&qid=1474990622&sr=1-1&keywords=managing+gigabytes)

The chapter shows a table with different results for many data structures and types of memory.

The google's paper shows the mode for verify your implementation and design:
http://infolab.stanford.edu/~backrub/google.html

Think about the cluster and jobs:
http://static.googleusercontent.com/media/research.google.com/pt-BR//pubs/archive/43438.pdf


What is the best way to encode an arbitrary collection of strings into int categorical variables?

Let's see. You want to be able to work with unspecified number of categories.
There are multiple ways to work with this.

Create a special category "other" and put all categories which are very rare into this category during processing. When you encounter a new category, put it into this "other" category. And you can use any common preprocessing - label encoding, one hot encoding and so on. This way you'll be able to make predictions for new categories. And when you have enough data for them, you can leave then as they are and refit you preprocessing and the model.
Target encoding. There are multiple ways to convert categories to float numbers. You train target encoding on the data which you have and can apply it to new categories (usually unknown values are assigned a global mean value).


Does out-of-bag sampling make Random Forests inherently less robust than other classifiers?

The out-of-bag-error is calculated from the samples that are not used anyway for the particular tree. The original set of records gets bootstrapped. So, a new set is generated that does not contain all samples anyway. The out-of-bag set can then be used to monitor the performance of the model. When the out-of-bag-error goes up (given it has a significant size) it means that the current tree is over-fitting the training sample. So, out-of-bag sampling can be used to prevent over-fitting and therefore makes the model more robust rather than less robust.

So, in fact all samples are used to train the random forest model. Though each tree only uses a subset of the dataset at a time. Don't confuse the RF model with the individual trees!

Look here for a more detailed description.


k-fold cross-validation: model selection or variation in models when using k-fold cross validation

There are two main (set of) things your model needs to learn from training: Parameters and Hyperparameters.

Parameters are those that can be inferred (learnt) from data. These are internal to the model and will be calculated in the training phase during. The hyperparameters however, cannot be learnt from data and need to be decided (make assumptions) before training. As you come from statistics, an example could be modelling a distribution from a set of samples. Parameters could be mean and std, and hyperparameter could be selecting a Normal distribution or a Beta distribution.

If you don't have hyperparameters (e.g. simple regression model), you only need to train the model and let the algorithm find the optimal weights (parameters) using the data. However, if you want to use let's say Support Vector Machines, then you need to provide some hyperparameters such as the cost, kernel, parameters of the kernel, and so on... before starting the training.

Cross Validation can be used to find the optimal hyperparameters. It is done by creating a grid of potential hyperparameter values and training each combination of them against the whole data, performing k trainings of k-1 portions (or folds) of data. For each specific combination of hyperparameters the model is then trained and evaluated k times, getting a number of predictions equal to the number of folds (k) for each hyperparameter combination. The combination that yields better results based on the specified metric, is considered the best option (final model).

Have a look at this nice blog for further clarification.

Cross validation procedure:


If you want to transform a data array, for example, X_train, with a preprocessor like SimpleImputer you should use method .fit_transform.

For example: X_train = fill_values.fit_transform(X_train)

If you use only fit it will return an instance of type SimpleImputer which is your error.


Autoencoders are meant to reduce the dimensionality of your data. Increasing the number of filters would do the opposite.


As it follows, two pipelines should be used:

num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:120,121:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,120]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))
cat_indices = [94, 96, 98, 99, 120]

from imblearn.pipeline import make_pipeline

pipeline=Pipeline(steps= [
    # Categorical features
    ('feature_processing', FeatureUnion(transformer_list = [
            ('categorical', MultiColumn(cat_indices1)),

            #numeric
            ('numeric', Pipeline(steps = [
                ('select', MultiColumn(num_indices1)),
                ('scale', StandardScaler())
                        ]))
        ])),
    ('clf', rg)
    ]
)
pipeline_with_resampling = make_pipeline(SMOTENC(categorical_features=cat_indices), pipeline)


How is the property in eq 15 obtained for Xavier initialization

The fixed answer of $1/3$ is a result of their decision to use the uniform distribution along with the parameterised arguments, namely $1/\sqrt{n}$.

For a uniform distribution, denoted with lower and upper bounds $a$ and $b$:

$$
U(a, b)
$$

the variance is defined as:

$$
\frac{1}{12} (b - a)^2
$$

So in the case of the authors, Glorot and Bengio, the two bounds are simply the square root of the number of neurons in the layer of interest (generally referring to the preceding layer, as they put it). This size is called $n$, and they set the bounds on the uniform distribution as:

$$
a = - \frac{1}{\sqrt{n}}
$$
$$
b = \frac{1}{\sqrt{n}}
$$

So if we plug these values into equation 15, we get:

$$
Var = \frac{1}{12}(\frac{1}{\sqrt{n}} - -\frac{1}{\sqrt{n}})^2
$$

$$
Var = \frac{1}{12}(\frac{2}{\sqrt{n}})^2
$$

$$
Var = \frac{1}{12} * \frac{4}{n}
$$

And so finally:

$$
n * Var = \frac{1}{3}
$$


What are the online platforms that cater to freelance data scientists looking for work?

Stackoverflow has a jobs page. This link is to a search with keyword data and option Remote.
freelandr offers you to `Land your next Freelance Job'.
Here are two specialised job agencies, although I don't know if they relay freelance opportunities.

  We are well aware of the data scientist being the sexiest job of the 21st century. JamieAi harnessing this fact connects technical talent with data-oriented jobs organizations of all types and sizes. The start-up firm has combined AI insights and human oversight to reduce hiring costs and eliminate bias.  Also, third party recruitment agencies are removed from the process to boost transparency and efficiency in the path to employment. Another example is Woo.io, a marketplace for matching tech professionals and companies.
  https://hub.packtpub.com/25-startups-machine-learning-differently-2018/

R-users specialises in the language R.


Which convolution should I use? Conv2d or Conv1d

When using Conv2D , the input_shape does not have to be (1,68,2). The number of samples does not have anything to do with the convolution, one sample is given to the layer at each time anyway.

What changes is the number of spatial dimensions of your input that is convolved:

With Conv1D, one dimension only is used, so the convolution operates on the first axis (size 68).
With Conv2D, two dimensions are used, so the convolution operates on the two axis defining the data (size (68,2))

Therefore you have to carefully chose the filter size. For instance, if you chose a Conv2D with a filter size (4,2), it will produce the same results as a Conv1D with size (4) as it will operate fully on the second axis of data.

Finally, there is no response to what is the best method. Generally, Conv2D work well on images and Conv1D on text. Given the size of your second dimension of data, Conv2D does not seem to make a lot of sense hence Conv1D should work well.


Isn't one-hot encoding a waste of information?

Good idea but...

You encode not just to transform from categorical to numerical features but to give that information to your model.

Let's say that you do that and feed it through a linear model to try to predict the price. Let's say Pear is really expensive(500€) and Melon cheap (1€).

Your model coefficients with one hot encoding will be:

$price = 500 * Pear[0,1] + 1 * Melon[0,1]$

If you do your encoding the linear combination won't work. What will be the coefficients?

One could argue that with decision trees this won't happen since it can make splits... but it would have to make two splits before determining if it is a melon (a greedy decision tree won´t do it) so again you will lose computational power here too.

You could try to run the experiments and see if this is your result. In the end, this should be science and one can do experiments to prove a hypothesis.

On the other hand, Yes,  One Hot Encoding increases the computational time and memory easily since you are creating a lot of features from one and if it has high cardinality you can end up with a lot of features.


Visualize graph network with more than 30k edges

Yes There are!

Networkx

I think 20k-30k node-edge would be OK on Networkx, IF YOU HAVE A GOOD MACHINE! Networkx is a great library in Python particularly for Graph Analysis so you have access to great analysis tools beside visualizing but visualizing 20k vertices needs much RAM and takes long.

Igraph

Igraph is another great tool for Graph Analysis with APIs in Python, R and C. You can do a lot with Igraph library including nice plotting facilities.

SNAP

SNAP was developed by Jure Leskovec and his colleagues at Stanford. A large scale graph processing library natively written in C with API in Python. This library can handle huge graphs on a single personal machine.

PS.1 Both of these guys have interfaces with different graph softwares like Gephi, etc.
PS.2 For large graphs I strongly recommend to work in C++ if you can. Speaking as a Network/Graph guy, C++ solves many problems out there!


Perceptron - Which step function to choose

They have the same meaning in this context although during training using Rosenblatt update rule, the former may have great changes during each update. Perceptron is used for binary classification which means there are two possible classes to classify. If the result of inner product, here dot product, is greater than or equal to zero, the class of inputs will be the first class and if it's smaller than zero the class of inputs would be the other class. Perceptron just has one neuron. It's a simple linear classifier. The value of threshold is just important. Means that if the product is greater than zero, the input belongs to e.g. positive class and if is negative it belongs to negative class. The step functions and Rosenblatt update rule are not used any more. They have so much oscillation. Today networks learn using gradient descending algorithms which uses the concept of derivative.

When you progress, you will see that neural nets which use other activation functions like Sigmoid or Tanh are different. the former has 0.5 expected value and the latter has 0 expected value which causes the second learn so much faster. Although now a days ReLU is more popular among other activation functions.


How can you determine the growth rate you need to achieve a certain customer base?

A growth rate is an exponential growth, you multiply your userbase by a certain fraction, which is above 1 in case of growth, else it would be decline. This effect compounds because the amount of users is bigger while being multiplied by the same multiplier. Let's call this multiplier a. We apply this 12 times, once for every month. We are interested in:

$$12000 \cdot a^{12} = 1500000$$

This equals:

$$a^{12} = 125$$

We can take the 12th root on both sides:

$$a = 1.495$$

This means you would need to grow approximately 50% every month.

For growing faster first and then plateauing you can do similar equations, although a bit more difficult. You have to express the growth at the first level in terms of the other level because else there are two unknowns which make for infinite solutions. Let's say the first 3 months the growth b is five times as high as the growth a last 9 months. This means $b = 5(a-1)+1$. This leads to the following equation.

First we have three months of growth rate b, which means after three months our userbase is $12000\cdot b^3$ and after that we have nine months of growth a which leads to the final user base being $12000\cdot b^3\cdot a^9$. Since $b=5(a-1)+1$ this comes down to $12000\cdot (5(a-1)+1)^3\cdot a^9$, which according to Wolfram Alpha comes down to a being around 1.2785 which means b is around 2.3925. This means the first three months a growth of 139% and then nine months of 27.85% growth.


sklearn.neighbors.NearestNeighbors - knn for unsupervised learning?

The unsupervised version simply implements different algorithms to find the nearest neighbor(s) for each sample.

The kNN algorithm consists of two steps:

Compute and store the k nearest neighbors for each sample in the training set ("training")
For an unlabeled sample, retrieve the k nearest neighbors from dataset and predict label through majority vote / interpolation (or similar) among k nearest neighbors ("prediction/querying")

The unsupervised version is basically only step 1, the training phase of the kNN algorithm.

(This is useful because if your dataset is large, a pairwise comparison for all samples (algorithm='brute') is often infeasible. Therefore two alternative algorithms for the training stage are implemented that make use of previous comparisons to reduce the number of distance calculations. See the documentation here.)


Why are RNN/LSTM preferred in time series analysis and not other NN?

I'll try to provide some insight which will hopefully help.

Can a normal NN model the time connections the same way like a RNN/LSTM does when it is just deep enough?

Every neural net gets better in theory if it gets deeper. For a regular NN to model time connections properly, you could use the last n time steps as your input and the n+1th time  step as your target. This will generate your training set and depending on your data, you could be able to model your time series fairly efficiently.

All of the most obvious pitfalls of this approach are actually addressed by RNNs/LSTM.

Does an RNN need more or less data in comparison to a NN to get the
same/ better results?

Difficult question, which would probably require some empirical results to check that theory. Also, with neural nets, sometimes, it's more about how fast it trains rather than how much training data it has that will make the biggest difference in performance.

To me, the main difference is that your regular NN will need a fixed-size input, whereas your RNN will be able to learn with input "up to" a certain size, which can be a big advantage to model the entire time series well.

Are there time series where normal NN or RNN/LSTM perform better?

Again, this is a difficult question as it will depend on the data, the architecture of the networks, the training time etc.

I haven't done empirical research on this, but I'd say that your best guess would be time series that are rather short (less than 100 time steps for instance). If the time series is short, you might not need to model such an intricate relationship through time, which a regular NN could perhaps do as well as an RNN.

Is it time data depended which Model will perform the best or are there some guidelines?

As explained above, every learning exercise will highly depend on the architecture and hyperparameter used, even the initialization of your weights, whether you use pre-training or not, and of course, your data. Again, I think that the shortest the time series, the more competitive regular NNs could be. But again, this is merely intuition-based and hasn't been checked thoroughly.

- Can the NN behavior be understood better than the RNN behavior?

It all depends on what you mean by understanding. RNN have more weights than NN so ultimately, there will be more things to analyze and eventually understand. But, with more data also comes more information, so perhaps there is more information to be gained from an RNN than a simple NN, even if it's a deep one.

Plus, sometimes, based on the initialization of the weights and other parameters, the interpretation of the model could vary, even if it's trained on the same data.


I would say that they are probability distributions. You can interpret them as the probability that a randomly drawn person from a given county belongs to the white majority. However, they are not probability vectors, as this would require them to sum up to 1. It's more that each element of the array defines a distribution, e.g a random person from county 1 is white with a probability of 0.52 and a member of a minority with probability 1-0.52=0.48. If you want to compare this distribution with your simulation, you would have to make the comparison element-wise, e.g. compare the 0.52/0.48 actual distribution with the 0.48/0.52 simulated distribution. This is also why reshuffling leads to different results: suddenly you are comparing the actual distribution from county 1 with e.g. the simulated distribution from county 5. This does not make much sense. So if you shuffle, you need to shuffle both arrays in the same way.

As you can interpret the distributions in each of the counties as being independen from each other, you can compute DKL or cosine or Earth's Movers for each county and sum them up probably.

I'm not sure which method is the best. I would probably go with DKL out of habit. You could compute it as such

dkl = 0.52*log(0.52/0.48) + (1-0.52)*log((1-0.52)/(1-0.48)) + ...

I hope this helps!


I'm not sure how you are applying a regression framework for document classification. The way I'd approach the problem is to apply a standard discriminative classification approach such as SVM.

In a discriminative classification approach the notion of similarity or inverse distance between data points (documents in this case) is pivotal. Fortunately for documents, there is a standard way of defining pairwise similarity. This is the standard cosine similarity measure which makes use of document length normalization to take different document lengths into account.

Thus, practically speaking, in cosine similarity you would work with relative term weights normalized by document lengths and hence document length diversity should not be a major issue in the similarity computation.

One also has to be careful when applying idf in term weights. If the number of documents is not significantly large the idf measure may be statistically imprecise thus adding noise to the term weights. It's also a standard practice to ignore stopwords and punctuations.


Solution to my question: If you run a tensorflow session you can choose if the placement of nodes on the different devices is hard or soft coded. In the specific case of CIFAR10 this means:

The Queues be placed on the GPU and that's why the error popped up, but:
If you turn on soft placement it works. This means, that even though the placement on the GPU of several parts of the model can't be done, Tensorflow solves this problem automaticly by placing them somewhere else
I don't actually know where the nodes get placed. I guess on the CPU, but I'm not sure. In tensorboard the color of the input pipeline nodes claims a placement on the GPU. Very confusing!

The soft placement can be done by starting the session like this:

sess = tf.Session(config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True))

I found this on the following tutorial page: https://www.tensorflow.org/tutorials/using_gpu


Should the weights for CrossEntropyLoss be exactly the inverse of the propotions of training data?

I'll refer to an answer for a similar topic. If you have enough data for classes 2 and 3, there's no reason to change your training scheme if you use standard metrics. The baseline should always be training without changing the weights, and if you see that the model does very bad on classes 2 and 3, you can change the training scheme. However, I have rarely seen it work better.


Is data subsampling appropriate for hyperparameter optimisation?

Your subsample has to be representative of your original dataset.
To do so, as you are in a supervised case, I would get a random subsample that keeps the classes distribution (for instance getting randomly 40% of each class).
Note
If you have classes with too few examples, I would also try not to sample them. Risk is even with random sampling you could loose information when a cluster is too small. Plus, if your problem is computation time, that won't be a problem to keep the too small clusters while sampling the bigs.


What if irrelevant features impact the outputs?

It would mean that the model trained with some irrelevant features is overfit: a good model uses only features which have some predictive power on the target variable, so if the sample is representative enough any irrelevant feature is ignored or assigned very little weight. However if the training dataset is too small or the model too complex then it uses details which happen by chance, so it could use some irrelevant features: this is overfitting.
In case the performance on the test set was high despite the model being overfit, it's likely that there is some data leakage additionally.


Better to use Collab indeed. Kaggle also provides notebooks with 38h GPU and also 30 hours of TPU per week you might want to have a look at that as well (plus Kaggle allows you to use your GCP credentials so you can link private google cloud storage buckets to your Kaggle notebook). On Kaggle you will also find plenty of public notebooks that can be of great help


1) YES. the updates won't be visible not until your restart your webservice. ( Both by the way ).

2) Gunicorn works well with Flask as a wsgi app. I'm not sure how to answer your question. All i can say is that i have webservices deployed to production with both Gunicorn+ Flask and Flask only, and they are both accessible over internet. Only difference is that Flask is not meant for production deployment, only for testing purposes.


What is the standard for dealing with null values in plotting?

Imo, It depends on two major factors:

What is the purpose of the plot
Does the null value indicate incompleteness (one of the values, but unknown which one), or does it indicate a last final option, (like: multiple colors)

Imagine if the purpose of the plot is to compare which toy colors are most popular to a particular person/group.

If null indicates multi-colored toys, it may be important to include the data but rename it to something more meaningful. (Multi-colored)

If null indicates that we just don't know the color data for those toys, it would probably be best to exclude that data from the plot. However, with such a large percentage, it would be good to note that "x%" were excluded from your plot because they had no color information associated with them. Perhaps purple is in such a majority because this person/group really likes purple, or maybe its because purple toys almost never made it into the null category for whatever reason.

The same line of thinking should be applied to whatever the purpose of the plot is. If the null values are relevant to the purpose, include them. If not, don't - and maybe take a note of how much data is excluded if it could influence the results substantially (such as this case)


Justification for values used in backpropagation

The objective of back-propagation is to isolate the effect of each weight on the total error. Once the effect is isolated, each weight can be changed individually such the total error is minimized.

During back-propagation, the effects of other weights are automatically isolated. Take, for example, the function:

$$
E_{total}(x) = w_1x + w_2x^2 \\
$$

Then

$$
dE_{total}(x)/dw_1 = d(w_1x)/dw_1 + d(w_2x^2)/dw_1 \\
dE_{total}(x)/dw_1 =x + 0
$$

So, you see, back-propagation ignores the effects of other weights by definition of differentiation. When you back-propagate with respect to $w_1$, all other $w$s are considered constant.

$dE_{total}/dw_i$ is the change in error as you change a particular weight $w_i$ and only that weight. So it makes sense that you would want to change $w_i$ in the opposite direction of that change i.e. $-dE_{total}/dw_i$.


How to start with receipt OCR text detection

Not sure if you tried Tesseract or pytesseract yet. I never worked on receipts but I had a different use case where I needed to segment images before I applied the tesseract for text recognition. For simple use case and especially in English, tesseract is quite powerful so I would start with the 2nd option first and see if it's good enough. Probably you may look at this answer on stackoverflow for idea.  https://stackoverflow.com/questions/55140090/pytesseract-reading-receipt


Resizing images for training with Mobilenets

Maintaining aspect ratio is important or not.

Yes, it's really important in most cases. As you can read from here, Why does aspect ratio matter? It’s all to do with the relationship of the main subject to the sides of the frame, and the amount of empty space you end up with around the subject. An awareness of the characteristics of the aspect ratio of your particular camera can help you compose better images. It also helps you recognise when cropping to a different aspect ratio will improve the composition of your image.

In deep learning tasks, it depends how you want to feed data to your network. It's better to train your network with real data. Consequently if you are going to face data with standard aspect ratio, you have too keep it during training.

  If yes (224,224) is a good choice or should I set it to higher resolution.

Depending on your task 224 may be good or not. In typical classification networks, it is an acceptable size, like the inputs of AlexNet or VGG and such customary nets. For localization tasks, not really. For instance, the input height and width of YOLO is more than that 1.


Determine the most important documents for supervised learning

This is commonly called active learning.
The most common active learning approaches for text classification are random, data-based, model-based, and predication-based.


How to keep track of assumptions/design choices and avoid errors in complex projects

I would go for parametrised Rmarkdown reports. I think it adresses all your needs, it's simple to use and maintain, offers great flexibility, you can embed all kinds of things and it offers different output formats.


Algorithm for classification of words into given categories

The task you are asking about is Named entity recognition

A good way to identify a word category is by using patterns for the context of the word.

In your case you can use patterns like:

"I took X" for a medicine name
"I had pain in my X" for a body part
"I suffered from X" for a disease

The easiest and fastest way is to discuss the pattern with a domain expert and do a sanity check on the data you have.

A more systematic approach is to deduce the pattern from your data.
I guess you can get a list of samples of the items.
The technical details depend on the patterns you use but for this example let consider tokenization.
Take the context of k-words next to the items on the list. Compare it to the context of other words. Contexts that are related to your items are suitable as patterns.

Bare in mind that these pattern will have very different precision and recall and they will lead to errors. It is important that you will have a labeled dataset so you will be able to estimate the patterns performance.

It will be fairly easy to get pretty good classification. It might be very hard to get a very good classification.


Does the performance of neural networks depend on the method used to unroll weights ?

I don't really understand your notation and don't have a lot of experience with unrolling, but from my understanding there can only be one way of transforming your 2 weights (theta1 & theta2) to a single one (theta).

If you adapt your python code to something like:

theta = np.array(theta1,theta2)
theta = np.ravel(theta, order='F')
#theta = [1, 4, 2, 5, 3, 6]

Does it work now? If so, you can't just choose how to "unroll" your weights.


Increasing minNumObj increasing accuracy in decision tree

Your results are more in line with my experiences.  Increasing the minimum number of observations per leaf decreases the capacity of the model, increasing bias and decreasing variance.  Tree models are often overfitted without some kind of complexity reduction, so this usually is a beneficial tradeoff in net performance.

I'd speculate that the papers' results are different because of their data's small size: 683 and 4601... that second one doesn't strike me as too small, but...


How to choose best model for Regression?

I think a better option will be that you try different regression techniques (it will be good for your learning curve) and see advantages and disadvantage.

For subset selection purposes few options are:

1- Best subset selection : in which you start with a null model and fill all possible combination of models and select best model on the basis of AIC, BIC or adjusted R2.

2-Forward selection: You start with a null model as well and add feature one by one to see which one gives lowest RSS or highest R2. Then add another feature again until you have the best model.

3-Backward selection: you start with a model with all p predictors remove one predictor evaluate adjusted R2 or AIC, BIC. do it again and again until you have a satisfactory model.

You don't have to do it manually both R and python have library that will help you in this.


Scikit alternative for categorical data modeling?

There are definitely ways to process your data to make categorical data compatible with sklearn (e.g one-hot encoding). An alternative you can look into is h2o, which supports categorical features natively (although it doesn't offer the breadth of models of sklearn).


How to retrain the neural network when new data comes in?

Yes, there is some methods to do this based on the library. The proper keyword for this functionality is Checkpoint. Hence, you can check functions to save or restore the checkpoints from hard disk to continue learning or prediction in a new session. For example you can see the following in Tensorflow:

  The tf.train.Saver class provides methods for saving and restoring models. The tf.train.Saver constructor adds save and restore ops to the graph for all, or a specified list, of the variables in the graph. The Saver object provides methods to run these ops, specifying paths for the checkpoint files to write to or read from.

For more details about this in Tensorflow, you can follow this link.

To complete the case you can follow this subject in another library such as CNTK in this link.


How to estimate GridSearchCV computing time?

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through grid search.

E.g. suppose you want to use a grid search to select the hyperparameters a, b and c of your pipeline.

params = {'a': [1, 2, 3, 4, 5],
          'b': [1, 2, 3, 4],
          'c': [1, 2, 3]}

cv = GridSearchCV(pipeline, params)

By default this should run a search for a grid of $5 \cdot 4 \cdot 3 = 60$ different parameter combinations. The default cross-validation is a 3-fold cv so the above code should train your model $60 \cdot 3 = 180$ times. By default GridSearch runs parallel on your processors, so depending on your hardware you should divide the number of iterations by the number of processing units available. Let's say for example I have 4 processors available, each processor should fit the model $ 180 / 4 = 45$ times. Now, if on average my model takes $10 sec$ to train, I'm estimating around $45 \cdot 10 / 60 = 7.5min$ training time. In practice it should be closer to $8min$ due to overhead.

Finally, because some parameters heavily affect the training time of that algorithm, I would suggest using the max_iter argument whenever available so that your estimation doesn't fall far off.


I would assume this is a lot more complicated, because the multiple images might be off by a fraction of a pixel, or taken from a marginally different angle.

I agree with your analysis that this problem presents different challenges than single-image super-resolution.

In the single image case, we know exactly where everything should be in the output image, because it is just a higher resolution version of the input.

So why not try and extend this idea to your case? Here is how you might do that:

Concatenate each low-res image and a low-res version of the output image as each training and test example
For prediction concatenate each low-res image and duplicate one of them to specify the desired orientation of the output

So if you have n 3-channel h x w input images in each training example, your input shape should be (h, w, 3*(n+1))

This should give the network enough information to learn the correct transformation.


K-fold cross-validation with validation and test set

You can leave a portion of your dataset as a hold-out test set for the final model validation from the beginning, and proceed with the k-fold cross-validation strategy with the rest of the data as follows:

source: https://scikit-learn.org/stable/modules/cross_validation.html
This way, you do not have the overlap you mention and, mainly, you make sure that the model is finally evaluated with a never-seen before data.


Can I create pretrain model with tensorflow?

You don't need to train your model each time form the beginning. After training, you can store you model and next time you can load it, train it from that point or use it for your recall phase. You can find good tips here and here which have discussed storing and loading models in Tensorflow.


Since the StackOverflow link in the question comments seems broken, here is another reply that addresses the same question: https://stackoverflow.com/a/44789327/6470915

Direct quote:

  In fact, Blei (who developed LDA), points out in the introduction of the paper of 2003 (entitled "Latent Dirichlet Allocation") that LDA addresses the shortcomings of the TF-IDF model and leaves this approach behind. LSA is compeltely algebraic and generally (but not necessarily) uses a TF-IDF matrix, while LDA is a probabilistic model that tries to estimate probability distributions for topics in documents and words in topics. The weighting of TF-IDF is not necessary for this.

That sums it up on the high level. It would be interesting to understand more technically, why the model would perform more poorly if TF-IDF is used. Actually, there is another reply in the SO link which claims that LDA can be improved with TF-IDF.


Tool to Generate 2D Data via Mouse Clicking

I recently discovered this site: http://www.librec.net/datagen.html

Outputs list of points, and color ID (class) for each point.

Screenshot:


What can I do with a Decision Tree with poor ROC

Decision trees has one big quality and one big drawback. Their big quality is that they are what is known as glass-box models. What I mean by that is that they expose what they learn in a very clear and intuitive way. The name comes from the fact that you can see through a glass box. So, because of that, decision trees are very useful for analysis, they are a nice support to understand the relations between variables, which variables are important, in which way they are important. And even if they does not provide crystal clear information, they can bring you ideas about that. This might be very helpful especially if you have domain expert knowledge and you can put things together in a meaningful manner.

Their main drawback is their high variability. This problem is mainly caused by their greedy approach. Each decision in the first level nodes shape differently the tree. You can even go further and see that a single additional data point is enough in many cases to get a totally different tree, especially if the sample is small or if the data is noisy.

There are two types of approaches to solve this issue. The first type of approach tries to improve the single tree you built. This kind of approaches are known as pruning. A simple example would be reduced error pruning. This approach is simple and produce good results. You train your tree on a sample of data. Then you take another sample of data, you fit the new data on the tree, and then evaluate again the nodes of the tree from the perspective of the new data. If a non-leaf node get at least the same error if would not be split than if it would be split, then you can decide to cut the child nodes and transform that node into a leaf node. There are however much nicer pruning strategies, which are stronger, perhaps based on cross validation or some other criteria, mostly statistical based. Notice however that for reduced error pruning you need additional data, or to split your original sample in two, one for training, the other for pruning. If you go further to estimate the prediction error you need a third sample.

The second approach would be either to build multiple times some trees and chose some based on cross validation, bootstrapping or whatever method you you, or use a tree ensembles like bagging or boosting algorithms. Note that for boosting and bagging you loose glass box property.

Ultimately you have to choose between understanding and performance, having as a decent compromise the pruning procedure.


Reconstructing original data points from t-SNE output

There is no closed form.

It is a local embedding, and you really cannot expect to find a good inverse mapping.

It's doable, but not very good. Use gradient descent in the input domain. Just like tSNE but with the t-distribution and the Gaussian reversed. However, the original Gaussian has a sigma parameter that you also need to find. So you need to also optimize this to have the desired perplexity. You'll need to do the math for this optimization problem yourself.


Deep learning model performance for different datasets?

I can think of one reason. Because you're using different data sets, the properties of your data set are likely to be different. So obviously what the model will learn will be different. And maybe the architecture of your network suits one data set more than the other. But also if you try training the model on your other data set(with less accuracy) for more epochs, it's possible that the accuracy will improve. Your model could be just be slow to understand this data set.

It's like when you see a human you can easily recognize a face. But when you see a software code, it's not always easy to find the bug. But if you're trained well enough you'll eventually be faster and better in recognizing the bugs.

I hope it makes sense to you.


Association between continuous input and categorical output

Try logistic regression. The sigmoid function and its variants allow for conversion of continuous inputs into a binary outcome variable.


It depends a bit on your definition, usually the stochastic part of Stochastic Gradient Descent refers to the fact that you sample mini batches and estimate the true gradient with this sample. Dropout adds stochastic behaviour by sampling masks that comes down to sampling new network architectures every time. You could certainly see this as a form of stochastic gradient descent but whether this is the case or not depends solely on the definition.


Deciding which samples the model will (probably) classify incorrectly

You might want to frame your problem as an uncertainty estimation problem.
The idea is that you want to evaluate how comfortable your model will be when making a prediction. If your model is not very comfortable with a prediction (even if gets classified as predic_proba = 0.99), then the uncertainty prediction should be high.
What @BrianSpiering is proposing is a way to calculate uncertainty with neural networks. This method is known as Monte Carlo Drop Out as a Bayesian estimation.
The idea is to apply drop out regularization when doing predictions and doing several times. This way your model predicts a probability distribution in which you can calculate several statistics as standard deviation. This will let you know how comfortable is your model doing such prediction.
This is just a method but there are several ways to estimate uncertainty.
There are some blogs and papers that might help you:

Uncertainty in Deep Learning. How To Measure?.Deep Learning Model Says: “sorry, I don’t know the answer”. That can be absolutely OK
Simple and Scalable Predictive Uncertainty
Estimation using Deep Ensembles: From Google DeepMind at NIPS


In what data science applications has the stack exchange dump been used?

This is a Community Wiki answer, everyone is welcome to add references.

Kaggle ran a competition to predict whether a question would be closed on Stack Overflow: https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow


The implementation of overlapping clustering (NEO-K-Means)

I have never heard of NEO-k-means. So I wouldn't call it "state of the art": which tools does include it?

Plenty of methods are published every week and have zero impact because they are redundant, badly written, only work on a single data set, hard to parameterize, or outright irreproducible (As I have not read NEO-k-Means, I'm not claiming any of this applies to this particular algorithm!)

Nevertheless, it is not the first to do non-overlapping clustering.

Fuzzy c-means (FCM): a soft-asssignment k-means variant
EM (Gaussian Mixture Modeling) is even more powerful, and has a sound theorertical backing
Subspace and correlation clusterings are usually overlapping, too.

I'd start with GMM and FCM, as these have been successfully used by many people.


How to choose a model for this cross-validation curve?

Choosing the best validation accuracy is the common practice, since validation is unseen data.

Sometimes you might have over-fitting to the validation set, mainly if it is too small or no very representative of the data (for example if it has considerably more examples of one class, thus a good model would be a model that says that (almost) everything belongs to that class).

If you are worried about over-fitting, you could increase your regularization strength.


Should i drop simple features after deriving more complex features from them?

It will depend on the underlying relationship e.g.,
$Y = \alpha X_1 + \beta X_2 + \gamma X_1X_2$
In the above case, the Target is dependent on individual features as well as the Interaction features($X_1X_2$). So, you can't skip the individual.
Same will be True for polynomial too,
$Y = \alpha X + \beta X^2$
You may remove the original features for a scenario similar to this,
$Y = \alpha X_1X_2$ Or,
$Y = \alpha X^2$
Another intuition is that we never see a zero correlation to the original features nor we get a 100% correlation to the Interaction features.


Simple implementation of Apriori algorithm in R

I found an implementation here: Learning Data Mining with R by Bater Makhabel

The source code can be downloaded here: https://www.packtpub.com/big-data-and-business-intelligence/learning-data-mining-r


Naive Bayes for SA in Scikit Learn - how does it work

Collecting your data

From the comments you state that you wish to classify comments into a label (1-poor, 2-fair, 3-ok, 4-good, 5-very good). Thus you will be training a model that maps a set of words (the paragraph) to a number which represents the rating. I will assume that you also have labels for some of the comments which we will call your training set.

I do not have access to your data but I have a similar dummy example which uses comments and tries to classify them as being a review of a hotel or a car rental. I think this example would suit this purpose nicely.

Pre-process the data

There's many techniques we can use for this purpose. For example some popula rones are bag-of-words, tf-idf, and n-grams. These techniques will take a comment and vectorize it into a set of features. The features which we will use are learned from the training set and then will be the same for new comments coming in. We will use bag-of-words for this example.

Bag-of-words

This method has a dictionary of words which we identify from our training set. We take all the unique words after stemming. We then count the number of times that each word shows up in the comment, that is the vector that represents a given instance. For example if our dictionary looks like

  ['car', 'book', 'test', 'wow']

And the comment was

  Wow, this car is AMAZING!!!!!

The resulting vector would be

  [1, 0, 0, 1]

We will repeat this vectorization process for each comment. We will then be left with a matrix where the rows represents each comment and the columns represent the frequency at which these words appear in the comment. We will call this matrix $X$ it is our dataset. Each of these instances will also have a label we will call that vector $Y$. We want to map each row in $X$ to it's label $Y$.

Assume this is the data you have for 40 instances in a training dataset for the cars/hotel comments. With it's corresponding label on the right. Label 0 is for cars and label 1 is for hotels.

The columns of the matrix $X$ are

  ['car', 'passenger', 'seat', 'drive', 'power', 'highway', 'purchase',
  'hotel', 'room', 'night', 'staff', 'water', 'location']

Split the data

We will split the data in order to test the accuracy of our model

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3)

Apply Naive Bayes

from sklearn.naive_bayes import GaussianNB

clf = GaussianNB()
clf = clf.fit(X_train, y_train)
clf.score(X_test, y_test)

This gives a result of

  1.0

Perfect classification!

How does it work?

Gaussian Naive Bayes assumes that each feature is described by a Gaussian distribution (you can pick other distributions which may be better suited for your data, such as multinomial). It will then assume that each feature is entirely independent of one another. Thus we can say that the probability of

$P(Y=0|X) = P(X_0|Y=0)P(X_0) * ... * P(X_n|Y=0)P(X_n)$

$P(Y=1|X) = P(X_0|Y=1)P(X_0) * ... * P(X_n|Y=1)P(X_n)$

for your $n$ features. The the one that is the largest is going to be our label. If $P(Y=1|X)$ is larger then we will say that the comment was that of a car. So to do this we need to train the parameters of our probability distributions $P(X_i|Y)$. To calculate this term we need to know the parameters of our distribution, in our case we are using a Gaussian distribution, thus we need the mean and the variance $\mathcal{N}(\mu, \theta)$.

We will build a dictionary of values for each label that contains the mean and variance for each feature. This is the training stage!

Homemade Naive Bayes

import csv
import random
import math
import pandas as pd
import numpy as np

class NaiveBayes(object):

    def __init__(self):
        self.groupClass = None
        self.stats = None

    def calculateGaussian(self, x, mean, std):
        exponent = np.exp(-1*(np.power(x-mean,2)/(2*np.power(std,2))))
        std[std==0] = 0.00001
        return (1 / (np.sqrt(2*math.pi) * std)) * exponent

    def predict(self, x):
        probs = np.ones((len(x), len(self.stats)))

        for ix, instance in enumerate(x):
            for label_ix, label in enumerate(self.stats):
                probs[ix, int(label)] = probs[ix, int(label)] * \
                                      np.prod(self.calculateGaussian(instance, self.stats[label][0], self.stats[label][1]))
        return np.argmax(probs, 1)

    def score(self, x, y):
        pred = self.predict(x)
        return np.sum(1-np.abs(y - pred))/len(x)

    def train(self, x, y):
        self.splitClasses(x, y)
        self.getStats()
        pass

    def splitClasses(self, x, y):
        groupClass = {}

        for instance, label in zip(x, y):
            if not label in groupClass:
                groupClass.update({label: [instance]})
            else:
                groupClass[label].append(instance)
        self.groupClass = groupClass

    def getStats(self):
        stats = {}

        for label in self.groupClass:
            mean = np.mean(np.asarray(self.groupClass[label]), 0)
            std = np.std(np.asarray(self.groupClass[label]), 0)
            stats.update({label: [mean, std]})
        self.stats = stats

clf = NaiveBayes()
clf.train(X_train, y_train)
clf.score(X_test, y_test)

Once again we get an accuracy of 1.0 on the testing set!


Understanding dropout and gradient descent

In dropout as described in here, weights are not masked. Instead, the neuron activations are masked, per example as it is presented for training (i.e. the mask is randomised for each run forward and gradient backprop, not ever repeated).
The activations are masked during forward pass, and gradient calculations use the same mask during back-propagation of that example. This can be implemented as a modifier within a layer description, or as a separate dropout layer.
During weight update phase, typically applied on a mini-batch (where each example would have had different mask applied) there is no further use of dropout masks. The gradient values used for update have already been affected by masks applied during back propagation.
I found a useful reference for learning how dropout works, for maybe implementing yourself, is the Deep Learn Toolbox for Matlab/Octave.


I suggest to calculate the Haversine distance between two points, and fit a linear regression to find the relation between the Haversine distance and the trip duration.
So your regression will be
$duration_t = timestamp_t - timestamp_{t-1} = \alpha + \beta*d(point_t,point_{t-1})$
Where $d$ is the Haversine distance. $point_t$ is a lat/long pair at time $t$.
Note however that there's an assumption that the user drives at the same speed. If half of your data was gathered while walking and half while driving, then your relation between time and ETA is possibly not linear.


Over-sampling: is my model over-fitting?

In order to get accurate results, you should not oversample the test set! Otherwise you are simply evaluating on synthetic samples that you yourself have created. The support on your classification report should mirror the imbalance in your dataset.
From what I understand you have 3500 samples, then you did some oversampling (probably brought them to around 6000) and then took 1757 from these for testing. This evaluation scheme is wrong. Take a look at the illustration below to see a more correct scheme.
      |--- train --> oversample train set --> train model---|
set --|                                                     |--> evaluation on test set
      |--- test --------------------------------------------|


In a short answer, It seems that you used the same hyperparameters for both cross-validation's last training and the simple training process (without cross validation).
In the long answer

In K-fold cross-validation, you tune the hyperparameters according to the mean of K performance in order to obtain the maximum generalizing ability of the model.
Then you train the model last time with the chosen hyperparameters using the whole training set. If you have a test set you use that for the final evaluating of your model.
To sum up, the last step of the training part is training the model with the whole training set even if you use k-fold cross-validation or not.
So it seems that with cross-validation and without cross-validation you use the same hyperparameters as "max_depth", "minimum sample leaf etc."


Timestamps in Ridge Regression Scikit Learn

The datetime type has arithmetic operations available. If you have two datetime types, you can find the delta between them - the result will be a datetime.timedelta class. The other quantitative data is easily incorporated into a linear regression model, either lasso or ridge. Pretty much all those scikit-learn models can use a vector of features.


L2 and L1 regularization are controlled via the lambda (=reg_lambda) and alpha (=reg_alpha) parameter respectively. Higher values of alpha mean more L1 regularization. See the documentation here.


Extract features from Decision tree leaf nodes

I will describe my personal approach to achieve what is taught in the Course you mentioned, a mind blowing course I think!

To generate DT features for your training set:

First of all, split your training data in k-folds. It is necessary to avoid overfitting. Then:

Hide a fold and build a DT using the (k-1) remaining folds. Control the number of leaf nodes in your DT using the max_leaf_nodes paramenter in sklearn. Set it as the number of categories (levels) you want in your new categorical feature.
Predict output for the held out fold. When you are predicting using a DT, each observation will fall in a leaf. So, the values for your new feature will be the index of the leaf each observation ends up in. You can get this index using the apply command in sklearn.

Repeat these 2 steps for each one of the k folds.

To generate DT features for your test set:

Build a DT using the entire training set. Don't forget to set the max_leaf_nodes parameter as the same used above.
Predict output for the test set. Get the leaf index for each test observation.

Now you have a new categorical feature containing the leaf indexes in the predictions of the DT.

Problems with this procedure

Since you cannot control the sequence of splits in each DT, the indexes found in each CV fold may no match. For example, category "0" found in the first fold may not be same category "0" found in the second fold.
At the same time, it is not recommended (following the advises in the Course) building a DT using the entire training set and then predicting leaf indexes for the training and test sets using the same DT. This may lead to overfitting, because you are introducing target information in your training set.

So, what can we do?

Instead of using the leaf index as the feature value you can count the number of nodes each observation goes through until it falls in a leaf, and use it as the feature value. In single words, it is the path length from the root node to the leaf node. I think it is a more informative value. This is the principle behind Isolation Forest.


If I understood your question correctly, you can do something like:

import numpy as np

columns = ['median', 'p25th', 'p75th']
df[columns] = df[columns].replace(r'UN', np.nan, regex=True)


Machine learning for missing data in time series

If column B truly exhibits the same patterns, evolution, and rate-of-change as column A, then that means the values of column B are just a translation of A.  That is, $B_i = A_i + c$, where $c$ is some constant.

If this is the case, then you don't need machine learning to impute values.  Knowing $A$ is just as good as knowing $B$.  You just need to estimate $c$.  One good way to estimate $c$ would be to compute the mean difference between $B$ and $A$.

On the other hand, if B is related to A, but does not exhibit the exact same patterns and rate-of-change, then machine learning might be a little more appropriate.  You'll want to train a regression model that predicts B as a function of A, then use the model to fill in missing values of B.

Given the degree of coupling you describe between A and B, linear regression is probably a good place to start.  You can experiment with higher-order polynomials (quadratic, cubic, etc) if linear regression doesn't get the job done.  An easy way to get started would be to experiment with polynomial regression in Excel or Python.


CNN only performs well when split into 2 models

You could try saving the model after training the one it performs best on. Then restore it and try feeding in the 2nd set.

saver = tf.train.Saver()

Then at the end of training, add the line

save_path = saver.save(sess, "/your/file/path.ckpt")

Then to restore.

with tf.Session() as sess:
      saver.restore(sess, "/your/file/path.ckpt")


It fits the scaler and then transforms the training data to pass it to the k-means. The method is called fit_transform in scikit-learn.


Having trouble understanding the x and y axis in SVM when training and testing data

The answer to your questions is simply : that is how training works :

The training set is a matrix in which the last column correspond to the labels and the rest correspond to the samples.

When slicing  the matrix, you have two parts: the first one concerns the rows, the second one concerns the columns.

ROWS:

:partition means in fact the first 80% rows - this concerns the training set x_train, y_train

partition: means the last 20% rows - concerns the test set, x_test and y_test

COLUMNS:

:-1 mean all but the last column - concerns x_train and x_test

-1: mean just the last column - concerns y_train and y_test

Therefore, here you get for x_train and x_test everything but the last value. Be careful, in fact you get 80% of the data is for x_train (:partition) and 20% is for x_test (partition:). Now, for the labels you get only the last column, and 80% of the rows are for y_train while 20% of the rows are for y_test.

One more detail, your x_train and y_train is a 5D array, according to the documentation.


This will give you a sample with 55 record from the larger data-set that you have.

 sample <- df[sample(1:nrow(df), 55, replace=FALSE),]

If you want to make a reproducible example of the sample, you need to set seed like this:

 set.seed(57)
 sample <- df[sample(1:nrow(df), 55, replace=FALSE),]

So later you can use the same seed for getting the same result.

p.s. replace=FALSE means that each record after that has been picked from the df to be included in the sample, will be excluded from the df and it will not be chosen multiple times.


What is the difference between active learning and reinforcement learning?

Active learning is a technique that is applied to Supervised Learning settings. In the supervised learning paradigm, you train a system by providing inputs and expected outputs (labels). The system learns to mimic the training data, ideally generalizing it to unseen but extrapolable cases. Active learning is applied normally in cases where obtaining labels is expensive so, we obtain new labels dynamically, defining an algorithmic strategy to maximize the usefulness of the new data points.
Reinforcement learning is a different paradigm, where we don't have labels, and therefore cannot use supervised learning. Instead of labels, we have a "reinforcement signal" that tells us "how good" the current outputs of the system being trained are.  Therefore, in reinforcement learning the system (ideally) learns a strategy to obtain as good rewards as possible.


28x28 mnist image unrolling to concatenate using numpy.vstack. Why is numpy.vstack so slow?

You can reshape a numpy array simply by:
newxdata =  xdata.reshape((60000,28*28))

for example. Or simply:
newxdata =  xdata.reshape((len(xdata),-1))

Note that reshape is a numpy function which can used also as:
import numpy as np
newxdata =  np.reshape(xdata, (60000,-1))

To speed up your loop you could alternatively use libraries like multiprocessing.Pool or CuPy or Numba.


The problem you describe may be tackled with online machine learning where you continuously update your model as new data arrive,  avoiding the computationally intensive part of retraining.

For deep neural networks, there is some work in this direction.

scikit-learn and Vowpal Wabbit also provide some online algorithms.


Single image feature reduction at inference time : SVM

Just like all other Class, you have  different methods i.e.fit, transform, fit_transform

fit(self, X[, y]) Fit the model with X.
fit_transform(self, X[, y]) - Fit the model with X and apply the dimensionality reduction on X.
transform(self, X) - Apply dimensionality reduction to X. X is projected on the first principal components previously extracted from a training set.

So, you fit on the data received until the current data and then use transform on the current data.
It's similar to the way we will do Scaling in online learning e.g. You fit your StandardScaler on train data and transform the new data on that.


Why is sklearn.metrics.roc_auc_score() seemingly able to accept scores on any scale?

The documentation says

  Target scores, can either be probability estimates of the positive
  class, confidence values, or non-thresholded measure of decisions
  (as returned by "decision_function" on some classifiers).

(My emphasis)

This is possible because the implementation only requires that the y_true can be sorted according to the y_score.

The false positive rate that is returned is given as follows:

  A count of false positives, at index i being the number of negative
  samples assigned a score >= thresholds[i]. The total number of
  negative samples is equal to fps[-1] (thus true negatives are given by
  fps[-1] - fps).

The false positives can be calculated with only taking into account the order of the predicted scores/thresholds.

Multiplying the scores by a scalar has no effect on the order of the values, thus the resulting score remains the same!

See the sklearn GitHub page, the relevant code is under the definition of _binary_clf_curve.


Both train and test scores high. Could my model be overfitting?

Overfitting is nothing but when the model tries to rot learn the data given to it during the training process.

In such cases, the validation accuracy goes down as the model cannot interpret and classify unseen data.

  Reviewing the accuracies of the model, the validation accuracy soars to 0.8 which is quite good.

The training accuracy is 0.96 which is a good sign and also it shows that model has generalised so well that it can classify all the data ( train and validation ) very well.


How can I plot data after cluster it into two cluster?

KMeans on such data is meaningless if used carelessly. It looks as if your result is based on whether the ID is larger than 200000 or not.

Rather than copy and pasting example code from the internet now also to visualize, I think you should spend more time thinking about what you are actually computing. Because solving the wrong problem doesn't help.


how to input the data set in to a word2vec by keras?

I have already searched for the solution and found this statements which are proper for this kind of datasets:
at first, you should concatenate 2 folders in one folder to apply the code below:

import os
import gensim, logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',      level=logging.INFO)

class MySentences(object):
    def __init__(self, dirname):
        self.dirname = './Dataset#2_BinaryClassClassification/lupus_alldataset/'

    def __iter__(self):
        for fname in os.listdir(self.dirname):
           for line in open(os.path.join(self.dirname, fname)):
               yield line.split()

sentences = MySentences('./DatasetBinaryClassClassification/alldataset/') # a memory-friendly iterator
model = gensim.models.Word2Vec(sentences,
    size=200,
    window=10,
    min_count=2,
    workers=5)

I found the 'class MySentences' in this site:enter link description here

I hope it is helpful.


Mechanical engineer having troubles with neural network. Am I going in the right direction? Is my data set insufficient?

If the training loss is “huge” it suggests that the is a bug in the model or the model is not sophisticated enough.
Do not confuse training loss with generalisation. Generalisation is measured by accuracy on the test set.
With regards to training data have you tried data augmentation? Can you apply transformations to the training data to derive new samples. E.g if I train a model to drive a car based on video input and control output the video and control inputs can be flipped to turn left turns into right turns?


here's what I've tried:
import pandas
import re

KW = pd.read_csv("regex.csv")
DF = pd.read_csv("Default-Profile.csv")

for i in range (0,KW.kws.size):
    for j in range (0,DF.field_name.size):
            if (DF.field_name.match(r"KW.kws[i]") == True && (DF.field_name.match(r"mand_kw[i]" == True)):
                    DF.field_friendly_name[j] = KW.field_friendly_name[i]
                    DF.is_included_in_report[j] = "True"


How to best use geographical information as a factor?

If you are predicting crime rates in a certain region, we may use clustering to deduce useful information. In clustering, basically, we will try to group similar data points together and treat them as a single class. We can understand this with an instance.

We have various points ( Latitude and Longitude ) and each of them represents a certain type of crime. Even by mere observation, we can conclude that some specific types of crime occur in a particular region only. Basically, we are going to cluster such points which are in the vicinity of each other and belong to the same class ( kind ).

  For example, an emergency call from an area ( with more cases of robbery ) arrives, the probability that the victim has also suffered from the robbery is more than any other crimes.

As we get more data, we can retrain our clustering algorithms to make more clusters and thereby increase efficiency.


For the first Q you already give a counterexample:

It is biased to k, it prefers k = N. And it will also overrate N-1, N-2, ... So it is not agnostic to k.

If k-means would always find the best C index, then the C index would just be redundant to SSQ, which is much cheaper to compute... But you probably have just been looking at way too simple toy datasets. Use real data.


Any Machine Learning algorithm to know a yes/no answer?

I sounds like your need to consider Language Understanding services instead. There are a couple of providers of this type of service. The one I have used before is called LUIS from Microsoft:

https://www.luis.ai/home/index

It is not the algorithm you are looking for per se, but to solve your problem by implementing an algorithm, you need to develop a language understanding system using models based on deep learning neural nets, which is not a trivial exercise.


Is using unsupervised learning to setup supervised classification reasonable?

Absolutely. This is a common strategy in ML. As you said yourself, using information coming from the data itself has the benefit of being less biased.

Would it even also be possible to use unsupervised learning to create the labels?

Technically yes. Though, some clustering techniques require you to specify the number of clusters, which isn't helpful. As you said, if you can cluster data points in a satisfactory manner, you don't need supervised learning anymore. Also, indeed, if your scenario requires you to have an understanding of what differentiates the clusters, you may not be lucky depending on which clusters come out. They are not always interpretable.

What I would suggest would be to turn your classification problem into a regression problem. 1.0 could be most likely to cause the disease, 0.0, least likely. This way, you don't have to worry about how many labels you need in the first place.


How do you calculate the number coefficients in polynomial regression?

Yes - it is $d\times l$. There would be 6 $β$ coefficients for 2*3:
$β_1x_1 + β_2x_1^2 + β_3x_2 + β_4x_2^2 + β_5x_3 + β_6x_3^2$
That does not include an intercept or any interaction terms.


A few names I found for X are:

feature vector
an input variable
an independent variable
an explanatory variable
an exogenous variable
a predictor variable
a regressor
a covariate


I'm not super familiar with Bishop, but it looks like Durrett, which deals with measure-theoretic probability, would be overkill.  Plus you would probably have to add real analysis to your prerequisites.  I would just work through the first 4-5 Casella chapters on probability and random variables.

You probably also don't need to go through the whole linear algebra book to get started.  Just make sure you understand the basic matrix operations well enough to deal with multivariate calculus (ex. taking multivariate derivatives and computing multivariate integrals).

The second half of Casella would likely be useful, but I suspect not strictly necessary.

I think if you can handle those things you'll have most of Bishop covered; you can always pick up more as needed.


I'd represent actual repeated values differently from missing data. The latter is straightforward; the former I'd interpolate with Gaussian process regression. This way you'll be able to get error bars like so:

Note how the sample functions (and thus error bars) expand and contract as you depart and approach the measurements, as you would intuitively expect.


Your training algorithm seems wrong. You should either implement Back Propagation, or tweak your own training algo towards that.

Now your training brings all the weights to an average value, that's why your output seems so close after 10000 iterations. What BP does is it modifies the weights that matter more.

Alternatively, if you don't want to use BP, you can play with evolution instead. It will give you the same result, just slower.

edit: can you post your training algo's code somewhere?
Also, you don't really need different objects (like Neurons, Connections, etc.), you only need matrix algebra for an efficient solution.


Some very helpful articles walked me through how to use the feature selection modules in Azure ML Studio:

https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/feature-selection-modules
https://docs.microsoft.com/en-us/azure/machine-learning/studio-module-reference/filter-based-feature-selection

I used the "Filter Based Feature Selection" Module:

I added it to my training experiment and added input data.

I chose the feature I was training my model to predict.

After running the experiment I viewed the visualization of the Features:

Now I am using this to find which columns are valuable and which need some work.


Optimizing parameters for a closed (black-box) system

There are lots of function optimising routines that could be applied, based on the description so far. Random search, grid search, hill-climbing, gradient descent, genetic algorithms, simulated annealing, particle swarm optimisation are all possible contenders that I have heard of, and I am probably missing a few.

The trouble is, starting with next to zero knowledge of the black box, it is almost impossible to guess a good candidate from these search options. All of them have strengths and weaknesses. To start with, you seem to have no indication of scale - should you be trying input parameters in any particular ranges? So you might want to try very crude searches through a range of magnitudes (positive and negative values) to find the area worth searching. Such a grid search is expensive - if you have $k$ dimensions and want to search $n$ different magnitudes, then you need to call your black box $n^k$ times.

This can be done in parallel though, and given you are confident that the function is roughly unimodal, you can start with a relatively low number of n (maybe check -10, -1, 0, +1, +10 for 15625 calls to your function taking roughly 8 hours 40 mins using 5 boxes). You may need to repeat with other params once you know whether you have found a bounding box for the mode or need to try yet more values, so this process could take a while longer - potentially days if the optimal value for param 6 is more like 20,000. You could also refine more closely, once you have a potential mode you might want to define another grid of values to search based around. This basic grid search might be my first point of attack on a black box system where I had no clue about parameter meaning, but some confidence that the black box output had a rough unimodal form.

Given the speed of response you should be storing all input and output values in a database for faster lookup and better model building later. No point repeating a call taking 10 seconds when a cache could look it up in 1 millisecond.

Once you have some range of values you think that a mode might be in, then it is time to pick a suitable optimiser.

Given the information so far, I would be tempted to run either more grid search (with a separate linear scaling between values of each param) and/or a random search, constrained roughly to the boxes defined by the set of $2^6$ corner points around the best result found in initial order-of-magnitude search.

At that point you could also consider graphing the data, to see if there is any intuition about which other algorithms could perform well.

With the possibility of parallel calls, then gradient descent might be a reasonable guess, because you can get approximate gradients by adding a small offset to each param and dividing difference that causes in the output by it. In addition, gradient descent (or simple hill climbing) has some chance of optimising with less calls to evaluate the function than approaches that rely on many iterations (simulated annealing) or lots of work in parallel (particle swarm or genetic algorithms). Gradient descent optimisers as used in neural networks, with additions like Nesterov momentum or RMSProp, can cope with changes in function output "feature scale" such as different sizes and heights of peaks, ridges, saddle points.

However, gradient descent and hill climbing algorithms are not robust against all function shapes. A graph or several of what your explorations are seeing may help you to decide on a different approach. So keep all the data and graph it in case you can get clues.

Finally, don't rule out random brute-force search, and being able to just accept "best so far" under time constraints. With low knowledge of the internals of the black box, it is a reasonable strategy.


Understanding python XGBoost model dump output of a very simple tree

The scores at leaves measure log-odds, not probabilities.  (With more trees, these scores get summed to give a final log-odds approximation, then goes through a sigmoid to get probability approximations.)

And indeed, $1/(1+e^{-0.00475667})\approx 0.501189$.


How to set class_weight parameter for cost sensitive learning?

\begin{array} {|r|r|}
\hline
- &(1) &(0) \\
\hline
(1) &- &FP\\
\hline
(0) &FN &-\\
\hline
\end{array}
Rows elements are Y_true and column is Y_pred.
FP means, we predicted Positive and it came out False i.e. Class was Negative(0 here).
It means we don't want the Model to mis-classify the Negative class.
So, it implies, we will put a bigger penalty for the Negative class. Hence,
class_weight = {0:3.0.0,1:1.0}


how to train custom word2vec embeddings to find related articles?

I find your question bit convoluted, so I will answer with the following bullet points:

Train your own word embeddings: There are many implementations out there, gensim is one.
Find related articles: On that point, without being an expert, I would suggest to do some research on Topic Modelling. There are also a lot of libraries you can use.
Word embeddings to sentence embeddings: This process is not as straightforward, the semantics change just by adding words together. You can use Word Mover's Distance or numerous other which train in a supervised way sentence embeddings or unsupervised.


Efficient recurrent network for sequences of varying length

Sequence bucketing

Depending on the input length of sequences you can dynamically change the padding and speed it up.

Take a look at here Sequence bucketing


What is the difference between SVM and GMM classifier

A Gaussian mixture model is a special case of a mixture distribution, which is a simple way of combining probability distributions.

However, the SVM does not make any assumptions. It is just a function, which depends on the distance of a data point from another point/plane. Som there is absolutely no probabilistic assumptions.


How to train a xgboost model on data that is too big for the memory?

You can train xgboost, calculate the output (margin) and then continue the training, see example in boost from prediction.

I‘ve not tried it myself, but maybe you could train on the first subset of your data (say 10%) and then continue on another subset, etc.

Update

Step by step procedure

Split the data into N manageable subsets, set n=1
Train xgboost on n-th subset
Calculate the prediction (margin) for n+1 subset using the model obtained from previous
Add the margin into the n+1 subset via setinfo
Increment n

Steps 2-5 to be repeated N times.


Why is learning rate causing my neural network's weights to skyrocket?

You might find Chapter 8 of Deep Learning helpful. In it, the authors discuss training of neural network models. It's very intricate, so I'm not surprised you're having difficulties.

One possibility (besides user error) is that your problem is highly ill-conditioned. Gradient descent methods use only the first derivative (gradient) information when computing an update. This can cause problems when the second derivative (the Hessian) is ill-conditioned.

Quoting from the authors:

  Some challenges arise even when optimizing convex functions. Of these, the most prominent is ill-conditioning of the Hessian matrix $H$. This is a very general problem in most numerical optimization, convex or otherwise, and is described in more detail in section 4.3.1.

  The ill-conditioning problem is generally believed to be present in neural network training problems. Ill-conditioning can manifest by causing SGD to get “stuck” in the sense that even very small steps increase the cost function. [my emphasis added]

The authors provide a simple derivation to show that this can be the case. Using gradient descent, the cost function should change (to second order) by

\begin{equation}
\frac{\varepsilon^2}{2} g^{T} H g - \varepsilon g^{T} g
\end{equation}

where $g$ is the gradient, $H$ is the Hessian, and $\varepsilon$ is the learning rate. Clearly, if the second derivatives are large, then the first term can swamp the second, and the cost function will increase, not decrease. Since the first and second terms scale differently with $\varepsilon$, one way to alleviate this problem is to reduce $\varepsilon$ (although, of course, this can result in learning too slowly).