Back propagation through a simple convolutional neural network

The backpropagation algorithm attributes a penalty per weight in the network. To get the associated gradient for each weight we need to backpropagate the error back to its layer using the derivative chain rule.
Flattening layer
The derivative of a layer depends on the function that is being applied. In the case of the flattening layer it is simply reshaping (a mapping) the values. Thus no additional loss will be added at this layer. All you need to know is how the flattening occurs.
For example if the forward pass flattening is
$flatten\begin{pmatrix}
  a & b\\
  c & d
\end{pmatrix} = \begin{pmatrix}
  a \\
  b \\
  c \\
  d
\end{pmatrix}$,
then you can easily map the associated cost so far back to the $2 \times 2 \times 1$ feature map.
Max pooling layer
In the forward pass the max pooling layer is taking the maximum value in a $3 \times 3$ window that is passed along your image. For example the bold values in the first $3 \times 3$ window would have a maximum of $11$.
$maxpooling \begin{pmatrix}
  \bf{1} & \bf{2} & \bf{3} & 4 \\
  \bf{5} & \bf{6} & \bf{7} & 8 \\
  \bf{9} & \bf{10} & \bf{11} & 12 \\
  13 & 14 & 15 & 16
\end{pmatrix} = \begin{pmatrix}
  \bf{11} & 12\\
  15 & 16
\end{pmatrix}$
Thus the resulting error backpropagation would only pass through the maximum values which were passed down by the forward pass. For all other values the error term would not backpropagate. Thus the current error matrix you had backpropagating until this point would be multiplied by
$\begin{pmatrix}
  0 & 0 & 0 & 0 \\
  0 & 0 & 0 & 0 \\
  0 & 0 & 1 & 1 \\
  0 & 0 & 1 & 1
\end{pmatrix}$
Thus only 4 error terms would continue onto earlier layers.
Convolutional layers
I have gone in detail about how to do backpropagation through convolutions here: CNN backpropagation between layers.


How to customise cost function in Scikit learn's model？

There are several ways in which you can achieve your desired result:

Implement the make_scorer function from Scikit learn
Make modification to the class_weight argument

In regards to your SVM question take a look at the below code:

 class sklearn.svm.SVC(C=1.0, kernel='rbf', degree=4, gamma=0.0,
 coef0=0.0, shrinking=True, probability=False, tol=0.003,
 cache_size=300, class_weight=None, verbose=False, max_iter=-1,
 random_state=None)

In the above example, the class_weight function can be changed to 'auto' or you can pass dictionary values which have the user-defined class weights.


How to train-test split and cross validate in Surprise?

EDIT: It seems I misunderstood the task at first, so here's my correction. Hope it works this time

It seems like what you're trying to do is similar to what is in the documentation under examples/split_data_for_unbiased_estimation.py (or this github issue which seems to be exactly what you want)

The code manually splits the dataset into two without using any sort of function call. Then sets the internals of the data variable to be only the train split.

import random

from surprise import SVD
from surprise import Dataset
from surprise import accuracy
from surprise import GridSearch

# Load your full dataset.
data = Dataset.load_builtin('ml-100k')
raw_ratings = data.raw_ratings

# shuffle ratings if you want
random.shuffle(raw_ratings)

# 90% trainset, 10% testset
threshold = int(.9 * len(raw_ratings))
trainset_raw_ratings = raw_ratings[:threshold]
test_raw_ratings = raw_ratings[threshold:]

data.raw_ratings = trainset_raw_ratings  # data is now your trainset
data.split(n_folds=3)

# Select your best algo with grid search. Verbosity is buggy, I'll fix it.
print('GRID SEARCH...')
param_grid = {'n_epochs': [5, 10], 'lr_all': [0.002, 0.005]}
grid_search = GridSearch(SVD, param_grid, measures=['RMSE'], verbose=0)
grid_search.evaluate(data)

algo = grid_search.best_estimator['RMSE']

# retrain on the whole train set
trainset = data.build_full_trainset()
algo.train(trainset)

# now test on the trainset
testset = data.construct_testset(trainset_raw_ratings)
predictions = algo.test(testset)
print('Accuracy on the trainset:')
accuracy.rmse(predictions)

# now test on the testset
testset = data.construct_testset(test_raw_ratings)
predictions = algo.test(testset)
print('Accuracy on the testset:')
accuracy.rmse(predictions)

PS: If you feel like this seems a bit hacky and weird... then the core-developer of Scikit-learn that wrote this code also agrees with that sentiment.


Load large .jsons file into Pandas dataframe

There are a few ways to do this a little more efficiently:

JSON module, then into Pandas

You could try reading the JSON file directly as a JSON object (i.e. into a Python dictionary) using the json module:

import json
import pandas as pd

data = json.load(open("your_file.json", "r"))
df = pd.DataFrame.from_dict(data, orient="index")

Using orient="index" might be necessary, depending on the shape/mappings of your JSON file.

check out this in depth tutorial on JSON files with Python.

Directly using Pandas

You said this option gives you a memory error, but there is an option that should help with it. Passing lines=True and then specify how many lines to read in one chunk by using the chunksize argument. The following will return an object that you can iterate over, and each iteration will read only 5 lines of the file:

df = pd.read_json("test.json", orient="records", lines=True, chunksize=5)

Note here that the JSON file must be in the records format, meaning each line is list like. This allows Pandas to know that is can reliably read chunksize=5 lines at a time. Here is the relevant documentation on line-delimited JSON files. In short, the file should have be written using something like: df.to_json(..., orient="records", line=True).

Not only does Pandas abstract some manual parts away for you, it offers a lot more options, such as converting dates correctly, specifying data type of each column and so on. Check out the relevant documentation.

Check out a little code example in the Pandas user guide documentation.

Another memory-saving trick - using Generators

There is a nice way to only have one file's contents in memory at any given time, using Python generators, which have lazy evaluation. Here is a starting place to learn about them.

In your example, it could look like this:

import os

# Get a list of files
files = sorted(os.listdir("your_folder"))
# Load each file individually in a generator expression
df = pd.concat(pd.read_json(file, orient="index") for f in files, ...)

The concatenation happens only once all files are read. Add any more parameters that are required where I left the .... The documentation for pd.concat are here.


Is it valid to shuffle time-series data for a prediction task?

It depends on how you formulate the problem.

Let's say you have a time-series of measurements X and are trying to predict some derived series of values (mood) Y into the future:

X = [x0, x1, x2,.....]
Y = [y0, y1, y2,.....]

Now, if your model has no memory, that is, it is merely mapping xN -> yN, then the model does not care what order it sees x and y in. Feed-forward neural networks, linear regressors etc. are memory-less models.

But, if your model has memory, that is, it is mapping a sequence of features to the next days moods xN-2, xN-1, XN... -> yN-2, yN-1, yN..., then the ordering matters. The model in this is keeping track of what it has seen before. The model's internal parameters are changing and persisting with each new example it sees. The current prediction depends on the last prediction. Recurrent neural networks have memory, so order matters.

You can get around the memory requirement by restructuring your dataset. You can concatenate consecutive features and map them to next day's mood instantaneously (xN-2, xN-1, xN) -> yN. In this way, your input feature will incorporate information about the past, but the model won't care about the order since all the temporal information is encoded in the new feature, and not the model. Your new dataset will look like:

Xnew = [(x0, x1, x2), (x1, x2, x3), (x2, x3, x4),...]
Y    = [         y2,           y3,           y4,...]


Decision Tree generating leaves for only one case

You are predicting binary class: 0 - sale failed, 1 - sale succeed. In the decision tree, the information value is sorted. So for example in the first node, you have 35011 predictions of 0 and 1785 predictions of class 1.

But then, in your code you have this: class_names=["Won","Lost"]). So you are telling your decision tree that name of class 0 is "Won" and name of class 1 is "Lost". I assume it should be inverted.

So in fact, the model always predict 0 - sale failed. Which seems reasonable as you said that out of 38000 samples, only 1700 are class 1.

Further, your dataset is highly unbalanced. So when cross validation gives accuracy of 95%, it says nothing, because only if your model gives all prediction 0, you will have accuracy of...95%.
For those cases, use models which can put weights to classes (SVM, Logistic regression or Random Forest classifier in scikit have this possibility). Think about undersample your dataset. Or add more of class 1 - dig in your database or do it artificially (SMOTE),

Plot confusion matrix and give CV different score measure - precision or recall, depends on your situation.

Last thing, don't expect that your decision tree with depth of 3 will perform well. It is nice to show to business, but it gives poor prediction.


You might want  to look at Multi-Label-Classification . If you have sufficient number of samples as your training data, you can build a model that can predict more than one label for a test sample. You can find more about the implementation at the sklearn page for the same here .


Which F1-score is used for the semantic segmentation tasks?

I looked very quickly and only at the first paper so I might miss something but it looks to me like the task is a binary classification problem. If this is correct then there's no need for averaging the F1-score.
Also in this paper the authors even give the formula of the F1-score (!), so I'd say that they are quite thorough in their description of the evaluation measures they use. I would take this as an additional indication that there's no averaging,  since it's unlikely that they wouldn't mention it.


Linear SVM with slack variables: Will it find a perfect decision boundary if it exists?

Having linear separable data means that an optimal solution exists. The support vector machine is an approach to identify the optimal solution with multiple steps through Lagrange multiplication. This method is guaranteed to give you the optimal solution if there is one.


How to use correct weights in linear regression model

You are correct for a linear separator line the derivative seems trivial. In this case gradient descent is not necessary because a closed form solution exists for the weights.

$w = (X^TX)^{-1}X^Ty$

We only use optimization techniques such as gradient descent for models where a closed form solution does not exist. However, if you perform gradient descent on the weights using the derivative and doing the following equation iteratively

$w^{new} = w^{old} - \nu \frac{dy}{dx}$

would yield the same result as the closed form solution. This is however very unnecessary when not necessary.


Is the k-center problem part of machine learning or computational geometry?

Where is the "learning"? Where is the "training", and "becoming better"?

It's an optimization problem, but optimization is everywhere.

Fits computational geometry much better than ML.

But as a rough guideline, check which books discuss this well. Can you name a book that belongs into machine learning that discusses this is sufficient depth?


How many run should we implement on the machine learning model?

It depends on your problem, start with a big number (32 ?) then decrease if the model is stable / when you deal with stability.
You need to know that having multiple runs may be part of the solution: you can create an ensemble of models to get more stable models. Here again, the way to create an ensemble and the number of models you need to run to build an ensemble is determined experimentally. The main alternative to add stability to your model is trough regularisation (that really depend on the model you want to use).


How to do Named Entity Recognition in Tables?

As far as I know you don't have a lot of options, you're probably stuck with heuristics:

Capital letters
Regular expressions (e.g. for dates)
List of predefined entities (e.g. from Wikipedia) stored in a dictionary


Why I am having ValueError in this Linear Regression?

There are most likely missing values in your data, those missing values are encoded as NaN.

The most common solutions for missing data are:

Drop the instances/rows that have any missing values.
Impute the missing values.


LSTM Multivariate time series forecasting with multiple inputs for each time step

The only possible way is to create a dataset separately for each user; in the end, if you have 10 users, then you would have 10 different unrelated time series in the same .csv, since each user can exhibit specific characteristics. Evidently we cannot expect to throw 10 different unrelated time series into an LSTM and expect decent results.

The solution to this is to extract the users with the most entries (you could start with the user with the most entries) and apply in the first instance a simpler algorithm, not necessarily an ml-based one but a statistical one, such as VAR.


Softmax classifier never allows for 100% probability in LSTM?

If want you mean by 100% certainty is:

1,

0,

0,

0,

as by your example, you cannot obtain this type of output using soft-max. For this to happen you will need that the softmax equals 0 for those i that are not correct, and this simply cannot happen because:

$$ e^{h_i} \neq 0 $$ $$ \forall h_i \in \mathbb{R} $$

See this link for better explanation. Note that $ e^x $ can be close to 0 but it will never be 0.
The kind of output that you want is the one that the max function would output, sadly the max function is not differentiable so a soft version is needed hence softmax.

I think that the problem is that you are using softmax after tanh, and this so you must eliminate one of the "squashing" functions, see this discussion

The reason to use softmax is if the classes are mutually exclusives, also one of the advantages of using softmax is that it reduces sensitivity to outliers by giving at least some probability to the other values. See this links for a more detailed explanation 1 and 2.


Classification with two 1-NN classifiers

There are multiple solutions:

Do linear scalarizing. E.g. If you have $d_0$ and $d_1$ then you could do:

$$D = w_1 d_1 + w_2 d_2$$

You could use the Pareto optimal front. The Pareto optimal front is the set of all distance pairs where one of the distances cannot be dominated without deteriorating the other one. This solution is probably not that well suited for your problem.
If you have a training set of linked object you could learn a model that predicts whether or not the objects are the same based on the distances. E.g. A logistic regression with the two distances as input. You could use a kernel or other non-linear tools to increase performance.

More ideas on this wiki about multi-objective optimization.


Force selecting samples in majority class with random forest

I would suggest that you drop those 30 events and all 1's from your dataset (i presume you know which 30 they are). Then randomly select 584 samples from the remaining dataset and the stick it back together. This would give you a full dataset equally weighted that contains all 30 for definite.

See the following pseudo code (it won't run because i don't know how your identifying these 30 cases so i'm generalising)

data = import()

ones = data.select('1')

interesting = data.select('something to select the 30 data points')

data.drop('1' and 'something to select the 30 data points')

randomList = randomListGen()

selected = data.select(randomList)

joined = join(selected, ones, interesting)

As i said before this won't run on anything, if you give the language you're working in and a minimum working example then i might be able to produce something more solid.


I would recommend trying both methods to see which works better.

However, in my opinion, you should fill first and then normalise it. Imagine if you normalise it first, you could end up with some values very close to 0. So when you fill in the missing values with 0, you are inherently saying that these missing values had the same initial value as those that were normalised to 0, which would be incorrect.


What are some function/package in R to find similarity of individual words not in the context of sentences?

If your intent is to find compare similarity in meaning, word2vec is the only appropriate choice. adist measures the edit distance between two words, and cosine similarity compares the similarity of two documents (treating the documents as bags of words). On the other hand, transforming a word into its word2vec embedding captures some of the latent meaning of the word. See this for a detailed explanation of how this is done.


What is the advantage of positional encoding over one hot encoding in a transformer model?

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would learn embeddings of earlier positions much more reliably than embeddings of later positions.

On the other hand paper on Convolutional Sequence to Sequence Learning published shortly before the Transformer uses one-hot encoding and learned embeddings for positions and it seems it does not make any harm there.


Fast way of computing covariance matrix of nonstationary kernel in Python

I still would apply numpy's covaranice function using numpy.apply_along_axis

import numpy as np

x = np.array([[0, 2], [1, 1], [2, 0]]).T
np.apply_along_axis(func1d=np.cov, arr=x, axis=0)


What tasks you train with one set of features and predict with another?

If you mean completely new set of features for prediction: it will not be helpful. Your model 'learns' something on the training feature space and you hope to apply the learning to new data-points in the same feature space. If the feature space for prediction is completely new, the learning will be useless!
If you mean somewhat new set of features: that often happens in text applications where new tokens might show up in the data which were never seen in training data. Generally handled by some strategy like zero vector or random vector.
Same features but different distribution: this can happen in practice but model performance will go down.


First of all check out this post. It has many reasons why Excel is inferior to other solutions, regarding data science tasks. Excel also can't handle large datasets (hundreds of thousands of records - not to mention anything in the vicinity of Big Data), image and sound data.

Excel is good for simple tasks concerning spreadsheets; it emphasizes more on presentation and ease of use, while having minimal support for actually analysing the data. Unless all you want to do is to calculate simple statistical measures (mean, average, etc) or building a very simple model (e.g. linear regression), Excel is inefficient. That being said, 99% the work a company has to deal with concerning data is simple enough to be manageable through Excel.

However Data Science mainly deals with regression, classification and complex models that excel isn't equipped to handle! If your students want to have a look at data science you need to teach them a tool that will be useful to them (R, python, etc.). These languages also have libraries with tons of built in models to "play with".

Another really huge reason I would go with the latter options is that they are open source. I personally feel that open source software should be preferred from an educational standpoint to proprietary solutions (this is also why I suggest python and R over Matlab)!


Do you know a dataset for regression where deep learning outperforms svm and random forests?

As a general rule, convolutional deep networks will perform better than SVM and Random Forest models on data that has hierarchical spatial/temporal structure, so for example sound, images, video.  Deep networks are very good at learning this spatial/temporal structure at different levels of granularity.  For classic datasets like Iris uses for teaching ML, however, I seriously doubt deep networks would perform better than SVM or Random Forest.


Adding more features in SVC leading to worse performance, even w/ regularization

30 samples probably just isn't enough. The more features you want to use, the more samples you'll need - otherwise you'll get huge model instability and overfitting (especially with bad/useless features). With only 30 samples you'll probably get the "best" results with 1 or 2 carefully selected features. Get 100 or 200 samples, then try again with 5 features.

Also make sure you're standardizing your features - for example by removing the mean and scaling to unit variance. SVMs don't like features that are a lot larger than other features.


Clustering with imbalanced data and groups

As you have the partial ground-truth (assuming for ALL clusters) I would suggest following a creative idea derived from Region Growing in image segmentation.
As your clusters are imbalanced in number of the points thus density, they are probably captured by localy using DBSCAN. Run DBSCAN with different parameters and evaluate on capturing your ground-truth into right clusters. The partitioning which gives the best result on your ground-truth evaluation will be your final clustering.


Recommended papers for deep learning based classification?

Maybe reading papers is a bit hard, because they usually don't explain everything. I suggest you watching deep learning specialization by professor Andrew Ng, which is brief and easy to figure out. If you insist on reading papers, this three sequential articles may help you. Choosing neural networks highly depend on your task. For most computer vision tasks convolutional neural nets are used. You can also take a look at here which contains different resources and discussions and many other useful stuff.


Using GPS signal, determine is this person driving a cab

Take the taxi routes and combine them with civilian car routes to form a data set for classification. Using a map (say, from Google)
 break down each route into a sequence of roads segments, from intersection to intersection. If you only have GPS traces this will involve spatio-temporal segmentation. (Intersections/terminuses are places where cars go but stop at, and the road segments are the places cars move through to get to from one intersection to another). Model these as categorical variables (road segment 1, 2, 3, etc); abstract out the physical location. Then train a classifier than accepts a sequence as input (e.g., a recurrent neural network). Use the time of departure as another feature, modeled as two real variables, $\cos(2*\pi*t/24), \sin(2*\pi*t/24)$, where $0<t<24$. If you have really accurate GPS information, I would also try to estimate the rate of lane crossing; taxi drivers are well known for their aggressive driving, and the rate of lane crossing would capture this well. Finally, use a pair of boolean variables to record whether the points of departure and destination are parking areas, if you can obtain this information. I imagine taxis will be more likely to start and stop at prohibited places.

The rest is your usual hyper-parameter optimization black magic.

If you want to be able to do anything besides classify a given route as taxi-driven or not, please say so.


How to make a linear model with a constant value in R?

To use only a single parameter, which will be a constant, you have to specify the lm formula as follows: y ~ 1. R will then fit the model, where the coefficient for this constant will be equal to the mean of y, see also this stats stackexchange answer.


Incorrect multi-variate anomaly detection - Isolation Forest Python

One way to improve the efficiency of the prediction is to convert the dataframe type to float32. I got better result while converting the data type to float32.
df = pd.DataFrame(np.array([[45,1411.27],[47754,1596200.68],[105894,3750304.55],[372953,14368324.86],[389915,14899302.85]]),columns=['ROW_CNT','TOT_SALE'],dtype=np.float32)
def outlier_func(df):
    model = IsolationForest(behaviour='new',n_estimators=1000,  max_samples='auto',
    contamination='auto', max_features=1.0)
    model.fit(df[['ROW_CNT', 'TOT_SALE']])
    df['scores'] = model.decision_function(df[['ROW_CNT', 'TOT_SALE']])
    df['anomaly'] = model.predict(df[['ROW_CNT', 'TOT_SALE']])
    anomaly = df.loc[df['anomaly'] == -1]
    anomaly_index = list(anomaly.index)
    return anomaly

outlier_func(df)


How do I visualize data for a natural language processing project?

I'm not an expert but let me try to think with you. What's your vocabulary size?

I think certainly starting with a small machine learning model is a good idea, but I think that a decision tree would quickly suffer with even a medium-sized vocabulary. You would need a huge tree to do anything. So I think I would start with pretrained word embeddings, and use a small neural net to predict the starting point. This helps, because words that are close in meaning have similar vectors, and the decision tree wouldn't be able to use that kind of information.

Your suggestions for histograms don't seem bad, but you would have a histogram that is as wide as your vocabulary, which seems like it defeats the purpose of visualizing it... If you went with word embeddings, how about using a technique like UMAP to plot the questions and articles in 2D?


Clustering or classifing n-gram-based text categories

If your ultimate goal is to cluster similar categories and assuming that you have labels of each text as category1, category2,...,categoryN from 1 to N, a bag of words method would be sufficient in order to create features so that you can run multiple desired clustering algorithms.

K-means can be a good starting point for getting similar groups of text document categories, you can try out different k for more sensible outcomes.

Methodology that I would adapt here, forgive me if understood your problem wrong, would be to concatenate all the textual data for a given category and create 1,2,3,..n gram vectors. The values of these vectors can be TF-IDF of a given n-gram, and so on. Still different feature matrices can be formed to compare category vectors but TF-IDF is a pretty common way to do it in.

After acquiring n-gram feature vectors for each text category, you can apply k-means algorithm with several distance metrics to find similar categories.

Suppose we have 5 categories of text as given:

Soccer Games
Basketball Games
Politics
Movies

The proposed method can capture the similarity between soccer and basketball games as they are both sports.

Hope this can help


Ideal difference in the training accuracy and testing accuracy

Theoretically speaking, in a perfect scenario, training and test data both represent the distribution of your problem accurately. Therefore, in an ideal case, training and testing should not have any significant differences in accuracy. This becomes more and more true when you have lots of data.

A difference of 5% is perfectly fine. In practice, it is common that training accuracy is slightly better than the test accuracy. I will say that the difference may not be the best indicator. What you should look at is correlation. Meaning that, as long as training and testing accuracy improve together at a similar rate, you're in the clear, regardless of how far apart they are. You can investigate that by training and evaluating on increasingly bigger subsets of the data. Ideally, training and testing should both improve as you add data. If test data starts decreasing, you have overfitting.


Policy Gradients vs Value function, when implemented via DQN

Is the basic idea of DQN Policy Gradient that simple, or am I getting things wrong?

This is not correct. DQN is a value-function approach, as you thought initially.

Your confusion seems to stem from the two options for action representation possible in DQN. You can either estimate a single $Q(s,a)$ function by having $a$ as an input to a neural network, or can estimate all possible $Q(s, *)$ with multiple outputs, one for each $a$. However, that is an implementation detail in how you set up the neural network. It might change the efficiency of the resulting system, but it doesn't change the nature of the RL algorithm at all.

Policy gradient methods are based around modifying a parametric policy function $\pi(a|,s,\theta)$ and learning $\theta$. The most basic policy gradient algorithm is REINFORCE, which requires an episodic problem, and updates on-policy after each episode.

Importantly, you cannot use the relationship between $Q(s,a)$ and resulting policy in Q-learning to create a policy gradient approach. The main blocker to doing this is that the effective policy $\text{argmax}_a \hat{q}(s,a,\theta)$ is not differentiable with respect to $\theta$, so it is not possible to calculate a gradient and make an update to improve the policy.

In general, if an algorithm learns a state value $V(s)$ or action value $Q(s,a)$ and then uses that to decide the policy, then it is value-based. If it learns the policy directly then it may be a policy gradient method (in RL terms it is very likely to be policy gradient, but you can also do policy search using e.g. genetic algorithms).

Policy gradient methods also include Actor-Critic approaches which learn both the policy and an associated value function (usually state value $V(s)$). This is a more advanced algorithm than REINFORCE, in that it can be applied to continuous (non-episodic) problems, and updates estimates on every step. One popular Actor-Critic approach is A3C


How to create a complex Gaussian random noise with a specific covariance matrix

Might be a bit late, but if you're still looking for an answer, here it is. You can use np.random.multivariate_normal. You need to provide the said covariance matrix as an argument to the function. You can find the documentation here.


is this grid search methodology correct?

Your hyperparameters are chosen based on the whole set of examples, and thus there is a leakage of information of the test set into the model. The validation will, therefore, be too optimistic (as you suspected).

What could be fixed is instead of:

clf.fit(X, y)

just use

clf.fit(x_train, y_train)

As you suggested yourself already. After that line, nothing needs to be changed. You are fitting the model on the training data, and then testing the performance on the (as of yet unseen) test data.

edit: actually, reading the documentation of GridSearchCV it is even a bit worse (depending on how your pipeline.fit method works under the hood): not only are the model hyperparameters chosen on the whole set, by using model = clf.best_estimator_, also your model is now trained using the training + test set, which is a bad thing of course, and will lead to the roc score on the test data to be much higher than on actual unseen data.


The larger an encoding dimension in NLP the better?

No, it isn't that way. From my personal experience, I have built custom word2vec's to some datasets and observed that at times, shorter vectors did better than longer ones. I think you can try shorters length vectors for smaller datasets. I cannot tell any metric in specific for comparison of dataset size and word2vec length though.


Keras Model Predict is not predicting all images flowing from directory?

I presume that your generator is at fault here :

predIdxs = model1.predict_generator(testGen,
    steps=(totalTest // 32))

You do an integer division on the size of your test set, but the result is not an integer and thus truncated (floored). Later on you (presumably) use original size for one column, and want to assign the data from your predict_generator (which is shorter) to another column of that same dataframe, creating a mismatch.


Weight-initialisation and model instability

As you have noticed weight initialization is extremely important, If for instance, you initialize your network with zeros, then it's guaranteed you will not learn from data. there are many ways to initialize your data:

Most simple approach (beginner approach) :
just use random initialization

Xavier Initialization:
this might be a more intermediate approach if you are interested on this check the following link:
https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79

I'd say that you gotta keep in mind when your network is "learning" it is basically navigating in a hyper dimension. most people do not actually standardize data. Standardizing data is essential since you make learning easier for your algorithm.

So short answer: standardize your data, your algorithm will converge faster into a minimum (of course not always global).


How to Identify p (lag order) for ARIMA Model in Python

Got it by following code:

    import pandas as pd
    k=0
    highestCorr = 0
    for i in range(1,10):
        cor = pd.Series.autocorr(series, lag=i)
        if(cor > highestCorr):
            highestCorr = cor
            k=i


Wrong output multiple linear regression statsmodels

I think that you are seeing what you are seeing because the model sees the relationship of each set of points in your data frame, which are governed by the equation:

 z = 20 + x +3*y + noise

But all the model sees is the resulting Z, NOT that equation which you know created z.

So it attempts to build a model which considers how Z was accomplished without knowing there was noise, while KNOWING that both x and y were in this equation because you explicitly told it the are in the model.

Based on this data. (at least this is what I got without a seed, when I ran your data..so it is close with probably different Z due to different noise)

x   y   z
1   1   32.824550
2   2   21.382597
3   3   80.615424
4   4   30.958157
5   5   42.192197
6   6   75.649622
7   7   29.815352
8   8   40.167267
9   9   59.752065
10  10  53.402601

Because x and y are the same for each point  and because your formula has x + 3*y + noise ,  z is also equal  to 4*x+ noise or 4*y+ noise or 2*x +2*y+ noise for each row. There are many ways to get the same change to Z with x & y in exact proportion plus some noise.

So the regressors are being assigned equal influence plus an equal share of the noise. It is the most parsimonious evaluation of x and y.

It is not what you expect knowing the equation, but it is the answer you should get. If you use a linear model which reduces variables of zero influent, you might even get a 0 or NA value for y.

To test that lm is working, simply reverse the order of y changing the relationship between x & y within that equations and you have a completely different result. I think the one you were looking to find.

y = np.arange(100,0,-1)

            coef    std err    t    P>|t|   [0.025  0.975]
Intercept   0.0439  0.000   119.009 0.000   0.043   0.045
x           1.2184  0.038   32.471  0.000   1.144   1.293
y           3.2130  0.038   85.629  0.000   3.139   3.288

You built your model right, but in this case the data set was not created in a way to test for what you hoped to see...but it was not wrong.


Normalize data with extreme outliers for forecasting

Play a bit with your data:

try doing the scaling without these outliers and see what happens
have you tried training the model regardless and see what happens? then train the model without the "outliers"
you could also binarise the feature (and thus transform it into a category) - then in this case you don't have to deal with the large range while the information is still in there.


How does the Transformer predict n steps into the future?

The Transformer is a seq2seq model.
At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the default way of training them. Note that, in the Transformer decoder, we need to apply masking to avoid the predictions depending on the current and future tokens.
At inference time, we don't have the target tokens (because that is what we are trying to predict). In this case, the decoder input in the first step would just be the sequence [], and we would predict the first token. Then, we would prepare the input for the next timestep appending the prediction to the previous timestep input (i.e. []), and then we would obtain the prediction for the second token. And so on. Note that, at each timestep, we are repeating the computations for the past positions; in real implementations, these states are cached instead of re-computed each timestep.
About some piece of Python code illustrating how the Transformer works, I suggest The annotated Transformer, which is a nice guide through a real implementation. You may be most interested in the function run_epoch for the training and in the function greedy_decode for the inference.
def greedy_decode(model, src, src_mask, max_len, start_symbol):
    memory = model.encode(src, src_mask)
    ys = torch.ones(1, 1).fill_(start_symbol).type_as(src.data)
    for i in range(max_len-1):
        out = model.decode(memory, src_mask,
                           Variable(ys),
                           Variable(subsequent_mask(ys.size(1))
                                    .type_as(src.data)))
        prob = model.generator(out[:, -1])
        _, next_word = torch.max(prob, dim = 1)
        next_word = next_word.data[0]
        ys = torch.cat([ys,
                        torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=1)
    return ys

In greedy_decode you can see how the predictions of the current timestep are concatenated to the input to create the input for the following timestep.


Learning Football Player Stats like FIFA's by only the game result

For that, you would probably need something more granular than just the score (e.g., in-game analytics, etc.) to help you assess the attributes of each player.  Even then, it would be difficult to map the outcome to multiple attributes (i.e., defending skills, offensive skills).

If you only care about a single score of a player (i.e., say, how effective is a player in helping her team to win, etc.), you could potentially look at something like Corsi which essentially measures puck possession (not without its criticism or flaws). There are many variants out there, but it should give you an idea how to leverage in-game analytics to assess a player.

But then, Corsi is merely a calculation from in-game statistic; you will need more than that to call it a machine learning exercise.


Intuitively speaking, ensembles benefit most from diversity.
Imagine being in a room of people making a decision together. If everyone more or less agrees, you don't benefit from having more people at the table. But if people tend to have different opinions, when they DO agree, it is a stronger message that the decision must be correct.
The same applies to ensembles. Models with high variance are more likely to produce different predictions, which will improve the quality of the prediction. High variance also minimizes the risk that multiple models are all wrong at the same time, based on the assumption that models are right more often than wrong.


What are non-machine-learning methods called?

Machine learning models work on principle of probabilistic approach where you try to fit the function to map input with output. There are no hard bound rules that classify the output based on input.

Deterministic approach like if else are categorized as  non machine learning algorithms where every time it follows the same rules to evaluate the results.


Building a Tic Tac Toe game which learns by itself

I took an introductory computer science course on Coursera a few years back where we had to use a Monte Carlo algorithm to build a tic-tac-toe AI.  Instead of using historical data, we generated data using a function that would play through a series of games with random moves, but I think the principle is basically the same.

Each square on the board has a separate score associated with it. For each game completed, the scores change based on which player won.  If the player designated as the machine player won, the machine player's score for each square on the board where it placed a mark would increase and the "other" player's score would decrease for every square where it placed a mark.  Conversely, if the other player won, its score would increase for each square on the board where it placed a mark and the machine player's score would decrease.  If the game is a tie, the number of points for each square would not change.

So, to answer your question, I think it is important to track both the winning and losing player's moves because we need to know not only what works but also what doesn't work.

Edit:  I just remembered that my first implementation did not deduct points on the grid for games lost and because of this, my AI had an all-out "offensive" strategy.  In other words, it focused on getting three-in-a-row without making any effort to block the human player.  This is why it's important to track moves that result in a loss.


Is it possible that a CNN has better accuracy than RNN in word classification?

CNN considers only the current input while RNN considers the current input and also the previously received inputs. It can memorize previous inputs due to its internal memory

Here's a comparative study on RNN vs CNN on different problems.

As you can see RNN-GRU outperforms CNN on most of the cases


Why is having high p-value makes dependent variable significant in backward elimination?

Actually, the Null Hypothesis is that the predictor is not significant. Taken from the book Introduction to Statistical Learning:

Null Hypothesis: There is no relationship between X and Y
Alternative Hypothesis: There is some relationship between X and Y

If we have a high p-value, we have an expressive result showing that the null hypothesis is correct and therefore the estimated coefficient come from a normal distribution around zero and can be discarded. I hope this answer your question, please let any comment if you need more help.


Is it correct to define the F-measure as the harmonic mean of specificity and sensitivity in such a way?

Yes, as they are effectively synonyms of one another. See for instance this link

If you pay attention, the first formula is the (weighted) harmonic mean of the recall and precision.


Does Homoscedasticity applies only for linear regression models?

It also applies to other methods, i.e. not just linear regression. For example, ANOVA and T-test also depend on homogeneity of variance.
One method to check the homogeneity of variance, compatible with the one-way ANOVA, is the Barlett's test.


What is the intuition behind decreasing the slope when using regularization?

Using regularization and shrinking the parameters, we reduce the sample variance of the estimates, and reduce the tendency to fit the random noise. The fitting to the noise is something that we wish to reduce. We can't increase the slope as we wan't to reduce overfitting.
L2 doesn’t necessarily reduce the number of features, but rather reduces the magnitude/impact that each features has on the model by reducing the coefficient value.
Shrinking can lead to positive effect when we have overestimated and negative when we have underestimated. But we are not shrinking everyone equally ,we are shifting with a factor that is larger if the estimate is larger away from zero.
Shrinking all the slopes towards zero will make some of them more accurate and some of them less accurate, but you can see how it would make them collectively more accurate.


Same SVM configuration, same input data gives different output using Matlab and scikit-learn implementation of SVM, in a classification problem

You have to be sure that the algorithm is the same and the kernel functions are really the same.
If you look at this python documentation page for kernels in scikit learn you will see there a description of poly kernel.

Notice that you have a gamma and a degree. Gamma is by default 'auto' which is evaluated at 1/n_samples. For the same kernel you have 'coef0' (a great name for a parameter), which is used in poly as the free term. I do not know how matlab put this values as defaults, but the usual formula for poly kernel in literature I found it to be $poly(x_1, x_2) = (<x_1,x_2> + 1)^{d}$. So no gamma and the free term is $1$. I think matlab uses that. (Anyway I found the 'improvements' in scikit learn to have a not so good smell).

Also in this SVC documentation page they state that there is a parameter called shrinking. I really do not know it's effect, but its auto, which means is enabled. Might be an issue.

Later edit:

I found this documentation page for svm in matlab which describes the kernel in the way I stated (no degree, free term with $1$). Also it states that 'SMO' is used by default, make it sure you use 'SMO' in python also.

On the other hand you have to understand that these kind of algorithms are solved by optimization methods which are usually iterative and to save some memory, or cycles their implementations can be different in small details, which will almost produce different results. I agree however that the results should be similar.


Checkers playing Neural Network evolved with Genetic Algorithm becomes too sensitive to input data changes

Full disclosure, I'm not at all an expert on genetic algorithms, so take this with a grain of salt.

First of all, I'm not sure the $r$ parameter in $w'=w*m'*r$ makes sense. You say it's a random standard Gaussian number, which means it could totally change the sign of $w'$. That seems bad. Am I understanding that correctly?

That aside, answering your actual question of controlling weights that get out of control: I would re-normalize the weights after each iteration, e.g. by dividing by its norm: $w'' = w' / ||w'||$. This will allow the weights to shift around, but will keep them from exploding.


What are the advantages/disadvantages of neural net packages in R?

take a look at these blogs- R for Deep Learning - 1 and R for Deep Learning - 2.

Hope it helps!


Can I use unsupervised learning followed by supervised learning?

You can definitely try to first cluster your data, and then try to see if the cluster information helps your classification task.

For example if your data looked like this (in 1D):

AA A AA A A      BBB B B B BB BB BB      AA AA A A AAA

then it may be reasonable to run a clustering algorithm on each class, to obtain two different kinds of A, and learn two separate classifiers for A1 and A2, and just drop the cluster distinction for the final output.

Other common unsupervised techniques used include PCA.

As for your football example, the problem is that the unsupervised algorithm does not know what it should be looking for. Instead of learning to separate american football and soccer, it may just as well decide to cluster on international vs. national games. Or Europe vs. U.S.; which may look like it learned about american football and soccer at first, but it put american soccer into the same cluster as american football, and american football teams in Europe into the Europe cluster... because it does not have guidance on what structure you are interested in; and the continents are a valid structure, too!

So usually, I would not blindly assume that unsupervised techniques yield a distrinction that matches your desired result. They can yield any kind of structure, and you will want to carefully inspect what they found before using it. If you use it blindly, make sure you spend enough time on evaluation (e.g. if the clustering improves your classifier performance, then it probably worked as intended ...)


Sequence pattern mining on continuous dataset

The established procedure is called symbolic aggregate approximation or SAX in which you first first do piecewise aggregate approximation (PAA), namely send non-overlapping mean (boxcar) windows of width $w$ over the data, and then transform the obtained values into symbols by equal-frequency-splitting the resulting data distribution into $k$ letters of the alphabet.

How to determine appropriate $w$ and $k$? They are a tradeoff between dimensionality reduction and information loss. One way to adapt $w$ is to look at window' data points variance; if the variance is too large, reduce the window size, and vice versa. Another information-theoretic is MDL as explained in this question.

Since you only have 12 values in a series, the windowing step may not be necessary.

Alternatively, you could try reshaping the data into three series:

$$Dd_1^{(c_1)}\dots{}d_{12}^{(c_1)}Dd_1^{(c_2)}\dots{}d_{12}^{(c_2)}D\dots$$

$$Uu_1^{(c_1)}\dots{}u_{12}^{(c_1)}Uu_1^{(c_2)}\dots{}u_{12}^{(c_2)}U\dots$$

$$Bb_1^{(c_1)}\dots{}b_{12}^{(c_1)}Bb_1^{(c_2)}\dots{}b_{12}^{(c_2)}B\dots$$

where D, U, and B are string literals (used as start markers) and $d_m^{(c_i)}$, $u_m^{(c_i)}$, and $b_m^{(c_i)}$ are the values of download, upload and billed, for customer $c_i$ in month $m$. Then you could use one of the frequent subsequence mining algorithms on those three sequences (separately) to find any prevaling patterns.


How can l get 50 % examples in training set and 50% in test set for each class when splitting data?

sklearn version 0.17 onwards the train_test_split should give you stratified results by using the stratify parameter.

Example code:

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(final_data, true_label, test_size=0.50, random_state=42, stratify=true_label)

From the documentation about the parameter stratify:

  stratify: array-like or None (default is None) If not None, data is split in a stratified fashion, using this as the labels array. New in version 0.17: stratify splitting

Hope this helps!


Remove Outliers from Dataframe using pandas in Python

You just don't have enough data in your dataset.

More accurately - your outliers are not affected by your filter function.

fence_low is equal to -35.974423375

fence_high is equal to 79.858537625

So the values of 0.01 are lying within this range.

I created an example notebook for you to show the difference you can check it out here:
https://github.com/OzmundSedler/learning/blob/master/IQR%20example.ipynb

Long story short - here is your data:

And here is the data, where I have added 15 more values in the range of [20:40]. In that case, 0.01 points are classified as outliers and filtered correctly.


How to set weights in multi-class classification in xgboost for imbalanced data?

As you say, scale_pos_weight works for two classes (binary classification). weight can be used for three or more classes. The parameter goes into the xgb.DMatrix function and must contain one value for each observation.
Example:
library(xgboost)
data(iris)

# We'll predict Species
label = as.integer(iris$Species)-1
iris$Species = NULL

# Split the data for training and testing (75/25 split)
n = nrow(iris)
train.index = sample(n,floor(0.75*n))

# For example, pick a weight of 1.5 for label "0", 1.0 for the other Species
weights = sapply(label[train.index], function(x) {ifelse(x == 0, 1.5, 1.0)})

# Train the data using weights
xgb.train = xgb.DMatrix(data=as.matrix(iris[train.index,]), label=label[train.index], weight = weights)

A similar question can be found here.


Can date data be used as a feature?

Yes.

  If so how do I handle them ?

Think about your problem. Why should the date be a reasonable indicator for the success of a startup? Answering this question tells you also in which way you need to transform it.

Most often, when I use some date information for models, I do the following:

Day of the week: integer/one-hot encoding for Monday, Tuesday, ..., Sunday
Month: integer/one-hot encoding for January, February, ..., December
Hour of the day: integer/one-hot encoding for 0, ..., 23
Seconds/minutes/hours/days since XY: Usually normalized or at least scaled in some way


Guided topic modeling: generating words from topics

I think instead of building something from scratch, you should utilize the domain knowledge of experts.

One example: Turn Wikipedia's topic portals (e.g., the medicine portal) into
a family tree.


Cost sensitive learning and class balancing

Not clear if you are saying the cost of a FN or a FP is higher, you only mention FN in your statement. Think that you mean a FN is more costly and that a positive means a 1.
In general, if an incorrect prediction for the minority case is more costly (FN), you should sample that minority case higher or the majority case lower so the ratio is closer to 1:1. Balancing will help increase the accuracy of your model when predicting the minority case. Accuracy in predicting the majority case will already be higher as there are more samples to use for that case. Undersampling, oversampling and SMOTE are all useful ways to accomplish this balancing of samples, and each has their own strengths and weaknesses.
However, doing this sample balancing will quickly increase the number of FP, so even though the cost may be lower for a FP, the cost will add up quickly. For example, every 1 FP that you decrease, you may get 10 or 20 more FN
After doing this balancing, you can start to adjust the weights to get the best ratio of FN to FP, trying to get the total cost as low as possible.
minimizing: total cost = FN x cost_of_fn + FP x cost_of_fp
Not sure if there is a mathematical equation to solve this, but you can run this iteratively, change weight ratios for the 2 classes, and calculate the total cost using a confusion matrix to get FN and FP, and graph the results for cost (y) vs weight ratio(x), looking for a minima. I would start with a ratio that is equal to the ratio of your costs.
Example: If the cost of a FN is 10 dollars and the cost of a FP is 1 dollar, then the ratio should be 10:1 for minority:majority class
A straightforward calculation shows how to derive the desired mix of class i: μ(i) in terms of the calculated weight w(i) and number of examples in the training set (before any up or down sampling) n(i). This formula is given below:

Reference: https://medium.com/rv-data/how-to-do-cost-sensitive-learning-61848bf4f5e7


Extracting Part of Speech (Source and Destinations) using text mining/NLP?

Named Entity Recognition is technique which can be used here. Location is one of the 3 most studied classes (with Person and Organization). Stanford NLP has an open source Java implementation that is extremely powerful.

For Example  let say sentence is "i will be going to Sweden from Boston."

Now here you can use regular expression to detect these LOCATION tags.


Why are discriminative models denoted as $P\left(Y|X\right)$?

Discriminative models output some tensor $Y$ describing a label given some input tensor $X$ describing some input data. The method of mapping $X$ to $Y$ might be deterministic or not - there's no need for it to be (consider the example of taking a deterministic model and adding a small amount of random noise to it, so that it is perhaps $1\%$ less accurate and more random). The accuracy of the model is the average accuracy over the validation/testing data. The model estimates the probability that the label is $Y$ given the input data is $X: P\left(Y|X\right)$.

Generative models generate a tensor that is drawn from a learned distribution that is similar (as a result of the learning) to the distribution of some training data. For sequence generation, the probability of a generated tensor ($X$, say) at time t is dependent on the probability that the tensor generated at time step $t-1$ was correct ($P\left(Y\right)$, say) and the probability of the new tensor ($X$) if the previous one is taken as given (probability of $X$ given $Y$, or $P\left(X|Y\right)$). The probability of the recently-generated tensor is $P\left(X|Y\right) \cdot P\left(Y\right)$.

I don't really like the notation in the text you're reading, and I agree with you that it's confusing. However, in answer to your question "Why do we denote $P\left(Y|X\right)$ for Discriminative models": model outputs are always estimates and should ideally reflect the fact that the input data doesn't perfectly predict the class. That is, even deterministic models should communicate their estimates in the language of probability; for example even a deterministic model might give an output that can be interpreted as "$80\%$ of the time, input data with these features means the class is C"


I've played before with a library for ("classic") NMF from University of Vienna: libNMF

While it wasn't useful for me at first, since they don't implement Convex-NMF, I had a look at their NNDSVD implementation.
They simply replace all negative values and values smaller than the machine epsilon with zero.

I tried that, but it fails, since the update rules for Convex-NMF are such that I get a division by zero. Since they add 0.2 to all values in their proposed initialization scheme (for "smoothing"), I just did the same here.

It works, the precision is worse than expected (a bit worse than k-means, which is my best approach for now), but I'll tweak it a bit more.


What you need is simply a language model. This is a very common task so you should be able to find code and data easily. This question gives some pointers for Python (be careful, the accepted answer is incorrect according to the two other answers).
Applying the language model to a sentence gives you a probability (or a perplexity score,  which works the opposite way), so you have to define a threshold in order to classify as real language or not.


How to combine GridSearchCV with Early Stopping?

Just to add to others here. I guess you simply need to include a early stopping callback in your fit().

Something like:

from keras.callbacks import EarlyStopping
# Define early stopping
early_stopping = EarlyStopping(monitor='val_loss', patience=epochs_to_wait_for_improve)
# Add ES into fit
history = model.fit(..., callbacks=[early_stopping])


Can identifiers be used to train a model?

You can use any variable during model training if this variable is available during inference time. This is the only technical restriction.
Another question is should you include this column or not. If this column is completely unique and does not have any relevant information in it then you can discard it. Also, categorical columns with high cardinality (number of unique values) might also have a negative impact.
On the other hand, if including this identifier improved the score a lot (not a random fluctuation) then there are might be more to it and you need to share more info on the dataset and used models.


What is the best way to replace NaN values for ranked columns

Your legend clearly states that missing values mean that there is no basement. You could fill the missing values with ’NoBase’ to make that point clearer (train[‘BsmtQual’].fillna(‘NoBase’, inplace=True)).
When you rank them then, you just add another ranking for ‘NoBase’, maybe 0, based on your example rating that gives a higher value to more quality.


What is a better approach for cross-validation with time-related predictors

Your approach 1) is correct, use n ordered data to predict n+1.
You will have to identify the correct window for prediction and a model not too flexible if you feel the amount of data you have is little.

Do not forget about feature engineering and data preparation. The seasonality you mention can be removed if you identity it correctly.

Validation of the model is done as usual, with a square loss function


Converting VOC2012 annotation .xml to segmentation .png

This is Deeplab's bash script for preprocessing the dataset which may help - https://github.com/tensorflow/models/blob/master/research/deeplab/datasets/download_and_convert_voc2012.sh

Some people reported running into some problems reading the annotations, which is covered here https://stackoverflow.com/questions/51676447/python-use-pil-to-load-png-file-gives-strange-results?noredirect=1#comment90316222_51676447


Normal distribution instead of Logistic distribution for classification

Nice comparison.

Generally, we are allowed to experiment with as many distributions as we want, and find the one that suits our purpose. However, the normality assumption leads to an intractable derivation consisting of the notorious erf function.

Let's first pinpoint what is $x$ in the context of logistic regression. Logistic regression model can be written as:
$$P(y=1|\boldsymbol{x})=\frac{1}{1+e^{-\boldsymbol{w}^t\boldsymbol{x}}}=F(\boldsymbol{w}^t\boldsymbol{x})$$
So your $x$ is actually $z=\boldsymbol{w}^t\boldsymbol{x}$. This means, although it is reasonable to assume that predicate $\boldsymbol{x}$ comes from a normal distribution, the same argument does not hold for a linear combination of its dimensions, i.e. $z$. In other words, the normal assumption is not as natural for $z$ as for $\boldsymbol{x}$.

But still, let's see what happens with normal assumption. The problem that we face here is analytical intractability. More specifically, to fit a similar model to observations using Maximum Likelihood, we need (1) derivative of cumulative distribution function (CDF) with respect to each parameter $w_i$, and (2) value  of CDF for a given $z$ (see this lecture section 12.2.1 for more details).

For logistic distribution, the required gradient would be:
$$\begin{align*}
\frac{\partial F(\boldsymbol{x};\boldsymbol{w})}{\partial w_i}&=\frac{\partial (1+e^{-\boldsymbol{w}^t\boldsymbol{x}})^{-1}}{\partial w_i}= x_i e^{-\boldsymbol{w}^t\boldsymbol{x}}(1+e^{-\boldsymbol{w}^t\boldsymbol{x}})^{-2} =x_if(\boldsymbol{x};\boldsymbol{w})
\end{align*}$$

However for normal distribution, CDF is the erf function which does not have an exact formula, though, its gradient is tractable. Assuming $z \sim \mathcal{N}(0, 1)$, the gradient would be:
$$\begin{align*}
\frac{\partial F(\boldsymbol{x};\boldsymbol{w})}{\partial w_i}&=\frac{\partial \left(\frac{1}{2}+\frac{1}{2}\text{erf}\left(\frac{z}{\sqrt{2}}\right)\right)}{\partial w_i}=\frac{x_i}{\sqrt{2 \pi}} e^{-\frac{(\boldsymbol{w}^t\boldsymbol{x})^2}{2}}=x_if(\boldsymbol{x};\boldsymbol{w})
\end{align*}$$

In summary, the normality assumption is not as justified for $z=\boldsymbol{w}^t\boldsymbol{x}$ as for $\boldsymbol{x}$, and it leads to an intractable CDF. Therefore, we continue using the good old logistic regression!

Here is a visual comparison of normal and logistic CDFs:

taken from a post by Enrique Pinzon, which implies a large analytical cost for a small difference!


Implement Sliding Window Algorithm to Detect Spikes

welcome to DS-SE and to Data Science in general! :)

Your problem can be solved really easily in Python. Please, take a look at Pandas DataFrame class to represent your data, it makes it really convenient because of all the pre-built methods that includes.

One of its methods is pandas.DataFrame.rolling() (see here) that does exactly what you asked for: rolling (sliding) window calculations, please take a look!

Hope it helps!


normalizing data and avoiding dividing by zero

While you could do this manually, Python also has a handy little function called MinMaxScaler, which will automatically apply max-min normalization to scale data between 0 and 1.

Assume we have an array of 200 values for variables s and t:

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

mu, sigma = 20, 10 # mean and standard deviation
s = np.random.normal(mu, sigma, 200)
t = np.random.normal(mu, sigma, 200)

Reshape your variables if necessary:

s=np.reshape(s,(-1,1))
t=np.reshape(t,(-1,1))

Now, you can see that we are forming two new variables, snew and tnew, which we are scaling using MinMaxScaler.

scaler = MinMaxScaler()
print(scaler.fit(s))
print(scaler.fit(s))
snew=scaler.transform(s)
tnew=scaler.transform(t)

Here is a sample of our new variables:

>>> snew
array([[0.24896606],
       [0.63121206],
       [0.60448469],
       .......
       [0.49044733],
       [0.28131596],
       [0.32909155]

>>> tnew
array([[0.91224005],
       [0.74540598],
       [0.3938718 ],
       .......
       [0.75749275],
       [0.80709325],
       [0.19440844]


How Calculate Effect (percentage) label of the input variables on the output variable by BernoulliNB

A Naive Bayes model consists of the probabilities $P(X_i|Class)$ for every feature $X_i$ and every label $Class$. So by looking at the parameters of the model one can see how important a particular feature is for a particular class. the opposite could be calculated as well: $P(Class|X_i)$ represents the distribution of the classes given a feature.
Now at the level of individual instances it's not so clear what would be the "effect" of a particular feature: For every class the posterior probability is:
$$P(Class| X_1,..,X_n) = \frac{P(Class)\prod_i P(X_i|Class)}{P(X_1,..,X_n)}$$
You can easily order the features by how much they contribute to the prediction, i.e. the class which obtains the maximum posterior probability (for instance obtain the top 3 features). However you cannot quantify precisely the effect of each feature, because the prediction is not a linear combination of the features.

[Details added following comments]
Due to the NB assumption that features are independent, we have:
$P(Class|X_1,..,X_n) = \prod_i P(X_i|Class)$
$P(Class|X_1,..,X_n) = P(X_1|Class) * P(X_2|Class) * .. * P(X_n|Class)$
From the conditional definition:
$P(Class|X_1,..,X_n) = P(Class,X_1,..,X_n) / P(X_1,..,X_n)$
which gives:
$P(Class,X_1,..,X_n) = P(Class) * P(Class|X_1,..,X_n)$
$P(Class,X_1,..,X_n) = P(Class) * P(X_1|Class) * P(X_2|Class) * .. * P(X_n|Class)$
Now we use the marginal to calculate $P(X_1,..,X_n)$:
$P(X_1,..,X_n) = \sum_j P(Class_j,X_1,..,X_n)$
$P(X_1,..,X_n)  = P(Class_1,X_1,..,X_n) + .. + P(Class_n,X_1,..,X_n)$
So at the end we have $P(Class,X_1,..,X_n)$ and $P(X_1,..,X_n)$, so we can calculate:
$P(Class|X_1,..,X_n) = P(Class,X_1,..,X_n) / P(X_1,..,X_n)$
Note that if you do all these steps you should obtain the same probability for $P(Class|X_1,..,X_n)$ as the one returned by the function predict_proba.
Caution: the functions  feature_log_prob_ and class_log_prior_ don't give you the probability directly, they give you the logarithm of the prob. So you need to apply exponential in order to get back the probability.


Modulo as activation function in neural network

The modulo function is not monotonic.
Even though it has a positive derivative except at points where it is not differentiable.
It is generally not recommended to use a non-convex Activation function. Hence, it is better to avoid it or change the structure of problem.


Reshaping of data for deep learning using Keras

mnist.load_data() supplies the MNIST digits with structure (nb_samples, 28, 28) i.e. with 2 dimensions per example representing a greyscale image 28x28.

The Convolution2D layers in Keras however, are designed to work with 3 dimensions per example. They have 4-dimensional inputs and outputs. This covers colour images (nb_samples, nb_channels, width, height), but more importantly, it covers deeper layers of the network, where each example has become a set of feature maps i.e. (nb_samples, nb_features, width, height).

The greyscale image for MNIST digits input would either need a different CNN layer design (or a param to the layer constructor to accept a different shape), or the design could simply use a standard CNN and you must explicitly express the examples as 1-channel images. The Keras team chose the latter approach, which needs the re-shape.


Can we Create Neural network(Simple one such as Multi Layer perceptron) that only contains positive weights only?

One way to do that is to customize the formula to update weights.

Normally, the formula is something like this:

weight = weight - learning_rate * error

You can change it to:

weight = max(0, weight - learning_rate * error)

This will perform well or not? I have no idea. We can't say much about the guaranteed performance of neural network models in general.

However, if by good results, you mean a model with all positive weights, I believe this one will qualify.


Data Science education curriculum design and guidelines in Computer Science and other Disciplines

Yes, data science work do exist in domains of management and computer science.

I think data science is a huge domain in itself to include in any of the curricula. So, I would explain how data science is used in the domains:

Management:
Data Science used in a lot of ways. I use data science and analytics in every phase of the customer acquisition and retention funnel; which includes designing models for using the customer event tracking data and identify patterns and predict churn. Here is an example article from the Moz blog, which explains about the use of deep for churn prediction and customer retention.
Computer Science:
Most of the data science has roots in computer science basics. However, including the concepts of distributed computing would help the students understand and appreciate the concept of Big Data, when they later enter the industry.

So, I think data science is a huge domain in itself, and has it's share of unique applications in a wide range of domains. However, bulk of data science concepts and techniques have taken roots in math, stats and computer science basics and concepts. So, I think the knowledge of those domains would be helpful and suffice.

So, these are some links which can help in curriculum re-design for accommodating data science concepts in the management and CS curricula:

MBA speciality.
Curriculum of the Data Science and Business Analytics program at the Santa Clara University.
Curriculum of the Masters of Data Science in Virginia University.

The third bullet is something very close to how data science can be included or blended with the Computer Science degree.


Why in spectral clustering number of eigen vectors is same as number of clusters that we want?

Theoretically, log2(k) components could be enough.

But usually clusters are not that well balanced, that you could get all 2^l combinations of eigenvectors stable, usually one will mask the other.

If you go back to the theory of spectral clustering, you'd have one eigenvector for each connected component. Now the clusters are unfortunately connected, so we don't completely get this ideal situation, but you'd expect one eigenvector for each "almost component", too.


Why is a lower bound necessary in proofs of VC-dimensions for various examples of hypotheses?

Lets use the quotes from the book.

  To give an upper bound, we need to prove that no set $S$ of cardinality
  $d + 1$ can be shattered by $H$

By doing this, we are proving that $\text{VCdim}(H) < d+1$, but it does not necessarily mean $\text{VCdim}(H) = d$. For example, we may go for an easier-to-prove failure like sets with $2d+1$ members, and consequently prove $\text{VCdim}(H) < 2d+1$, although we know $\text{VCdim}(H) = d$. Therefore:

  To give a lower bound $d$ for $\text{VCdim}(H)$, it suffices to show that a set $S$
  of cardinality $d$ can be shattered by $H$

We need to prove the lower bound as well to show that $d \le \text{VCdim}(H) < d+1$, which implies $\text{VCdim}(H) = d$. The last equality means that for each size $1$ to $d$, $H$ can shatter at least one set $S$ with that size.

Note that for the $2d+1$ example, we will fail to prove $2d \le \text{VCdim}(H)$, therefore $\text{VCdim}(H) \neq 2d$, and consequently we must try to prove a smaller (probably harder to prove) upper bound $d+1$.


Machine Learning - Euclidian Distance Classifier exercise

Clearly your exercice is not very clear, at least it is not to me.
I guess you should consider $I$ as the variance or std deviation of the variable, just make sure that is a parameter of your code so you can change it later in case it is not what you assumed it was. Check up in your course if it corresponds to something, but it doesn't seem anything familiar to me.
Not sure what your second question means, but here are the vectors that need to be created according to me :

Vector of points (1st question), each points lives in 2-D space (feature size).
Vector of classes (2nd question), this vector looks something like [1, 0, 2, 2, 1, 1, ...]. For this vector, it would mean that first value of your vector of points belongs to class 1, second value to class 0 , third to class 2, ...

As the exercise is asking about euclidian distance classifier, you just need to create an algorithm that takes a point, computes the euclidian distance with each class center, and classify it in the closest class. (Not sure that kind of classifier can be considered Machine Learning but w/e).
You probably noticed it, but the mean values given are 3-D vectors, which is quite weird as I said that our points live in 2-D space. I guess the last value of each mean vector is the class of the point generated (just my guess, not blaming your teacher, but as I said your exercise isn't very clear). So according to that interpretation, the classes would be distributed according to distribution :

[0, 1], I variance or std deviation for class 2.
[0, 0], I variance or std deviation for class 1.
[1, 0], I variance or std deviation for class 0.

Hope it helps, if you have any remaining questions, feel free to ask.


Preprocessing for finetuned CNN model from pretrained models

If a network was trained let's say on centered input values (i.e. ranging from -1 to 1) with subtracted mean channel values computed on the ImageNet, it is desirable to keep this preprocessing to make fine-tuning easier.

From other hand, CNN sometimes can be surprisingly robust to this kind of input fluctuations and, unless you freeze the pre-trained part, can adapt to the changes in the input. So, if you just change mean channel values from default [103.939, 116.779, 123.68] to [128, 128, 128] this should work ok. Otherwise, just compare how the accuracy is improving during fine-tuning using the vgg16 preprocessing and your method.


Splitting hold-out sample and training sample only once?

In general, it's a good idea to split up your data into three sets:

Training Set (60-80% of your data)
Cross-Validation Set (10-20% of your data)
Test Set (10-20% of your data)

When you select a model using only a train and test set, you are selecting the model which performs the best on the test set after. This seems reasonable at first, but this is usually an overly optimistic estimate of your model's generalization error. This is because you are essentially fitting your model to another parameter $d$, where $d$ is your test set.

In order to address this problem, you can train a model on your training set and then choose the model that performs best on your cross-validation set. Now, this model is no longer fit to the test set and you can safely estimate the generalization error on the test set.

It sounds like you are already following this approach in selecting the model by choosing which one performs best on a 10-fold cross-validation on your training set. In this case, the error on your test set should be a good estimate of generalization error.

You also mentioned that randomness might affect your training due to outliers in your training data:

  ...as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness.

In machine learning, it is often assumed that our data is iid (independent and identically distributed) from some unknown distribution. On average, we should get data points that are representative of this distribution when we split up our data. A common practice to better ensure this is to randomly shuffle the entire dataset before splitting up your data.


Discriminator of a Conditional GAN with continuous labels

I think the answer is, you have to either create a very large number of categories, or do it completely differently than with the code I provided.  The paper as a good solution to the problem.


How to decide the number of trees parameter for Random Forest algorithm in PySpark MLlib?

Maybe you will start to have negative impact of performance by increasing the number of trees. But definitely, at some point, increasing the number of trees will not add much accuracy in your model.

The usual way is to perform a k-fold cross-validation for different number of trees (and any other combination of model's parameters) and choose the one with the best performance. If you have any question about cross-validation, check this page. Instead of using simple accuracy, try to use other metrics that take into account the complexity of the model, like AIC.

Obs.: the final accuracy of the algorithm should be calculated using a separate dataset, not used in the cross-validation (usually called test set).


Please check this website: http://dataportals.org/, its a collection of data portals from around the world. Might find here what you are looking for.


I think you should look into multivariate regression. You could use these variables (type, area etc) along with other factors such as seasonality. Create dummy variables such as date of the week, week number , month etc to capture seasonality. For example rainy month may have less bike demand. These are inherent features from the data.


Microsoft custom vision vs Tensorflow model?

There are many differences as these are inherently complete different products with different goals.

customvision

[+] cloud deployment comes out of the box (including a rest API)
[+] labelling tool to add data and label them
[-] you have no control over the learning algorithm
[-] difficult to run your model locally/completely for free

tensorflow (or any framework really)

[-] you need deploy your model yourself
[-] you need to manage your data yourself
[+] you have full control over your network and how you train it
[+] you can embed your model into your code, run it locally, whatever you feel like


Difference between convolution structures

In absolute, it is one CNN wich takes 3 inputs images. You could see it as 3 separate features extractors (CNN) which merge their results while trained together.
The author obtain 3 2D input from a 3D images by keeping 3 2D images; one in each plane.
Each of these images has multiple channel because they slices the input among the respective axis.

It is called a "multichannel" CNN because it uses differente representation (3 2D images) of the input (3D image) and therefore extract 3 times the features in order to merge then. (like looking an elephant from each side and above)


Considering that you are trying to predict values in the future according to a set of edges on you graph, you can do this in several ways. Humans predict some events of the future looking at the past, and you can model this with traditional time series forecasting methods, such as ARMA. You can build time series aggregating values for each segmentation and apply your forecasting.


When should the last action be included in the state in reinforcement learning?

If your environment fulfills the Markov property, there is no reason to include the actions $a$ in the state $s$, as the action $a_t$ that lead to the new state $s_{t+1}$ should not provide any additional information, i.e. the knowledge of the old action should not influence the reward and transition functions from $s_{t+1}$ onwards.

Therefore, if your environment is a proper MDP, there is no reason to include actions in the next state.

It might be helpful in some rare cases, where your environment is not actually Markov to include the last action in your state, but in that case you should also think about if there is a better way to represent your state, such that it becomes Markov.

In your concrete case, where the actions do not change the state, there is no reason to include the last action as part of your state.

Side Note:
If your problem is a simple discrete bandit, you will probably be better off by looking into bandit approaches, rather than using a DQN based approach.


Generating Similar Words (or Synonyms) with Word Embeddings (Word2Vec)

Word Mover’s Distance (WMD) is an algorithm for finding the minimum distance between multiple embedded words.

  The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

Source: "From Word Embeddings To Document Distances" Paper

In your problem, it will allow you to find that "Tacos Mexican Restaurants" is similar to "Burritos Taqueria" even though they share no common string literals.


Since one could see there is no difference in the minmal loss value, this could well be an issue of differnce in floating point precision with respect to CPU and GPU. You should try to cross check unit operations in the intermediate layers(Considering its an ANN) and look for possible discrepencies to make sure it is a floating point precision variance.


What is a good non cryptographic Hash for string feature translation?

See also: Neural Network parse string data?

I do not see a problem with using MurMur3 per se.

For the categorical labels, you can use one-hot encoding / one-of-k encoding.

For the strings, it's an application-specific question.  Presumably if you use exactly those strings as features, it will be very sparse.  The effect of this will depend on the algorithm that you are using, and how the training data compare to the data you see in practice.  You are running the risk that you will effectively either only create a traditional IP/email whitelist/blacklist OR throw out the feature altogether.

You must decide what you want (eg should a certain email address always get a certain output label?) and have some intuition about the application so as to generate more features from IP address and email address.  For example, from email address you can extract the local part (eg "john1972") and domain, and from each of those you can extract:
- length
- character tri-grams
- count/proportion of numbers to alphachars
- number of hyphens
- dictionary validity
...
(From domain you can also extract TLD and possibly subdomains.)
You can try to tokenise .  You can even hit external services to get information like number of Google hits, detected language, spam score etc.


What dataset was Stanford NER trained on?

The original paper mentions two corpora: CoNLL 2003 (apparently here now) and the "CMU Seminar Announcements Task". However according to the page linked in the question the actual NER was trained on a larger combination of corpora:

  Our big English NER models were trained on a mixture of CoNLL, MUC-6, MUC-7 and ACE named entity corpora, and as a result the models are fairly robust across domains.

So it might be difficult to obtain the exact original training data. However most of these corpora were compiled for some shared tasks and should be available online. There are probably more recent ones as well: a quick search "named entity recognition shared task" returns many hits.


The simulation aims at quantifying the uncertainty of the poll. Candidate A wins win with 51% of the votes. How likely is this?

The code simulates this poll (1000 times as you see in the range function) so that we get the probability of A winning the poll. You can calculate it with e.g.
dist.loc[dist.A>dist.B]['A'].count()/len(dist). You'll find here that A wins with a confidence of ~ 73%. With this you can answer questions like "How big should the size of polled population ($n$) be to get a confidence of 95%?"


EfficientNet: Compound scaling method intuition

Receptive field refers to the number of input pixels that a convolutional filter will operate on. There's a nice distill article about how to calculate receptive field size for your filters (with a nice visualization of receptive field size) and an interactive calculator here if you're only curious about how receptive field size grows with changes to depth and filter size.

Increases to receptive field size typically come from adding layers and from increasing the kernel size. A larger kernel operates on more pixels, which grows the receptive field. Increasing the depth of your network refers to adding additional convolutional layers. These downstream filters operate on the feature maps produced by the initial conv. layers of your net, which increases the receptive field for the filters in those additional layers (if this isn't clear, this is a good guide). The distill article also goes into detail about how other operations affect receptive field size.

As for the claim of a gain in number fine grain patterns captured, it's more in line with the intuition that more filters will give the network more ways to learn specific features of your data. See articles around visualizing convolutional filters for a sense of what types of features are captured (this tutorial on object detection links to a nice visualization).

Hope this helps!


Extract relevant vocabulary from a document

Not sure if this is what you are looking for. Couldn't understand your question properly do you want 4-grams then this is the way you could do it. Assuming you have cleaned your text file by eliminating articles.

from nltk import ngrams
file=open('abc.txt','r')
txt = file.read()
n = 4
fourgrams = ngrams(txt.split(), n)
for grams in fourgrams:
  print (grams)

In case your file abc.txt contains "This is random text to demonstrate the use of n-grams"

OUTPUT:
('This', 'is', 'random', 'text')
('is', 'random', 'text', 'to')
('random', 'text', 'to', 'demonstrate')
('text', 'to', 'demonstrate', 'the')
('to', 'demonstrate', 'the', 'use')
('demonstrate', 'the', 'use', 'of')
('the', 'use', 'of', 'n-grams')

If you do not want repetition then

words = 'This is random text we’re going to split apart'
x=[]
for word in words.split():
    x.append(word)
    if len(x) == 4:
        print(x)
        x=[]
print(x)

OUTPUT:
['This', 'is', 'random', 'text']
['we’re', 'going', 'to', 'split']
['apart']


When do you use FunctionTransformer instead of .apply()?

FunctionTransformer is useful because it allows you to apply a custom function in  a pipeline.  Because Pipeline() from sklearn.pipeline only works with objects that implement the .transform() and .fit() methods, you use FunctionTransformer to change your custom function to allow .transform() and/or .fit() to be used on it.

You could transform a DataFrame or Series by using .apply() (or something similar like a list comprehension), but you wouldn't be able to use that function in Pipeline() without first using Function Transformer.

(answer adapted from a DataCamp module "Multiple types of processing: FunctionTransformer" from the class "Machine Learning with the Experts: School Budgets")

Example:

# Import FunctionTransformer
from sklearn.preprocessing import FunctionTransformer

# Obtain the text data: get_text_data
get_text_data = FunctionTransformer(lambda x: x['text'], validate=False)

# Obtain the numeric data: get_numeric_data
get_numeric_data = FunctionTransformer(lambda x: x[['numeric', 'with_missing']], validate=False)

# Fit and transform the text data: just_text_data
just_text_data = get_text_data.fit_transform(sample_df)

# Fit and transform the numeric data: just_numeric_data
just_numeric_data = get_numeric_data.fit_transform(sample_df)

# Print head to check results
print('Text Data')
print(just_text_data.head())
print('\nNumeric Data')
print(just_numeric_data.head())

<script.py> output:
    Text Data
    0
    1        foo
    2    foo bar
    3
    4    foo bar
    Name: text, dtype: object

    Numeric Data
         numeric  with_missing
    0 -10.856306      4.433240
    1   9.973454      4.310229
    2   2.829785      2.469828
    3 -15.062947      2.852981
    4  -5.786003      1.826475


Why convolution over volume sums up across channels?

There are multiple points that I try to explain them.

First, each filter for convolutional networks for images is a 3d volume. Consequently, whenever it is said we have $n$ filters, means we have $n$ volumes of those 3d filters.

Second, you can consider each convolutional layer as an MLP which is applied to small regions of the input. These are applied to different regions of the input to investigate whether a typical pattern is in that region or not. These patterns are going to be learned by means of cost functions. You can easily consider that for each filter which is a volume, you are concretely doing a summation over weighted inputs, exactly as MLPs.

Third, as the result, your trained filters will decide to choose the information of which channels depending on the task using the cost function. They may be in a single plane or they may be among multiple of them.


By using the functional API you can easily share weights between different parts of your network. In your case we have an Input $x$ which is our input, then we will have a Dense layer called shared. Then we will have three different Dense layers called sub1, sub2 and sub3 and then three output layers called out1, out2 and out3.

x = Input(shape=(n, ))
shared = Dense(32)(x)
sub1 = Dense(16)(shared)
sub2 = Dense(16)(shared)
sub3 = Dense(16)(shared)
out1 = Dense(1)(sub1)
out2 = Dense(1)(sub2)
out3 = Dense(1)(sub3)

We can define our model now like this:

model = Model(inputs=x, outputs=[out1, out2, out3])

It will expect a tuple/list of three elements now, one for each output.

You can go a lot further with these concepts. Let's say we would like to learn individual weights for the person layers but still want to have the same weights for the linear combination towards the output layer, we could achieve that by doing this:

out = Dense(1)
out1 = out(sub1)
out2 = out(sub2)
out3 = out(sub3)

EDIT: Concatenating is basically the opposite of what you want to do, it's pasting (intermediate) outputs of different parts of your network into a new layer. You actually want to split off to multiple different parts.


By increasing batch size your steps can be more accurate because your sampling will be closer to the real population. If you increase the size of batch, your batch normalisation can have better results. The reason is exactly like the input layer. The samples will be closer to the population for inner activations.


Misclassification Rate for Random Forest Plateauing too Early

If you and your colleagues ran the same model on the same data you should get the same results (give or take a stochastic error). Did your colleagues use the same environment, same packages and same versions?

Also, it is known that building more trees gives better performance and if possible you should build more not less, as RF does not overfit with more trees, the error / accuracy stabilizes at some point. What that point is (number of trees) varies from data to data, so you cannot really determine this beforehand.


Improve performances of a convolutional neural network

It sounds like the model is underfit; to verify this, you would check that training and testing accuracy are similar, but, both lower than desired.

One cause is likely the very low number of training images for a model of this complexity.

You could try augmenting the training set with transformations of those images, for example by rotating the images in the training set? (This was done in the original alexnet paper: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf see section 4.1 on p.5).


I think you have listed the main ways of doing it - you can achieve it by iterating or merging. What is "best" will depend on your use case.

Here is a way to do it by iterating over a dataframe. This way gives you more control about what gets filled i.e. you could add more conditions on which values to fill in. I first make a new dataframe that has all the complete dates and alternating A B A B values in column C1:

import pandas as pd
import numpy as np

dates = pd.date_range(start="1/1/2019", end="1/10/2019")
repeated_dates = np.repeat(dates, 2)
df = pd.DataFrame(index=repeated_dates, columns=["C1", "C2"])
df["C1"] = (len(df) // 2) * ["A", "B"]

# See first 5 rows
print(df.head())

           C1   C2
2019-01-01  A  NaN
2019-01-01  B  NaN
2019-01-02  A  NaN
2019-01-02  B  NaN
2019-01-03  A  NaN

We will later fill the values of the C2 column.

Next make a dataframe (it would actually be your starting data), by dropping a few rows from our "results" dataframe above:

df_missing = df.drop(df.index[[3, 9]])

C2_col = []

# grouping by the index (i.e. the date) gives us two rows at a time
for date, group in df.groupby(df.index):
    try:
        # see which values your data has for this data and extract them
        day = df_missing.loc[date, ["C1", "C2"]]
        C2_A, C2_B = day.C2.values

    # If the date wasn't there, we can catch the error and give any values we want
    except KeyError as e:
        # Could now use more condition e.g. on the date or previous values, etc.
        C2_A = C2_B = "was_missing"

    # Keep the values in a list
    C2_col.extend([C2_A, C2_B])

# Overwrite the column that was full of NaN values
df["C2"] = C2_col

We can see in the final result that all dates are present, along with the A B A pattern, and we could insert whatever we wanted into those dates with missing values:

print(df)

           C1           C2
2019-01-01  A          NaN
2019-01-01  B          NaN
2019-01-02  A  was_missing
2019-01-02  B  was_missing
2019-01-03  A          NaN
2019-01-03  B          NaN
2019-01-04  A          NaN
2019-01-04  B          NaN
2019-01-05  A  was_missing
2019-01-05  B  was_missing
2019-01-06  A          NaN
2019-01-06  B          NaN
2019-01-07  A          NaN
2019-01-07  B          NaN
2019-01-08  A          NaN
2019-01-08  B          NaN
2019-01-09  A          NaN
2019-01-09  B          NaN
2019-01-10  A          NaN
2019-01-10  B          NaN


How Can I Compute Information-Gain for Continuous- Valued Attributes

In decision trees, the (Shannon) entropy is not calculated on the actual attributes, but on the class label. If you wanted to find the entropy of a continuous variable, you could use Differential entropy metrics such as KL divergence, but that's not the point about decision trees.

When finding the entropy for a splitting decision in a decision tree, you find a threshold (such as midpoint or anything you come up with), and count the amount of each class label on each size of the threshold. For example:

Var1 | Class
0.75 | 1
0.87 | 0
0.89 | 1
0.96 | 0
1.02 | 1
1.05 | 1
1.14 | 1
1.25 | 1

Let's assume a threshold of 0.99 because I want two equal-sized bins and that's the midpoint between 0.96 and 1.02. The classes on the left are [1, 0, 1, 0] (since it's half/half, entropy is 1.0), and on the right [1, 1, 1, 1] (since it's all equal, entropy is 0.0).

To make a table like the one you show here, you need to define N threshold points (2.0, 3.5, ..., 7.5, in your case), which will produce N+1 bins of data, you take the class of each bin and calculate the entropy. A more correct visualization to your table should be changing the "Split point" values to "2.0 or lower", "Between 2.0 and 3.5", "Between 3.5 and 4.5", ..., "7.5 or higher".

Note this all applies to classification (decision) trees, and not regression trees, where the "class" is actually a continuous variable.


Here's an easy solution. The sort order changes, but that shouldn't be difficult to address if you really care:

import pandas as pd

df = pd.DataFrame(a)
df.columns = ['a','b','c','d']
df.index = df.a.astype(str) # to preserve correspondence
df.groupby('a').min()['b']

a
13.0    25.0
24.0    87.0
25.0    29.0
27.0    25.0
62.0    62.0
64.0    57.0
Name: b, dtype: float64

Edit: I think you meant to name your array y instead of a. This works for me:

 from decimal import Decimal

 y=np.array([[196512, 28978, Decimal('12.7805170314276')],
    [196512, 34591, Decimal('12.8994111000000')] ,
    [196512, 13078, Decimal('12.9135746000000')] ,
    [196641, 114569, Decimal('12.9267705000000')] ,
    [196641, 118910, Decimal('12.8983353775637')] ,
    [196641, 100688, Decimal('12.9505091000000')]])

 df = pd.DataFrame(y)
 df.columns = ['a','b','c']
 df.index = df.a.astype(str)
 dd=df.groupby('a').min()['c']

In [210]: dd
Out[210]:
a
196512    12.7805170314276
196641    12.8983353775637
Name: c, dtype: object


A general rule of thumb is to not distribute until you have to. It's usually more efficient to have N servers of a certain capacity than 2N servers of half that capacity. More of the data access will be local, and therefore fast in memory versus slow across the network.

At a certain point, scaling up one machine becomes uneconomical because the cost of additional resource scales more than linearly. However this point is still amazingly high.

On Amazon in particular though, the economics of each instance type can vary a lot if you are using spot market instances. The default pricing more or less means that the same amount of resource costs about the same regardless of the instance type, that can vary a lot; large instances can be cheaper than small ones, or N small instances can be much cheaper than one large machine with equivalent resources.

One massive consideration here is that the computation paradigm can change quite a lot when you move from one machine to multiple machines. The tradeoffs that the communication overhead induce may force you to, for example, adopt a data-parallel paradigm to scale. That means a different choice of tools and algorithm. For example, SGD looks quite different in-memory and in Python than on MapReduce. So you would have to consider this before parallelizing.

You may choose to distribute work across a cluster, even if a single node and non-distributed paradigms work for you, for reliability. If a single node fails, you lose all of the computation; a distributed computation can potentially recover and complete just the part of the computation that was lost.


I would not necessarily call it data issues. There is always some threshold that you just can not surpass, depending on the dataset ofcourse. Generally feature engineering and understanding the data will yield much greater increases than just hyp.par. optimization, which as you can see from the picture, yields often marginal increases (there is a case where its worst than default parameters)


Statistical methods for Sequence learning

it sounds like a stochastic process problem. Have you looked into estimating transition matrices for markov chains?


test_train_split with stratify integer overflow

How about this?

# Split the data between the Training Data and Test Data
xTrain , xTest , yTrain , yTest = train_test_split(X , y ,
                                                  test_size = 0.30 ,
                                                  random_state = 0,
                                          ----->  stratify = y)