Anomaly detection on multidimensional time series

One option is to treat it as a "supervised learning" problem.

Since

  a set of n data, measured every x ms

You have a time series. This can be treated as a supervised problem.

Say that you have observations for 10 minutes. In supervised learning, each sample would be :

X : Set of last t-1 , t-2,  t-3 and t-N observations (N is fixed, say 300)
Y : Observation t

You can train a model to predict next observation, given last 300 observations. If actual value is beyond a  certain % from the prediction, it is an anomaly.

Example : https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/


Training AI to do repetive retouching job

This is a very (very) broad question.

What you are trying to do is utilize some neural model to perform image processing. So your output needs to be the same size as your input, meaning that you need to use a fully-convolutional model for this.

I suggest you read the following article:

Fast Image Processing with Fully-Convolutional Networks

They are doing exactly what you need, only for different image-processing methods (and more than one).

However! I think that before going on to implementing a neural model, you should read a bit about the general subject of machine learning and deep learning. How Do I Learn Neural Networks?


Using K-fold cross-validation in Keras on the data of my model

The cross_val_score seems to be dependent on the model being from sk-learn and having a get_params method. Since your Keras implementation does not have this, it can't provide the necessary information to do the cross_val_score. Try the manual k-fold cross validation found here: https://machinelearningmastery.com/evaluate-performance-deep-learning-models-keras/


Elastic Regression fitting good mean and bad variance

Your model actually looks pretty good. What it sounds like you are asking to do is to overfit your model. I would not recommend that you do that. You can do that by finding more variables that you can input into the model, fitting extra polynomial terms or anything else like that, fitting a neural network will potentially do it for you too. However, you generally want to smooth your predictions out, like what you have.

One thing that you could try is to add an autocorrelation term. That might cause your model to behave as you intend. With negative autocorrelation your predicted values will have a tendency to bounce back and forth around the mean. But I wouldn't recommend doing that, your performance will probably suffer, just judging by the graph that you provided.


Frequency of occurrence - dummy variables

There's no general rule that can apply to all cases, and there's a lot of context missing in your post to say anything conclusive.

Having said that, I think that a good approach is to treat the cities with a lot of occurrences each on its own, and then group all others under a 'other' category.

Going further, you could have multiple 'other' groups, grouped by various criteria, for example, geographical criteria, or anything that makes sense in your context.

Hope this helps.


How to solve this classification problem: multi-class or multi-label?

So, in regards to your two questions:

In terms of including the screening status as a label: this really depends on what exactly you want your model to do and the applications of this model. If you want this model to help clinicians understand which patients should be screened and which shouldn’t, then of course include this feature as an additional predictor.

In terms of screening status prediction task being multi-label or multi-class: in this case the classes appear to be mutually exclusive. Therefore, this would be a multi-class classification problem. A multi labelling problem is for example predicting which combination of symptoms would appear if a patient experienced a certain disease.


Why will the accuracy of a highly unbalanced dataset reduce after oversampling?

Imagine that your data is not easily separable. Your classifier isn't able to do a very good job at distinguishing between positive and negative examples, so it usually predicts the majority class for any example. In the unbalanced case, it will get 100 examples correct and 20 wrong, resulting in a 100/120 = 83% accuracy. But after balancing the classes, the best possible result is about 50%.

The problem here is that accuracy is not a good measure of performance on unbalanced classes. It may be that your data is too difficult, or the capacity of your classifier is not strong enough. It's usually better to look at the confusion matrix to better understand how the classifier is working, or look at metrics other than accuracy such as the precision and recall, $F_1$ score (which is just the harmonic mean of precision and recall), or AUC. These are typically all easy to use in common machine learning libraries like scikit-learn.


The idea behind word2vec is to represent words by a vector of real numbers of dimension d. Therefore the second matrix is the representation of those words.

The i-th line of this matrix is the vector representation of the i-th word.

Let's say that in your example you have 5 words : ["Lion", "Cat", "Dog", "Horse", "Mouse"], then the first vector [0,0,0,1,0] means you're considering the word "Horse" and so the representation of "Horse" is [10, 12, 19]. Similarly, [17, 24, 1] is the representation of the word "Lion".

To my knowledege, there are no "human meaning" specifically to each of the numbers in these representations. One number is not representing if the word is a verb or not, an adjective or not... It's just the weights that you change to solve your optimization problem to learn the representation of your words.

This tutorial may help : http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/ even though I think the image you put was from this link.

You can also check this, which may help you get started with word vectors with TensorFlow : https://www.tensorflow.org/tutorials/word2vec


Methods for standardizing / normalizing different rank scales

For item-ratings type of data with the restriction that an item's rating should be between 1 and 10 after transformation, I would suggest using a simple re-scaling, such that the item's transformed rating $x_t$ is given by:

$$x_t = 9\left(\frac{x_i - x_{min}}{x_{max} - x_{min}}\right) + 1$$

where $x_{min}$ and $x_{max}$ are the minimum and maximum possible rating in the specific scale for the item, and $x_i$ is the item rating.

In the case of the above scaling, the transformation applied is independent of the data. However, in the normalization, the transformation applied is dependent on the data (through mean and standard deviation), and might change as more data becomes available.

Section 4.3 on page 30 of this document shows other ways of normalizing in which your restriction (transforming to the same absolute scale) might not be preserved.


Minimum number of features for Naïve Bayes model

The number of features to get a minimum accuracy is an empirical question, that will depend on the specific problem. There are problems where a single feature will result in "good enough" performance, there are other problems where no number of features is sufficient.
Naive Bayes does provide the most predictive features for each class. So the increase in performance by adding features can be calculated.
For your specific problem, more features (more text signals) and more observations (more of Wikipedia) would increase accuracy.


The right answer is the fourth.

From this, the formula to calculate the number of parameters in a convolutional layer is (nml+1)*k with n = m = 5, k = 100, l = 3 and +1 for the bias.


Tests to find which theory agrees better with observation

I'm not sure about the best statistical way, but one practical approach would be to compare actuals v expected for each of the two hypothetical curves.

Calculating the sum of squared differences (i.e. sum((y_obs - y_theory)^2)) will give you a numerical estimate of goodness-of-fit for each curve. You could also use other metrics depending on your application (e.g. RMSE, RMSLE, SMAPE...).

Or, for each theoretical curve, you could look at the distribution of differences between the actual observation and the value predicted by the curve. It can be handy to view these empirical differences graphically.

I'm sure there are other statistical ways to evaluate goodness-of-fit.


Can I use confident predictions to correct incorrect labels?

At the moment, you don't really know the performance of your model because you have quite a few wrong labels in your test set.

You mentioned that you want to use the new probabilities to correct the validation and test sets. However, if you do that, you will of course get higher results because you are using the labels coming from your own model. However, if you verify by hand than it is ok.

Therefore, what I would suggest is:

Train the classifier with all your data. Don't do any splits.
After training, pass all your data through the model. Separate all the data that the model gets wrong.
Manually check all the mistakes. You will have a few scenarios: The model gets is wrong but the label was correct) in this case, leave the label as it was. The model gets it wrong but was wrong) this are the interesting cases for you, where the model is wrong but in fact it is correct since the original labels were wrong. Correct these.
Now perform train/validation/test splits.
Retrain your model with the train/validation splits.
Test your model with the test split.


What commercial software should a data scientist purchase?

If you are looking for a technical package I would definitely go with Mathematica. It has functions for a very wide range of disciplines that come standard with it (no extra packages to buy). This allows you to expand into techniques as you grow without having to change platforms or buy extra packages.  It also has a great interactive format in CDF with both a free and professional reader. The documentation is interactive as well which really helps a lot.

However, you have to be open to a little bit of structure, typing & chaining functions together, and things like this.  Once you get started it comes very quickly.


Getting wrong ch2 values from sklearn chi2

I don't know what is your source that said that value of chi-square should be between 0 and 1. Imagine as in that equation in the picture say two terms t and c always co-occur, meaning D(t|~c)=0 (number of documents with no c but t in them) and D(c|~t)=0 (number of documents with c in them but no t). This means the equation in the snap reduces down to

$\chi^2(t, c) = N*(AD)^2/A^2D^2 = N$.

Edit 1:

After reading your comment, I understood you are confused regarding the 'normalised' part. A normalised value does not always mean a scaled value between 0 and 1. Normalised value does mean a value that is scaled appropriately for comparing and in my opinion, the denominator gives that comparable effect. The type of normalisation that you are associating this situation to is feature scaling. Check out normalization.


I do something similar with keras and GPU training, where i also have only a small memory amount available. The idea would be split the numpy files into smaller ones, let's say 64 samples per file and then load each file and call train_on_batch on those images. You can use keras' train_on_batch function to achieve this:

train_on_batch

train_on_batch(self, x, y, class_weight=None, sample_weight=None)

It's designed to perform a single gradient update over one batch of samples.

Custom Generator

Another idea is to use generators which provide you with data given a directory. They can also be used for data augmentation, i.e. randomly generating new training data from your data. Here is an example from the keras documentation:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

# just rescale test data
test_datagen = ImageDataGenerator(rescale=1./255)

# this generator loads data from the given directory and 32 images
# chunks called batches. you can set this as you like
train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# same es the train_generator
validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# loads sequentially images and feeds them to the model.
# the batch size is set in the constructor
model.fit_generator(
        train_generator,
        samples_per_epoch=2000,
        nb_epoch=50,
        validation_data=validation_generator,
        nb_val_samples=800)


Improve a regression model and feature selection

I think the next option you have to take is to add more features. You have a huge amount of training examples, which is good, but the number of features is very low. Adding more features is one of the most used methods to improve performance in machine learning.

Furthermore, it would be good to try to understand how your features affect your model. Imagine you have a linear model like y = theta1*feat1 + theta2*feat2 + theta3*feat3. If theta3 is near to 0, then feat3 is not affecting the model.


Trying to find the correlation between inputs and output

In a correlation framework above, the biggest driver of the output is the input which has the greatest absolute correlation value.

Correlation lies in the range [-1,1], and:

Negative correlation (correlation < 0) implies that the input and output move in opposite directions - i.e. as the input increases, the output decreases (and vice versa).
Nil correlation (correlation == 0) implies that the two variables are completely unrelated.
Positive correlation (correlation > 0) implies that the input and output move in the same direction - i.e. as the input increases, the output increases (and vice versa).

In the chart above, it looks like all-but-one of the inputs are negatively correlated with the output. This implies that as these inputs increase, the output decreases and vice versa.

A few things about your approach:

There is more than one type of correlation - Spearman (rank) and Pearson (linear) correlation are two examples. Be mindful of which you are using.
It would be helpful to rank/sort the result before plotting it. It would be easier for you to visually identify the drivers of the output if your chart was sorted.
You may also want to drop output from the chart. A variable's correlation with itself is always 1, and so this does not add any value to the graphic.


Yes, the new 2 dimensional values will be a projection of original 12 dimensional points onto the two principle components (vectors).

please refer to the first figure in this clear tutorial:
http://lazyprogrammer.me/tutorial-principal-components-analysis-pca/


In K-means, what happens if a centroid is never the closest to any point?

You should leave the lone centroid unmoved. In the next iteration, it's possible that the cluster centers have moved in such a way that now there are some instances that are closest to the lone centroid, and it can get picked up. At the end of the k-means algorithm, you could remove clusters that have no instances associated with them, but it doesn't really matter.

At the end of the day, it's probably not a good idea to use k-means in the manner that you described, and we should not expect it necessarily to produce good results.


How to train NER LSTM on single sentence level

Looks like this is done by the hyper-parameter called timestemps or sequence length. The steps/length indicates when the cell state is reset. Which can be set to a sentence length.

Just have to figure out whether this can be a variable length or the sentences should perhaps be padded?


Creating a generic mathematical formula using a genetic algorithm

I would suggest to use a sequence of operations of the form
op1(var) op2 op1(var) op2 op1(var) ...
where op1 refers to a unary operation and op2 refers to a binary operation. Making unary operations mandatory gets rid of a distinction of cases. If you don't want a unary operation in front of a variable, just use the identity function.

I wrote two implementations for unary composition of a list/tuple of unary operations. One in Haskell, one in Python:

square:: Num a => a -> a
square x = x * x

compose_multiple:: [a -> a] -> (a -> a)
compose_multiple []     = (\x -> x)                    -- for the empty case, use the identity
compose_multiple (f:fs) = f . compose_multiple fs  -- recursive case

This gives us 256 for

compose_multiple [square, square, square] 2

In Python:

def compose(op1, op2):
    def result(x):
        return op1(op2(x))
    return result

def compose_multiple(operations: tuple):
    if len(operations) == 1:
            return operations[0]
    return compose(compose_multiple(operations[:-1]), operations[-1])

This gives us 256 for

def square(x):
  return x * x
compose_multiple((square, square, square))(2)

An implementation for the composition of binary operations should work on the lines of the unary case.

The complexity of the final code can be reduced by using infix notation (which is easy to do in Haskell using parenthesis () in the function definition).
I have no experience using infix notation in Python. Maybe https://pypi.org/project/infix/ might help.


Differences between normalization and standarization in multiple regression

I'll go through your question one by one.

  1) Can someone explain why we have to transform dependent variable using log-transformation (Normalization) when appear positive skewed y variable in regression model?

Not necessarily log transformations, any kind of transformation (square, square-root, log, Z-scores, you name it) necessary to make the distribution of your data look more "Normal" (i.e. Gaussian). That is because all mainstream (frequentist) statistical models rely on the normality assumption of data (and residuals). When data are not Normal enough, the computation of parameters such as confidence intervals, standard errors, and p values will be unreliable.

  2) After log-transformation whether do I need to standardize that y variable using min max scale or StandardScaler methods?

That is not mandatory either. Sometimes it is useful to scale your dependent variable in a range such that all its likely values are "easy to reach" by the parameters of your predictive model.

  3) If independent variables also have skewed data then do I need to Normalization for that variables or standardizing is enough?

Normalization is a technique that can work. There is no general rule though. Sometimes you can use min-max scaling, some other times Z-scores work better. Other times again you'd achieve better performance with some custom scaling technique that invented ad hoc.


After differencing! A stationary time series is a key (as in, crucial) assumption of all ARIMA models. You are presumably trying to determine the orders of your AR and MA parts, using the Box-Jenkins methodology (looking at graphs of acf/pacf). In this case, if the true process is indeed AR/MA/ARMA then the series must be stationary by assumption (which means, differencing the time series if needed to make it stationary).
Look at both of the plots. For the AR(p) part of your model, look at the PACF plot and look for consecutive significant lags and then a sharp cutoff to zero (insignificant lags). For the MA(q) part, look for the same thing but using the ACF plot. If you see that one plot shows something as described above and the other not, then perhaps your model is only AR(p) or MA(q) rather than ARMA(p,q). If none of the plots seem to suggest anything, then likely you do not have a stationary time series and/or an ARMA/ARIMA model is inappropriate.


Is the direction of edges in a Bayes Network irrelevant?

TL;DR: sometimes you can make an equivalent Bayesian network by reversing arrows, and sometimes you can't.

Simply reversing the direction of the arrows yields another directed graph, but that graph is not necessarily the graph of an equivalent Bayesian network, because the dependence relations represented by the reversed-arrow graph can be different from those represented by the original graph. If the reversed-arrow graph represents different dependence relations than the original, in some cases it's possible to create an equivalent Bayesian network by adding some more arrows to capture dependence relations which are missing in the reversed-arrow graph. But in some cases there is not an exactly equivalent Bayesian network. If you have to add some arrows in order to capture dependencies, you might end up with a graph which represents fewer independence relations and therefore fewer opportunities for simplifying the computations of posterior probabilities.

For example, a -> b -> c represents the same dependencies and independencies as a <- b <- c, and the same as a <- b -> c, but not the same as a -> b <- c. This last graph says that a and c are independent if b is not observed, but a <- b -> c says a and c are dependent in that case. We can add an edge directly from a to c to capture that, but then a and c being independent when b is observed is not represented. That means that there is at least one factorization which we cannot exploit when computing posterior probabilities.

All this stuff about dependence/independence, arrows and their reversals, etc., is covered in standard texts on Bayesian networks. I can dig out some references if you want.

Bayesian networks don't express causality. Judea Pearl, who did a lot of work on Bayesian networks, has also worked on what he calls causal networks (essentially Bayesian networks annotated with causal relations).


How to interpret a random variable in the variable importance?

I'll go through your questions one by one:

  ... how to decide on n ?

This is a completely arbitrary choice that depends on your subjective preferences, the nature of your data, your problem at hand, the amount of computational power you can afford, you name it... There is no rule-of-thumb on this. What is the purpose of your analysis?

--

  If I do that, and then I re-run the model, including only the
  variables that score above the random one, then the accuracy score in
  the test set, decreases. Why ?

The variables that have an explanatory power below a given threshold are comparatively less relevant but not useless, they can capture some of the variance of your dependent variable. Removing them means renouncing to a little bit of predictive power. That's why accuracy decreases.

Hope this helps, otherwise let me know.


How to improve results for clustering of words

I cannot write comments yet, so I will start with an answer that still contains some questions, feel free to complement/correct it!

A) The data

Without any contextual information, it looks like the only solution you have to link entities is their character similarity. I don't know which distance metric you use, but have you tried these metrics that are well designed for string data?:

Levenshtein distance (also called string edit distance): basically computes the number of operations needed to transform one string into another
Jaro-Winkler distance: close to levenshtein but favoring strings with a common prefix
Jaccard similarity: roughly computes the ratio of common elements between two sets, be it characters, ngrams, words..

You can try these metrics without the n-gram transformation first, but also on transformed data and use cosine distance afterwards.

You can also perform additional normalization but you need to assess whether it is relevant for your dataset, which I don't know, so I am just firing ideas, not advices: stripping accents, special characters, short tokens (like the 'P.' in your example), which will reduce the variance in your data.

B) The clustering

One thing that could explain your result is that with KMeans or Affinity Propagation, any data point must belong to a cluster, so names that should be alone are assigned to the cluster so that it minimizes the loss of the algorithm.

Have you thought of trying DBSCAN? It can label some data points as noise, and if you are using one of the string distances above, you can encode prior knowledge about the maximum distance between two potential matches through the epsilon parameter.

But as with any unsupervised method, you will never have the guarantee to get rid of noise.


Merging sparse and dense data in machine learning to improve the performance

This seems like a job for Principal Component Analysis. In Scikit is PCA implemented well and it helped me many times.

PCA, in a certain way, combines your features. By limiting the number of components, you fetch your model with noise-less data (in the best case). Because your model is as good as your data are.

Consider below a simple example.

from sklearn.pipeline import Pipeline
pipe_rf = Pipeline([('pca', PCA(n_components=80)),
                    ('clf',RandomForestClassifier(n_estimators=100))])
pipe_rf.fit(X_train_s,y_train_s)

pred = pipe_rf.predict(X_test)

Why I picked 80? When I plot cumulative variance, I got this below, which tells me that with ~80 components, I reach almost all the variance.

So I would say give it a try, use it in your models. It should help.


Running sklearn trained classifier in a windows machine without Python

The simplest thing to do is to get Python installed properly on your production system. If you aren't able to do this for whatever reason, you can still run Python without installing it, but it's a bit more work to get everything to work properly, particularly if you don't have Administrator privileges.

If you truly can't install Python, your best bet would probably be to use something like py2exe to build a self-contained version of your application that includes a Python interpreter and required libraries.


Random Forests with Big Data - number of trees v. number of observations

I would recommend using a combination of both options #1 and #2.

You could first try tuning your hyper-parameters to find out till what extent could you reduce the number of trees to a point where the random forest model's prediction starts deteriorating on the test set.

This is because changing the value of mtry, the randomly selected number of features for a new tree, is the only meaningful hyper-parameter that should impact accuracy of the model. Since averaging converges as the no. of trees increases, the no. of trees could be reduced to a point where its performance is not impacted as much. Hence, you need to iterate and choose a a limit beyond which very small number of trees may not produce a strong enough ensemble. A random forest needs works best by using more base learners for reducing the variance by averaging each individual tree's output.

It is not clear from your case whether you're using the Random Forest for a a classification or a regression problem. In case this is a classification problem, and if your data-set is imbalanced in terms of ratio of positive vs. negative classes; then you could reduce the size of the training set by under-sampling the majority class to bring it nearer to a 1:1 ratio. Since you have a large number of records, such class based sampling could improve accuracy as well as reduce data size for training.

Additionally, if you've got a fine tuned Random Forest with good performance, then you could also evaluate dropping features that are least important as determined by the algorithm on OOB samples. This would reduce the time taken to train the model.


R: Pam function under cluster package not accepting data set having more than 65536 entries

If you read the package News section you will find:

  pam() now signals an error when there are more than 65536 observational units (whereas it could segfault previously), thanks to a patch from Mikko Korpela, Helsinki.

So it seems there isn't a solution.


Decision tree not using all features from training dataset

When a feature is not that informative of your target, the algorithm can choose not to use it. This can be for two reasons:

All the information is already in the other features, meaning it adds nothing to include it
There is some regularization going on, meaning your algorithm punishes complexity if it doesn't add enough

It can also be that the regularization is very explicit, meaning there is a maximum depth set as hyperparameter. In this case the other features are more informative and useful for your task.


Interpreting coefficients after transforming Y (dependent) variable

By using BoxCox transformation on the Y variable, you are turning your model from linear regression into non-linear regression. Note that although it seems you are training a linear regression, it’s only linear with respect to the transformed version of Y but non-linear regarding the original Y. Therefore you can not expect a linear behaviour from a non-linear model which means you can not work out the actual sales change resulting in changes in X1 and X2. That’s because unlike linear models, in non-linear models f(x+b) is not equal to f(x) + f(b)


What is the best way to store images in python for machine learning

This might be a bit more complicated.
I normally reuse computer vision and deep learning software to do that. Even if I don't do Deep Learning.
Particularly I use Pytorch, for its bridge with Numpy and pandas.  Here is a tutorial.
This allows me to use a GPU if wanted, and to reuse a lot of code since for deep learning and images there is tons of code snippets out there.


The CNN for MNIST: 5x5x32 Convolutional -> 2x2 MaxPool -> 5x5x64 Convolutional -> 2x2 MaxPool -> Flatten -> 1x512 Full connected -> ReLu -> 512 x 10 Full connected -> Softmax.
The CNN for MNIST has 8 layers with the following structure: 3×3×32 Convolutional → 3×3×64 Convolutional → 2×2 MaxPool → Dropout → Flatten → 1 × 128 Full connected → Dropout → 128 × 10 Fully connected → Softmax.

Converted to both to same formats. They are not the same.


How to train a neural network model in python or any language that can train itself from a excel file and validate itself also from a excel file?

You are mentioning two different problems:

data gathering: here you want to get your data from an excel file for both the training and the validation
modeling your data: here you want to do it with a neural network.

For the data gathering, I suggest you to use pandas, a python library which can extract data from excel files with pandas.read_excel as well as many other source (csv files, parquet files). Then once you have the data in python you can  try to understand them with a neural network. The book I used to learn it was handson machine learning by Aurelien Gueron (https://github.com/ageron/handson-ml). Hope you will find this useful.

Before going to neural network you should also have a look at scikit learn a python library which is used for machine learning when you do not have enough data to train a neural network, if you are getting you data from excel files I guess you have less than 100 thousand data rows which might be a bit too few to train a neural net.


I would suggest you to check this excellent presentation by Li Deng (Microsoft Research). Many of the slides contain references to relevant research papers and even several interesting books on the topics of interest (it should be pretty easy to find). It might be also helpful to check references, listed in this research paper by Prof. Andrew Ng and his colleagues at Baidu Research. Finally, a focused Internet search will provide you with comprehensive list of resources for further research.


Does mini-batch gradient descent nullify the effect of stratification on the training data set?

It doesn't, the workflow when training a model is like that:

Create 10 evenly distributed splits from the dataset using stratified shuffle
train set = 8 splits;
validation set = 1 split;
test set = 1 split
Shuffle the train set and the validation set and create minibatches from them
Train for one epoch using the batches
Repeat from step 3 until all epochs are over
Evaluate the model using the test set

If we skip the stratified shuffling in step 1 the classes of the train set, validation set and test set wont be evenly distributed.
If we skip the shuffling before each epoch in step 3 the mini-batches in each epoch will be the same.
The proportions of the train set, validation set and test set can of course vary.


Padding the sentences is consuming huge memory

Assuming you have tried alternatives like storing it in csr matrices in scipy, you can move away from padding to avoid memory issues by declaring your batch_size=1 and may be create batches using a groupby for equal length sequences. If you are using keras, then please check Masking.


Generically speaking a neural network can automatically learn the needed feature transformation for a model to perform well.  What tends to matter more is how the features relate to your output.

If you're ever bored, you can run your original matrix, a transformed matrix that has been de-correlated via a principle axis transformation, and the original matrix combined with correlation information.  I suspect performance using the same architecture will be relatively similar.


Extract all the data of a particular month from dataset of different years

df.loc[df['Temperature'].isin(['2010-08', '2011-08'])]

isin() will help in setting filter for multiple values.

The other way is to use regex to extract data of particular month from different years:

import re
df['Temperature'].filter(lambda x: re.match('-08$', x))

It will check if the value ends with -08 (in this case, if the month is August) and select only such records.

Hope it helps!


One of the key features in hand-write is the frequency of the signature, i.e. high frequencies - how "shaky" is the signature, and low frequencies - how rounded is the signature etc.
For that, you would like to use Fourier transformation on the image, and train the results on labeled data, if available.


Should I update my regularisation L1 and L2 regularisation parameters in online setting?

As you say @Alexis, cross-validation is the standard way of selecting a model (or its parameters) in offline learning (especially when previous knowledge in not available). For online training, one usually incorporates some sort of (automatic) compensation for the model parameters that adjust them at the same time that training takes place (the alternative being to cross-validate on some gather data or equivalently to pre-optimize your parameters). Check also this answer as well as, for example, this paper (it shows that this issue is a very active research area)!


Here's a solution using the caret package for R. A Random Forest is first trained on the data. All observations for which the probability (from the voting) is less than 99% are then passed to model 2, linear discriminant analysis. Only the probabilities from unseen resampling observations are used, since the Random Forest will otherwise fit the training data perfectly. That is what caret is needed for.

The accuracy is a little higher for the uncertain cases, but this is probably overfitting since I have tried several different models and the data set is small.

I'd like to know if this really improves the performance in your application for the out of sample test data. Are there any papers that recommend this approach? This approach seems to resemble boosting. I tried it on some of my data but could not improve the out of sample performance I got from model 1 (a Random Forest).

data(iris)
library(caret)
myTrainControl <- trainControl(method = "cv", number = 10,
                               savePredictions = T,
                               classProbs = TRUE)
set.seed(4213) # To get the same resamples every time
M1 <- train(y = iris$Species,
            x = iris[, !(names(iris) == "Species")],
            method = "rf",
            trControl = myTrainControl)

M1pred <- M1$pred[M1$pred$mtry == 2, ]
confusionMatrix(M1pred$pred, M1pred$obs)
# Accuracy 96%

# Inspect the predicted class probabilities:
probs <- cbind(M1pred$setosa,
                   M1pred$versicolor,
               M1pred$virginica)
colnames(probs) <- c("setosa", "versicolor", "virginica")
maxCol <- max.col(probs)
probs <- cbind(probs, maxCol)
maxProbs <- apply(probs, 1, function(x) x[x["maxCol"]])
summary(maxProbs)
# Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
# 0.5120  0.9805  0.9980  0.9645  1.0000  1.0000

# Let's define anything below 99% as uncertain:
uncertain <- which(maxProbs < 0.99)
length(uncertain) # 45
# Find rows in iris data that belong to 'uncertain' cases:
uncertainIndex <- M1pred$rowIndex[uncertain]
    confusionMatrix(M1pred$pred[uncertain],
                reference = M1pred$obs[uncertain])
# M1 Performance in uncertain cases:
#              Reference
# Prediction   setosa versicolor virginica
# setosa          5          0         0
# versicolor      0         17         3
# virginica       0          3        17
#
# Overall Statistics
# Accuracy : 0.8667

# Train new model on uncertain data only:
irisUncertain <- iris[uncertainIndex, ]
set.seed(4213) # To get the same resamples every time
M2 <- train(y = irisUncertain$Species,
            x = irisUncertain[, !(names(irisUncertain) == "Species")],
            method = "lda",
            trControl = myTrainControl)

M2pred <- M2$pred

confusionMatrix(M2pred$pred, reference = M2pred$obs)
#              Reference
# Prediction   setosa versicolor virginica
# setosa          4          0         0
# versicolor      1         16         0
# virginica       0          4        20
#
# Overall Statistics
# Accuracy : 0.8889
# (Small discrepancy: Why does caret report an accuracy of 88,5% for M2?)

# For new data predictions can be made as follows:
# (just as an example from the original data again)
# Some 'uncertain' cases are 'certain' now using the full M2 model
newdat <- irisUncertain[30:34, -5]
M1maxProbs <- apply(predict(M1, newdat, type = "prob"), 1, max)
ifelse(M1maxProbs < 0.99,
       paste("M2:", predict(M2, newdat)),
       paste("M1:", predict(M1, newdat)))
# 85               94              107              126
# "M2: versicolor" "M1: versicolor"  "M2: virginica"  "M1: virginica"
# 71
# "M2: virginica"


How does BERT deal with catastrophic forgetting?

As far as I know, no neural network can be immune to catastrophic forgetting during fine-tuning (which is essentially controlled retraining).

The key is to not fine tune the pretrained model for longer epochs, or with higher learning rates. This ensures that the learned knowledge from the lower layers is preserved more or less intact, while also helping the model learn from the new data used for fine-tuning, as mentioned here : https://github.com/huggingface/transformers/issues/1019


What are the 'hottest' future areas of Machine Learning and Data Science?

There is a brilliant answer by Yann LeCunn recently on a Quora session.
What are some recent and potentially upcoming breakthroughs in deep learning? As he quotes, adversarial training proposed by Ian Goodfellow is one of the hottest area. Apart from that, I think Memory Networks, RNNs are widely used right now and can solve various problems up ahead.

Coming to what companies might do, I see a lot of text analytic companies. Considering the amount of consumer data we have now, there is a great need to be able to understand and analyze it. Computer Vision using ML or Deep learning is fast catching up. Using cameras for face detection, object tracking, fraud detection, tracking number plates is a huge field for companies to handle.

Also, I think this might help you to prepare. To catch up with the overlooked developments, refer this which might give you an insight about what can develop in the future.


How to check if sentiment analysis is required?

TextBlob, a Python package, does this (and much more). It uses a pre-trained model, thus requires no training. Given a sentence, TextBlob will return the polarity from -1 to 1. -1 is negative, 1 is positive, 0 is neutral. It will also return subjectivity from 0 (very objective) to 1 (very subjective).


Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

You can use map reduce algorithms in Hadoop without programming them in Java. It is called streaming and works like Linux piping. If you believe that you can port your functions to read and write to terminal, it should work nicely. Here is example blog post which shows how to use map reduce functions written in Python in Hadoop.


How to identify word in a sentence representing the song genre?

You are describing named-entity recognition (NER) which seeks to locate and classify named entities mentioned in text into pre-defined categories. In this case, the pre-defined categories are music genres.
There are many algorithms for NER. In addition to Recurrent Neural Networks (RNN),  Hidden Markov Models (HMMs) and CRF (Conditional Random Field) are often used.
In most cases, the algorithm is not the most difficult part of NER. The difficult part of NER is creating a large, accurate annotated training corpus.


How does inverse of logistic function produces "linear relationship", (so we can use least-squares)

It's true, the function $1/f_{log}(t)$ is not linear, in other words there is not a linear relationship between $t$ and $1/f_{log}(t)$ but that's not what the author meant; the the important word is "among";

This means that there is a linear relationship among $1/f_{log}(t)$.

So there are relationship among the y values; specifically, there is a linear relationship between $1/f_{log}(t-1)$ and $1/f_{log}(t)$.
So we can change the code to plt.plot(1/y_of_x_minus_1, 1/y), and we will see a straight line:
import math
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd

get_logistic = lambda b1, b2, b3: lambda x: (b3/(1+b2*np.exp(-b1*x)))

logistic_args = (1,1,1)
logistic_function = get_logistic(*logistic_args)

x = np.linspace(-10,10)
x_minus_1 = x-1

y = logistic_function(x)
y_of_x_minus_1 = logistic_function(x_minus_1)

plt.plot(1/y_of_x_minus_1, 1/y)
plt.show()
pd.DataFrame({'x': x, 'x_less_1': x_minus_1, 'y_for_x_less_1': 1/y_of_x_minus_1, 'y': 1/y})

Notice the straight line:

EDIT
The linear relationship among $y$ values helps us estimate the parameters for the logistic function, using a linear least squares approach:

This (linear relationship) can serve as a basis for estimating the parameters $β_1, β_2, β_3$ by an appropriate linear least squares approach, see Exercises 1.2 and 1.3.

And the exercise 1.2 asks:

Then we have the linear regression model
$z_t = a + bz_{t−1} + ε_t$ , where $ε_t$ is the error variable. Compute the least squares estimates $\hat{a}$, $\hat{b}$

And the least-squares formula for the general case of $y=β_0+β_1x+ε$ goes like this (note: beta's mean something different than the three beta's of logistic function, above) (same formulas shown here and here)

$\hat{\beta}_1 = \frac{\sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^{n} (x_i-\bar{x})^2}$
$\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}$

I will try to come back and edit the answer to demonstrate how the linear relationship lets us solve for $\hat{\beta_0}$ and $\hat{\beta_1}$ (aka $\hat{a}$ and $\hat{b}$), which lets us figure out the 3 beta's of the logistic function, as suggested in the exercise 1.2:

Compute the least squares estimates $\hat{a}$, $\hat{b}$ of $a$, $b$ and motivate the estimates $\hat{β_1}:= −log(\hat{b})$, $\hat{β_3} := (1 − exp(−\hat{β}_1))/\hat{a}$ as well as $\hat{β_2} := ...$

For an exercise I tried to change the itl.nist.gov form ....

$\sum_{i=1}^{n} (x_i-\bar{x})(y_i-\bar{y})$

...to the wikipedia form...

$\sum x_i y_i - \frac{1}{n} \sum x_i \sum y_i$

Starting like this...
$\sum (x_i-\bar{x})(y_i-\bar{y})$
...then, multiply the binomials...
$\sum (x_iy_i - x_i\bar{y} - y_i\bar{x} + \bar{x}\bar{y})$
...then, distributing the sum...
$\sum x_iy_i - \sum x_i\bar{y} - \sum y_i\bar{x} + \sum \bar{x}\bar{y}$
...and because it sums over all $n$ values ($\sum_{i=1}^{n}$), the same $n$ values used for the means/expected values of $x$ and $y$ ($\hat{x}$, and $\hat{y}$) ...
$\sum x_iy_i - \sum x_i\bar{y} - \sum y_i\bar{x} + n \bar{x}\bar{y}$
... then, we transform $n\bar{x}$ into $n \frac{\sum{x_i}}{n}$ then into $\sum x_i$ (we could instead transform the $n\bar{y}$; would reach the same end result)...
$\sum x_iy_i - \sum x_i\bar{y} - \sum y_i\bar{x} + \sum x_i\bar{y}$
... then two terms will cancel....
$\sum x_iy_i  - \sum y_i\bar{x} $
... then, the wikipedia form (which is almost the covariance, aka the $
 \text{Cov}[x,y]$ (just imagine another $\frac{1}{n}$ distributed among the terms)....
$\sum x_iy_i  - \frac{1}{n}\sum y_i\sum x_i $


Given the size of your data set, the best approach to cross validation is the
Leave-One-Out method. You haven't discussed the language or package you used for your model, but generally speaking you set the $k$ equal to the number of records.  In your case, that 39.  This will cause your model to train on 38 instances and predict the 39th, with each instance eventually will receive a classification.


Decision Trees Should We Discard Low Importance Features?

As for many questions, the answer is "it depends":

features which have a low individual importance may still add predictive power to your model, because the model benefits from combining their information together with information of other features.
However they may introduce noise in the model and cause overfitting, thus decreasing the performance of the model.

The best way to answer your question is to experiment:

order the features by decreasing importance
loop from say 5 features to 30, each time selecting the top N features by importance, and training/testing a model based on this subset of features.
plot the performance

You're likely to observe that the performance increases quite a lot at the start for each "important" feature added, then slows down as feature importance decreases and probably doesn't increase at all at some point, possibly even decreasing a bit.


Decided to go away and find the answers that would satisfy my question, and write them up here for anyone else wondering.
The .best_estimator_ attribute is an instance of the specified model type, which has the 'best' combination of given parameters from the param_grid. Whether or not this instance is useful depends on whether the refit parameter is set to True (it is by default). For example:
clf = GridSearchCV(estimator=RandomForestClassifier(),
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
optimised_random_forest = clf.best_estimator_
return optimised_random_forest

Will return a RandomForestClassifier. This is all pretty clear from the [documentation][1]. What isn't clear from the documentation is why most examples don't specifically use the .best_estimator_ and instead do this:
clf = GridSearchCV(estimator=RandomForestClassifier(),
                    param_grid=parameter_candidates,
                    cv=5,
                    refit=True,
                    error_score=0,
                    n_jobs=-1)

clf.fit(training_set, training_classifications)
return clf

This second approach returns a GridSearchCV instance, with all the bells and whistles of the GridSearchCV such as .best_estimator_, .best_params, etc, which itself can be used like a trained classifier because:
Optimised Random Forest Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]
GridSearchCV Accuracy:  0.916970802919708
[[139  47]
 [ 44 866]]

It just uses the same best estimator instance when making predictions. So in practise there's no difference between these two unless you specifically only want the estimator instance itself. As a side note, my differences in metrics were unrelated and down to a buggy class weighting function.
[1]: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV.fit


Contemporary alternatives to SIFT for image feature extraction?

The Capsule Networks by Geoff Hinton et al. are based on a similar idea to SIFT: in fact, if you look at the 2011 paper, Transforming Auto-Encoders by Hinton et al. they explicitly cite SIFT as an inspiration for Capsules.

The main idea is that they construct a network with a new type of unit called a Capsule which outputs a vector (rather than neurons outputting scalars, as we know in traditional neural networks).

You can think of these vectors as an analogue to the SIFT keypoints and feature descriptors, but with the benefit that they can be learned via back-propagation.

There is a recent article, Dynamic Routing Between Capsules by Sabour, Frosst and Hinton elaborating on this which shows very promising results.

I have written a short summary of Capsule Networks on the AI Stack Exchange, see the question What's the main concept behind Capsule Networks?.


distribution difference between image and text

Suppose you trained two identical neural nets on different datasets.  Network A is trained using a dataset of cat pictures.  Network B is trained using a dataset of traffic sign images.  Because the two networks are identical, they will obviously produce a feature map in the same space, right?  But the distribution of features in that space will be different for the two networks, because they were trained on different datasets, and you need different feature extractors to recognize cats vs. traffic signs.
This is analogous to what you have read about the text/image features.  Suppose we train a network to embed image data in some N-dimensional space, and then we train another network to embed text data in the same N-dimensional space.  Although the resulting feature vectors are in the same space, they are almost certain to have different distributions, because they were trained using different datasets.
Unfortunately, we cannot give a general answer about how the distributions are shaped and what exactly the differences are.  These details will vary from case to case.  Although we may not know exactly how the distributions are different until we have them in hand, we can be confident that they are in fact going to have substantial differences.


Find Net Reclassification Improvement/Index metric using Python

Just stick to F1.

Yes NRI is cools but there are edgecases and its not a magic bullet. Read this paper

IF you really want to implement it you can embedd r programs in python.

Since there is already implementation in r what you could do is something like this. Just change it to use NRI.


Remove columns with a certain number of consecutive zeros

Short answer:

# Max number of zeros in a row
threshold = 12

# 1. transform the column to boolean is_zero
# 2. calculate the cumulative sum to get the number of cumulative 0
# 3. Check how much of each count you get and remove 0 counts
# 4. Get the maximum number of cumulative zeros
# 6. Drop the columns with more than your threshold

to_remove = []
for col in df.columns:
    count_zeros = (df[col] == 0).cumsum().value_counts().drop(0)
    if len(count_zeros) and count_zeros.idxmax() > threshold:
        to_remove.append(col)

df = df.drop(to_remove, axis=1)

You're welcome


Should I set higher dropout prob if there are plenty of data?

You are right, increasing the dropout proportion will help.

However, this looks like a setting where early stopping will be a very good choice. Although dropout, weight decay and batch-norm can work propperly, the fact that you easily overfit your training set would make it an appropiate scenario to try early stopping.

In addition, as the neural network takes very short to be trained you can train many of them (on some subset of the training set, making them weak learners) and create an ensemble to make the final predictions.


A naive definition of Parameter convergence is when the weights or the values of the parameters reach a point asymptotically. What I mean is that, when your model training is not altering the parameter values(maybe less than epsilon-small values) it might be a good fit. For decision trees, I found this paper which explains rate of convergence and more. It might be a good read if you want to get more details.


Can't understand this simple matrix multiplication in python


Can continuous variables decrease classification model accuracy?

There's no concrete property of classifiers in general that lend themselves to using categorical variables rather than continuous variables.

You can definitely try binning the variables, but many classifiers (particularly, tree-based classifiers) will implicitly bin the variables optimally within the algorithm itself.

If you add the continuous variable, and the performance of your classifier doesn't improve, it's very possible that the variable is not predictive of the target or that all of the information (in the information-theory sense) that that variable carries is already expressed by other variables already in the model.


What I would suggest is to build a sklearn pipeline in which one step will be the sklearn PCA and the last step will be your Keras model.
Sklearn pipelines are easy to put into production and can handle a lot more of transformations.


Scaling of ordinal data before both hierarchical and KMeans clustering

If all applicable variables are ordinal and they're all in the same scale ie: 1-5, scaling would be totally unnecessary.

The whole point of scaling is to bring attributes with values in differing magnitudes to the same scale. Bottomline, here you don't need to scale the data.

Hope this helps!


What does IBA mean in imblearn classification report?

If you look at the imblearn documentation for classification_report_imbalanced, you can see that iba stands for "index balanced accuracy". For more information on what the index balanced accuracy is and it's value in cases on imbalanced datasets, have a look at the original paper.


Using Machine Learning to Predict Temperature

I would not recommend going ahead with the data that you know might be wrong.

Looking at your current data, the reason you got a bad result with linear regression is because the relation between them is not linear for the current data. For eg. There is high variation in response (i.e. user_set_temp)  for same value of your predictor (i.e. environment_temp).

First, get hold of correct temperature for your recorded timestamps from local weather data to replace environment temp with this data, till you get the issue rectified with your original recorded environment_temp. When you rectify the issue, then I would recommend you to use both weather and environment temp to predict, as a person might set the temperature depending upon a combination of both.

After you get a good representative data, this should be a reasonable procedure to help you predict the temperature:

Exploratory data analysis for uni-variate timeseries:
This will help you observe patterns in data, which will help you decide on which new features to engineer/ create from timestamps (step 2) you have of recordings. Also, look at the acf/ pacf plots to help figure out the lags in data that might help you predict better. Fit a basic uni-variate model like STL or ARIMA model to get a good base model for prediction (this model you will compare new models with to see how they perform)
Feature Engineering: Time of day, day of week, week of month etc identified in step 1. Also, look for any other features that might help your algorithm find patterns.
Regression Models: Test with advanced algorithms like RandomForest, Gradient Boosting and RNN-LSTM to check how they perform against each other and the base model.


What can be done with highly correlated variables (>.95 and <-.95)

An alternative to the one provided by @Kasra is dimensionality reduction. It's another way of solving your multicollinearity problems, while avoiding deleting variables more or less arbitrarily.

You can use simpler, linear techniques such as PCA, or more complex non-linear techniques such as Autoencoders. t-SNE is a non-linear technique that is typically used for visualization, I do not recommend to use it for a Training set.


Xgboost quantile regression via custom objective

Perhaps the blog below provides an answer to your question.

https://www.bigdatarepublic.nl/regression-prediction-intervals-with-xgboost/

Without go through code in much detail, probably, your problem can be described as followed (from the blog):

  In the case that the quantile value q  is relatively far apart from the observed values within the partition, then because of the Gradient and Hessian both being constant for large difference x_i-q, the score stays zero and no split occurs.

Then the following solution is suggested:

  An interesting solution is to force a split by adding randomization to the Gradient. When the differences between the observations x_i and the old quantile estimates q within partition are large, this randomization will force a random split of this volume.


I am not an expert on Neural Networks, but if you are using Random Initialization then your MSE would vary slightly from each rerun.


how to find probability of one or more events to happen from an incomplete data set

Bayes Theorem applies to conditional probabilities.
$
     P(A|B) = \frac{P(A).P(B|A)}{P(B)}
    $

The question you have posed is one of multiple events occurring together. If the events can be considered to be independent of each other, then,

P(A & B & C) = P(A).P(B).P(C)

In your question above,

P(M & A & L) = P(M & A).P(L) = P(M).P(A).P(L)

Based on the data provided by you, you should have these probabilities or can derive them by appropriate summations.

Let's say that .1 probability of males are between 1-10 years of age. Further, probability of males in location L is 0.2. So the probability of males in 1-10 years of age in Location L is 0.1 X 0.2 = 0.02 . This is assuming that all locations have the same probability distribution of age ranges.


Regression will work well if your data set is large, but only for predicting current house prices (say, for example, estimating the value of your house). That's what people generally mean when they talk about predicting house prices from current house sales data.

The question of how house prices will behave in the next year is much, but much more complicated, and would not depend simply on the data you currently have. You would need to involve other information and a much more complex model, which would need to involve things like current level of household debt, inflation rate, economic outlook, etc. Daunting.

Generally, speculative prices follow some stochastic process. They depend on the current value, but diverge more and more the farther you go in the future.


GAN to generate a custom image does not work

Generating text as an image is extremely difficult and I have never seen a GAN applied in the image space to generate pages of text. The reason this is so hard is because of the way in which text is perceived by humans and the way a GAN works.

Humans read arbitrary symbols which are sequenced from left to right along the same line and combined into rows. Moreover, these symbols are combined into groups which represent words. This is extremely complex. The symbols must be intelligible, the words must be real ones as invented by humans. Lastly, the combination of words into sentences need to be logical and follow guidelines of human language. And even FURTHER the sequence of sentences must be coherent to transmit a message.

A GAN operating in image space will try to learn the distribution of the training set in a pixel-wise manner as that is your inputs. The distribution of the pixels will not effectively be able to group characters together in a logical manner, and the words will not be real, and the sentences will all be nonsense. You will most likely end up with blurry lines of random looking symbols, kind of like a zebra print.

Another problem is the amount of data you have. Even if this problem was possible with a GAN you would need tens of thousands of instances to train a GAN effectively.

What I suggest

I suggest to read the texts that you are training with and use this data to train a LSTM which has been shown to be very effective for generating language. But, be warned even with the best LSTMs you rarely get text that can fool humans into thinking its real.

The LSTM will provide you with your text.

Then you can train a GAN to generate the characters of the alphabet and you can use this generator to print out the text that the LSTM generates.


I would recommend taking a look at the transformation you did:

Consider whether you would want to add a really small number like 0.0000001 to zero before taking the log. Probably not, as you will get a large negative number. However, there should be little to no difference in results once you start adding large enough numbers like 0.1. I would recommend trying each and looking at a density plot of the resulting data. Investigate which distribution looks the least skewed and go with it.


Predicting Multiple Values Values Using Time Series Forecasting

Thanks for this question I think it is a nice use case to play with time series forecasting in all (or many) of its types.
As you suggest, there are several possible approaches, and all of them are valid hypothesis a priori to check and validate with your goal. Answering the question:

yes, it is possible to build a single model to predict the sales amount of your products at once (aka a multi-output forecasting model) where each of your products would be a feature to predict --> https://www.tensorflow.org/tutorials/structured_data/time_series#multi-output_models, and you can whether predict a single time-step or a multi time-step forecasting

I would say building a model for each product makes sense, as you will potentially have many more customers than products throughout of time; this way, for each product you will have several customers who bought it, and modelling the clients characteristics for a certain type of product sounds reasonable; in this case, you have a single-feature to predict (in the same source of info you have this model type, agsain for both single time-step or multi time-step forecasting: https://www.tensorflow.org/tutorials/structured_data/time_series#single_step_models


Training on derived features that won't be present in a test set

Thats not how it works, there is not juice to be extracted if the data is missing from the test set.

You will have features in train that might be discriminative, but missing in test. Then when you try to predict there wont be any inputs that you can map to the outputs.

This beeing said I should mention that NaiveBayesclassifier will not use missing features, so there would be no harm in including them but you wont get any additional information, just more unnecessary complexity.


Adding and subtract inbetween row inputs and value equal to the first column next row using pandas

So are you evaluating the equality of a+b-c and d for each sample? Meaning, you aim to save the boolean value? If so:

import pandas as pd
#a sample dictionary
data = {'x1':[1,0,4,5,8,1],
     'x2':[3,4,5,6,8,9],
     'x3':[4,5,1,-2,4,5]}
df = pd.DataFrame(data) #converting the dictionary to dataframe
df['y'] = 0 #initializing the y column with 0, also you can initialize with FALSE boolean value
for i in range(len(df)-1): #iterating between all the rows of dataframe
    df['y'].iloc[i] = df['x1'].iloc[i] + df['x2'].iloc[i] - df['x3'].iloc[i] == df['x1'].iloc[i+1]

Note: for the last row, since the content of column y should be calculated based on the next row, the value cannot be calculated, that is why we have set (len(df)-1).

Result:

   x1  x2  x3      y
0   1   3   4   True
1   0   4   5  False
2   4   5   1  False
3   5   6  -2  False
4   8   8   4  False
5   1   9   5      0

The problem with your code:
First, you are not considering that subsequent row to calculate the accuracy of the expression $a+b-c ==d$
Second, if that print(j) is outside the loop, is it? then you are looping in the rows of data frame while j is less than data frame's length, So when the flow is out of this while, a single value j would be returned.


How to perform Cleaning of a very large set of addresses

I guess you are in a situation that different fields of an address are mixed together. Try break the address text into Shingles and then try Locality-sensitive Hashing

Prof. Jeff Ullman's Text Book may help you put together all the techniques you need, starts with Section 3.2


You need to compute the Euclidean distance between each point and sort theses distances.

That said, you can find an implementation in the imbalanced-learn toolbox. More precisely, you can see the different steps of the implementation that you mentioned: (i) fit a KNN, (ii) find the NN of each sample, (iii) generate a new samples.


Data Cleaning Techniques - What to drop?

Check the percantege of missing values in each column. If it is high then you should remove this column, otherwise you are able to try to impute the missing values. For the latter you may replace them with the mean (median or mode etc) or try to predict them using the other known values. You should check if these are values are mising at random or not.


The problem is that you are using the models in training mode (the default mode), and the stochastic elements like dropout are active in training mode, therefore you obtain different results, not only between different models, but also on different runs of the same model.
You should invoke model.eval(), which makes some elements like dropout or batch normalization behave in inference mode, i.e. dropout is disabled altogether, batch-normalization uses the remembered statistics.
Also, it is better to avoid gradients being computed with torch.no_grad(), as you do not need them. The same effect can be achieved with torch.set_grad_enabled(False).

import torch, transformers

torch.set_grad_enabled(False) # avoid wasting computation with gradients

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
torch_model = torch.hub.load('huggingface/pytorch-transformers', 'model', 'bert-base-uncased')

torch_model.eval() # <------ set model in inference mode

inputs = tokenizer.encode_plus("Hello", "World", return_tensors='pt')
output_embedding = torch_model.embeddings(inputs['input_ids'], inputs['token_type_ids'])
output_self_attention = torch_model.encoder.layer[0].attention.self(output_embedding)[0]

# compare output with using the huggingface model directly
bert_self_attn = transformers.models.bert.modeling_bert.BertSelfAttention(torch_model.config)
bert_self_attn.load_state_dict(torch_model.encoder.layer[0].attention.self.state_dict())

bert_self_attn.eval()  # <------ set model in inference mode

output_self_attention2 = bert_self_attn(output_embedding)[0]
output_self_attention != output_self_attention2 # tensors are equal


Why use regularization instead of decreasing the model

Regularization does decrease the capacity of the model in some sense, but as you already guessed, different capacity reductions result in models of different quality and are not interchangeable.

L1 can be interpreted as making the assumption that the influence of different factors (represented by neurons) on each other shouldn’t be assumed without significant support by data (i.e. the gain achieved by larger influence has to outweight the L1 loss associated with increased absolute value of the parameter that „connects“ them).

L2 does the same, but makes this dependent on the connection strength, i.e. very light connections basically need no support (and are therefore not driven further to exact zero) and very large connections are almost impossible.

Dropout can be interpreted as training a large amount of smaller networks and using the approximated average network for inference: „So training a neural network with dropout can be seen as training a collection of 2^n thinned networks with extensive weight sharing, where each thinned network gets trained very rarely, if at all.“ Dropout: A Simple Way to Prevent Neural Networks from Overfitting

All these methods make certain network parameter combinations highly improbable or even impossible to achieve for a given dataset, which otherwise could have been the result of the training. In this sense, the capacity of the model is reduced. But as one can imagine, some capacity reductions are more useful than others.


Regarding Training loss and validation loss

To the learning-curves look exactly like what you would expect. The training-loss goes down to zero. That means your model is sufficient to fit the data. If the training-loss would get stuck somewhere, that would mean the model is not able to fit the data. So, your model is flexible enough. Furthermore the validation-loss goes down first until it reaches a minimum and than starts to rise again. If your validation set is large enough and representative you can argue that your model starts to overfit the data.

Next step could be to prevent the model to overfit. You can do that by adding some kind of regularization, dropout, batch-normalization.

Considering the test set. Instead of using two distinct values try to decide for one metric that does what you want. For example the F1 score. It is hard to say whether these values are good or not. It ultimately depends on what you define as "good enough". If 90% is sufficient for you than that is ok,  but if you need a model that is 99.9% accurate it would not be good enough. Apply the metric to all sets training, validation and test. The validation error should be a good estimator of the test error unless you use it to fit your data. If you use it to identify the optimal number of training rounds it must be considered part of the training set and you would expect the error to be lower than in the test set. Again, if the test set error is bigger, usually something is off.

If your validation and test errors are very different from each other that probably means they sample different regions in the space of your data or the sets are not big enough so that stochastic effects play a role.


Is it necessary to take log transformation on the data values to get the minimum mean squared error?

If you applied the log transformation to all your labels as well as the inputs, then it is normal that the MSE decreases. It is the case because when you apply log to big numbers they all become smaller and less spread. The standard deviation of the log price is a lot smaller than the one of the price. The MSE of any model trying to predict the log price (for instance the model that predicts the mean log price every time) will be smaller than the MSE of a model trying to predict the price.
Using the log square footage to predict the price could be useful if you are using a linear regression for instance and if the relation between square footage and price were logarithmic. It is probably not the case but you can try it.
However changing the target (applying log to it) changes the loss (you are computing the MSE between log price and prediction) and thus changes the problem you are trying to solve. You cannot compare the performances of the two models directly using the outputed MSE.


Does it make sense to use TF-IDF to extract most important tokens from a corpus?

Provided limited information & context you have provided, I would suggest you to look for feature selection when each dimension belongs to a word. Feature selection will give you most important words. Most important words in the sense, words deciding the decision surface of the model.


First off, if your data has as many variations (in function of time, context, and others) as to make it hard to apply a single strategy to cope with it, you may be interested in doing a prior temporal/contextual/... characterization of the dataset. Characterizing data, i.e., extracting information about how the volume or specifics of the content varies according to some criteria, usually provides with a better understanding (more consise and precise) than simply inferring algorithms on a brute-force fashion.

So, answering each question:

characterization is definitely a means of reducing the number of iterations while trying to select proper algorithms for specific data;
if you have a discrete set of criterias on which your data varies, it becomes much easier to scale up solutions, as will know what information you'd gain/lose if simpler/specific solutions were applied;
after a characterization, you should be also easier to select parameters, since you'd know what kind of specific data you'd be dealing with;
finally, you may use data mining/machine learning algorithms to support this characterization. This includes using:

clustering algorithms, to reduce the dimensionality of data;
classification algorithms, to help deciding on specific properties the data in function of time/context/... may present;
association rules, to predict particular knowledge from the dataset, while also improving/fine-graining the data used for later analysis;
and other possible strategies and analyses.

And here is a list of some criterias on which to analyse data, which you may find helpful.


Which of the NIPS 2014 papers are most significant, and why?

The awards went to "Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)" by Anshumali Shrivastava and Ping Li and "A* Sampling" by Chris J. Maddison, Daniel Tarlow, and Tom Minka.
Further Quora discussion can be found here.


[Not specific for Raman spectroscopy]

There is a trend in some areas of machine learning to aim at end-to-end learning, that is, devise machine learning algorithms that take raw input data and gives the desired output (e.g. classification label, regression values). This way, end-to-end learning avoids introducing expert knowledge or preprocessing, usually relying only on huge amounts of data. This has been specially true in the last years in the field of deep learning, where images are taken with no preprocessing, audio is not filter-banked, and games are played by receiving only raw pixels.

So it is certainly possible to skip data preprocessing completely.

We should nevertheless take into account that most of the successful end-to-end machine learning approaches impose constraints that enable the learning to exploit the very aspects of the input data that were previously defined by experts. An example for this are Convolutional Networks, that impose a feature locality constraint that enables the learning to identify features, which could have been pre-defined as gabor filters instead.

BUT, in more traditional data science, feature engineering is a key part of the process of devising ML systems and data exploration and expert knowledge are normally used to identify the kind of preprocessing to be applied to data.


What is the difference between (objective / error / criterion / cost / loss) function in the context of neural networks?

The error function is the function representing the difference between the values computed by your model and the real values. In the optimization field often they speak about two phases: a training phase in which the model is set, and a test phase in which the model tests its behaviour against the real values of output. In the training phase the error is necessary to improve the model, while in the test phase the error is useful to check if the model works properly.

The objective function is the function you want to maximize or minimize. When they call it "cost function" (again, it's the objective function) it's because they want to only minimize it. I see the cost function and the objective function as the same thing seen from slightly different perspectives.

The "criterion" is usually the rule for stopping the algorithm you're using. Suppose you want that your model find the minimum of an objective function, in real experiences it is often hard to find the exact minimum and the algorithm could continuing to work for a very long time. In that case you could accept to stop it "near" to the optimum with a particular stopping criterion.

I hope I gave you a correct idea of these topics.


In your code the Area Under the Curve (AUC) is used to calculate the area under the Cumulative Distribution Function (CDF). Let's go through the code to see how this is done. However, in the last section, I do not agree with the two lines used to calculate the area. If all the labels have a ground truth label of 1, and we have a perfect classifier where the probability of being Class 1 is always 100%, we should have an AUC of 1. However, we will get 50%. I will show this later. You should note that the way this code is written will only work for a binary classifier, with labels 0 and 1.

We will first calculate the probability of each instance being in class 0 and class 1 by

probs = forest.predict_proba(X_train)

Then we will extract the second column, this contains the probability of each instance being in Class 1.

probas = probs[:,list(forest.classes_).index(1)]

Then we will join the probabilities of each instance with its ground truth label. Then we will sort the groupings based on their probability. The instances with a probability of 1 will be at the start of the list and those with a probability of 0 will be at the end of the list. This is done using

joint = zip(probas, truth)
joint = sorted(list(joint), key=lambda x:x[0], reverse=True)

Then we will separate the probabilities and the ground truth labels into separate lists using

probas = [x[0] for x in joint]
truth = [x[1] for x in joint]

The blue line is the probabilities and the ground truth labels is the orange line.

This is where the problems start. The next two lines do not include the probabilities but only the ground truth labels.

truth_cumulative = np.cumsum(truth)
truth_cumulative = truth_cumulative / np.sum(truth)

This means that if we had all class 1, with probability 1. We would get an AUC of 0.5. This does not make any sense.

For example, if I force it by doing this

probas[100:100000] = 1
truth[100:100000] = 1

There will only be a few 0 values left. Then after the ordering and cumulative sum we get the plot as follow, which is obviously wrong.

We need to do exactly what is written in the comment but not done through code. We will get the false positive rates for each false negative rate. Much like what is described below in the Receiver Operator Curve (ROC).

This can be done with

# Training Set
probas_train = forest.predict_proba(X_train)
fpr_t, tpr_t, thresholds_t = roc_curve(y_train, probas_train[:, 1])

# Testing Set
probas_ = forest.predict_proba(X_test)
fpr, tpr, thresholds = roc_curve(y_test, probas_[:, 1])
roc_auc = auc(fpr, tpr)

train, = plt.plot(fpr_t, tpr_t, label = 'Train')
test, = plt.plot(fpr, tpr, label = 'Test')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

print('Area Under the Curve (AUC): ', roc_auc)

  Training Set  Area Under the Curve (AUC):  0.998140359403  Testing Set
  Area Under the Curve (AUC):  0.937729144675

Receiver Operator Curve (ROC)

This was invented during WWII as a means to detect aircraft using radar. For example, if we have a sensor which requires some kind of threshold to detect planes. We can determine the true positive rate (TPR) and false positive rate (FPR) which results in our experiment. We will then plot these as the x-axis as the false positive rate, and the y-axis as the true positive rate.

$TPR = \frac{\text{True Positives}}{\text{Positives}}$

$FPR = \frac{\text{False Positives}}{\text{Negatives}}$

If the sensor is perfect we will always have $0$ false positives and always a $100\%$ true positives. This results in a curve which looks like the blue curve. If the sensor was completely random and garbage, then you would have a random guess which is the diagonal line.

A good classifier will be as close to the blue line as possible. Note, that a line below the diagonal is also good, you should just invert your decision. You have a consistent classifier but you are inverting the decision.

Intuitively, consider a sensor which reads a value and it is above a threshold $\theta$ we call it a detection. If we lower the threshold of detection, we should expect more detection, however if we end up with an equal amount of true positives and false negatives, then we are going along the diagonal line. This is a bad classifier.

Here we have 2 sensors, red and yellow, we want to decide which sensor is better. Just looking at the curves is not very effective because it's hard to see which one is definitely higher. So we can devise a metric named the Area Under the Curve (AUC). This is exactly what it means in the name. Then we an say the curve with the higher AUC is the better one.


How to "count" certain events in a time series

Try generating a dictionary of patterns you want to identify.  You can then use convolutions/ cross-correlations to identify where these patterns appear in your data.
https://en.wikipedia.org/wiki/Convolution
https://en.wikipedia.org/wiki/Cross-correlation
http://paulbourke.net/miscellaneous/correlate/

This method is also called 'matched filter'.


How can the process of hypertuning of XGBoost parameters be automated?

In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR.

I've used the caret package extensively. Here is a reference of the parameters supported by caret for tuning a xgboost model.

To automatically select parameters using caret, do the following:

First define a range of values of each parameter you would want caret to search. Define this in the tuning grid.
Start model training using caret after specifying a measure to optimize, e.g. accuracy or Kappa statistic, etc.
Plot or print the performance comparison for various parameter values, refine and repeat if required.

Refer to the caret guide here to get step-by-step instructions on using it.

For handling class imbalance, I've found from my experience that adjusting weights is not as helpful as under-sampling majority class and over-sampling the minority class, or a combination of the two. However, it all depends on the size of data available and the case at hand.

In case you need to tune some parameters which are not supported by caret, then, you could write your own iterative loop to train and test the model for different values of that parameter and then choose one that works best. I think most of the really relevant parameters have already been included in caret.

You would need to adjust these parameters in case the population itself changes over time. Or, the methods to gather data and their accuracy may change which could result in performance deterioration. You could run a simple check by comparing the performance of your model over the current dataset vs. a 6 month older dataset. If the performance is similar, then you may not need to update the model in the future.


Gaussian Mixture Models EM algorithm use average log likelihood to test convergence

It makes unit testing easier; invariant to the size of the sample.

Reference: the github discussion that led to the change.


How to scrape a website with a searchbar

I would suggest reading about http query methods, specifically about
GET and POST. You can pass parameters with query and open directly
company page.

For example:

http://google.com/search?q=GET+and+POST

where (q=GET+and+POST) is a parameter.
Once you have page you can parse it with your favorite library.
(for example beautifulsoup)

EXAMPLE:

Getting number of results from couple of google queries with python 3 and beautifulsoup:

from bs4 import BeautifulSoup
import urllib.request

# List with google queries I want to make
desired_google_queries = ['Word' , 'lifdsst', 'yvou', 'should', 'load', 'from']

for query in desired_google_queries:
    # Constracting http query
    url = 'http://google.com/search?q=' + query
    # For avoid 403-error using User-Agent
    req = urllib.request.Request(url, headers={'User-Agent' : "Magic Browser"})
    response = urllib.request.urlopen( req )
    html = response.read()
    # Parsing response
    soup = BeautifulSoup(html, 'html.parser')
    # Extracting number of results
    resultStats = soup.find(id="resultStats").string
    print(resultStats)


Computation of kernel matrix using radial basis kernel in svm

The kernel matrix $K$ or Gram Matrix is a positive-semidefinite symmetric matrix that has the form:
$$ K(x_1,x_2,\dots,x_n) =
\begin{vmatrix}
\langle x_1,x_1 \rangle & \langle x_1,x_2 \rangle & \dots & \langle x_1,x_n \rangle \\
\langle x_2,x_1 \rangle & \langle x_2,x_2 \rangle & \dots & \langle x_2,x_n \rangle \\
\vdots & \vdots & \ddots &\vdots \\
\langle x_n,x_1 \rangle & \langle x_n,x_2 \rangle & \dots & \langle x_n,x_n \rangle
\end{vmatrix}
$$

Where $\langle a,b \rangle$ detones the inner product/dot product of vectors $a$ and $b$ and $\{x_1,x_2,\dots,x_n\}$ is a set of vectors (in this case, training samples).

In order to calculate the Gramian Matrix you will have to calculate the Inner Product using the Kernel Function. For a RBF kernel function $\kappa_{RBF}$ this can be done by

$$K_{ij} = \kappa_{RBF}(x_i,x_j) = e^{\gamma D_{ist}(x_i,x_j)^2} $$

where $\gamma$ is a function hyperparameter, $K_{ij}$ is the element in row $i$ and column $j$ of the matrix $K$ and $ D_{ist}(x_i,x_j)$ is some distance between two vector measured in some vector space.

Usually, the distance measure used is the $L_2$ norm or euclidean distance. Remember that when computing the Kernel function you will use the square $L_2$ norm, so there is no need to take the square root.

There is a open library named pykernels that has implementations to dozens of kernel functions. This here is the implementation for the RBF (or Gaussian) Kernel:

class RBF(Kernel):
    """
    Radial Basis Function kernel, defined as unnormalized Gaussian PDF
        K(x, y) = e^(-g||x - y||^2)
    where:
        g = gamma
    """

    def __init__(self, gamma=None):
        self._gamma = gamma

    def _compute(self, data_1, data_2):
        if self._gamma is None:
            # libSVM heuristics
            self._gamma = 1./data_1.shape[1]

        dists_sq = euclidean_dist_matrix(data_1, data_2)
        return np.exp(-self._gamma * dists_sq)

    def dim(self):
        return np.inf

it uses the square distance define as:

def euclidean_dist_matrix(data_1, data_2):
    """
    Returns matrix of pairwise, squared Euclidean distances
    """
    norms_1 = (data_1 ** 2).sum(axis=1)
    norms_2 = (data_2 ** 2).sum(axis=1)
    return np.abs(norms_1.reshape(-1, 1) + norms_2 - 2 * np.dot(data_1, data_2.T))

The class is defined as a subclass of the Kernel class and has some methods to deal with kernel combinations.

Note that this is not the fastest way to implement euclidean distance, you can do better but this is a simple form to do it. also that "np.abs" should not be needed

Note 2 that this SO Question has a lot of optimized version of RBF Kernel


watch is linux command and not related to nvidia-smi command,which using that the nvidia-smi command will be updated every 1 second and it give you a view to see how gpu parameters changing like: RAM usage, temperature, P0(means performance,which p0 is best performance and performance decrease when you get p1,p2,...)

about "can it be used for improving performance of the gpu?" the performance of gpu is showing by p0 so your gpu currently is in max performance.

note that minimum interval for watch command is 0.1 second so if you use watch -n 0.1 nvidia-smi then every 0.1 second the output will be update.

for further information type man watch or man nvidia-smi in terminal.


How to compensate for class imbalance in prediction model?

There is a lot of information and techniques for rare event or imbalanced classes. Sorry to post links but (https://stats.stackexchange.com/search?q=rare+event and https://stats.stackexchange.com/search?q=imbalanced) but I do not want to duplicate all of the other work.

In my experiences, often I left the data set alone. I usually got good results. If that is the true ratio, then my model should know that. Sometimes I downsampled the majority class. I played with SMOTE (https://arxiv.org/pdf/1106.1813.pdf) before as well.

For your questions above, yes - changing the data with weights or over/unsampling may bias the results.  Need to check. If you downsample, make sure you are not tossing out signal from the majority class.


Variable input/output length for Transformer

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target sequence length).

The Query is what determines the output sequence length, therefore we obtain a sequence of the correct length (i.e. target sequence length).

In order to understand how the attention block works maybe this analogy helps: think of the attention block as a Python dictionary, e.g.

keys =   ['a', 'b', 'c']
values = [2, 7, 1]
attention = {keys[0]: values[0], keys[1]: values[1], keys[2]: values[2]}
queries = ['c', 'a']
result = [attention[queries[0]], attention[queries[1]]]

In the code above, result should have value [1, 2].

The attention from the transformer works in a similar way, but instead of having hard matches, it has soft maches: it gives you a combination of the values weighting them according to how similar their associated key is to the query.

While the number of values and keys has to match, the number of queries is independent.


Using classifier's probabilities as independent variables to predict Y

This is called "stacked ensembling," or just "stacking."

Wikipedia

I'm not aware of any theoretical mathematical support, but intuitively it may smooth out the results of the $r$ initial models, possibly (depending on the final model) promoting models on data segments where they perform best.  Stacking consistently wins Kaggle competitions, though the added complexity might make it less attractive for certain applications.


I'm not sure I get your question, but not enough reputation to comment. Thus, I'll make some assumptions and answer to the best of my understanding with them.

For example with SQLite (this is one of the simplest, variations possible in different dialects), let's assume your data is created with:

CREATE TABLE "Customer" (
  id INTEGER NOT NULL,
  name VARCHAR(64) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE (id)
);

CREATE TABLE "Movie" (
  id INTEGER NOT NULL,
  customer INTEGER NOT NULL,
  movie VARCHAR(64) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE (id),
  FOREIGN KEY(customer) REFERENCES "Customer" (id)
);

Let's populate with some data:

INSERT INTO Movie (id, customer, movie) VALUES(0, 0, 'Rambo');
INSERT INTO Movie (id, customer, movie) VALUES(1, 0, 'Rocky');
INSERT INTO Movie (id, customer, movie) VALUES(2, 1, 'Mamma Mia');
INSERT INTO Movie (id, customer, movie) VALUES(3, 1, 'Grease');
INSERT INTO Movie (id, customer, movie) VALUES(4, 0, 'Terminator');

As long as you are happy with applying some aggregate functions, you can just use a single SELECT like this:

SELECT
  customer, MAX(id),
  COUNT(DISTINCT id), AVG(LENGTH(movie)), SUM(LENGTH(movie))
FROM Movie GROUP BY customer;

If you need to get data from the Customer table, you might need to make a join like this:

SELECT
  Customer.name, Customer.id, MAX(Movie.id),
  COUNT(DISTINCT Movie.id), AVG(LENGTH(Movie.movie)), SUM(LENGTH(Movie.movie))
FROM Movie
JOIN Customer on Movie.customer=Customer.id
GROUP BY customer;

If you want to do k-means, SQLite is quite limited, but with other dialects you can still do it. See this paper for an example. On the internet one can easily find examples in PostgreSQL, MySQL, SQL Server,... you could google yours.

Most other SQL dialects would have a richer set of predefined functions. For more complicated analysis you might need to define your own functions, and probably work with the results out of your SQL.


Paraphrase detection is still a very active and very challenging research area, so it's unlikely that there are full-fledged standard libraries for this task since there is still no clear "best solution" to this problem.

In order to build a corpus you might want to look at how shared tasks/competitions have done it before. I know at least of SemEval which often proposes tasks related to paraphrases (there might be others). I haven't checked but usually the overview paper of the task (e.g. here) explains how the corpus was built and presents the main approaches submitted by participants.


does storing file in hdfs parallelize it for Spark?

As you can see for the following examples presented here at the documentation: https://spark.apache.org/examples.html
you're able to read directly from HDFS without much trouble. And spark will parallelize the data for you properly.

We only use parallelize when you're coming data structures that were built in Scala themselves (like a val assignment or something like that).


sklearn KNN fit throws out error : value too large for dtype('float64')

I'm almost certain that you have missing values, check that and fill them beforehand
Assuming you data frame is called df
df.info() will give you that info
Additionally df.describe() is a good method to validate the maxmimum value is not np.inf


Scikit learn: which regressors natively support multi-target regression?

In scikit-learn, there is a MultiOutputMixin class. So, you could check if the estimator inherits from that one.
Alternatively, you could use estimator tags and check if {'multioutput': True} is included in the tags of the estimator.


I think you are confused on many levels here.

Logistic regression is a classifier. It gives you the probability that given an $x_{i}$, the probability it belongs to a class or classes.
So you will be having your evalinvest as classes, not continuous values.
When you call predict method on top of trained LogisticRegression model, it predicts class for each $x_{i}$ in your test set. So you expect a single array of class labels they belong to.
sklearn.metrics.confusion_matrix function takes in the original class values to the ones that are predicted by the model you had trained and returns what $x_{i}$ has been classified into what class. So it returns an n(c) x n(c) array where n(c) is the number of classes. So the diagonal of the matrix indicates the number of elements of class i being classified as class i. And an ideal model should be expected to get this numbers good. The mistake which you had made you sent in your X (train) matrix into it and y (labels) into it, which is wrong. You are supposed to send in y_true(true labels) and y_pred (predicted labels from the model). Check the documentation for more.
Ideally the function should have thrown an error. But unfortunately you had only one row in your X and one in y and they both are of same size and it passed assert it had to make.
And np.ravel converts an multi-dimensional array into 1D array. So if you doing a binary classification, you would be having a 2x2 matrix, whose flattened array would have been four elements and the assignment would have worked. But the ravelling you had made releases $n(c)^2$ elements. So there you go ValueError. See the docs

I hope my other answer on confusion matrices may clear things a little more.

Hope it clears some mistakes you are making.


How to choose an appropriate Machine Learning algorithm?

Try as many models as you have time for and pick the best one, otherwise use your intuition to narrow the candidates. If you have no intuition, start with a simple model and move to more complex models as performance requirements dictate. Often there are reasons beyond pure performance metrics to choose a model in the real world, such as simplicity, interpretability, and speed.

You will not, however, have to pick between supervised and unsupervised learning, for these are drastically different things. If I ask you to group all similar items, how are you going to apply supervised learning?

You might like this map about choosing the right estimator.


Why is backpropagation used for finding the loss gradient?

If you know that $f(x) = wx + b$, then you also know that $\frac{\partial f}{\partial b} = 1$.  The function LossGradient does not use that information, and would estimate it by evaluating the function $f$. Then it computes the loss. In general, it computes $f$ and $Loss$ once for every learnable parameter.
Backpropagation uses knowledge about the function being optimized to be more efficient. This means first evaluating $f$ once, then evaluating $Loss$ once, then updating all of your parameters. This efficiency may not matter to you when you have a small number of parameters, but it will make a huge difference if $f$ has hundreds of millions, or hundreds of billions of parameters.
It's not that an approach like yours is never useful (see derivative-free optimization), it's just that if you have extra information, using it is helpful.


How to model user choice probability: binary model vs multi class model

This is a textbook multi-armed bandit problem where Morpheus needs to learn the correct policy about offering pills. As you’ve said the Neos are independent, and making the assumption that there is a best pill overall, we need an algorithm that will experiment with each of the pills to find out which one is most likely to be accepted. This is the same as having three one armed bandit slot machines and trying to find out which one will pay out most frequently.
In the case where the Neos are observable (so that we have some information about each Neo and can predict what pill they would like based on their characteristics) this becomes a contextual bandit problem. This is the basic form of reinforcement learning problems
In a contextual bandit problem, you need to balance exploration (trying out offering different pills to different Neos to find out what they like) with exploitation (choosing what seems to be the best pill based on what we saw so far). This is why straight-up supervised multinomial classification approaches (as in e.g. Benji Albert’s answer) will struggle to converge: they don’t explore the “action space” (i.e. try out a bunch of responses) sufficiently in order to generate a variety of training examples for themselves.