Determine model hyper-parameter values for grid search

I am not sure if I understand your question correctly, but in your model, you are tuning your "alpha" parameter, you have a range from 1 to 0. (1 -> 0.1 -> 0.01 -> 0.001 -> 0.0001 -> 0).

The grid search will evaluate each algorithm (SVD, CHOLESKY,...) with each possible value of your "alpha" parameter.
It will define the score for each alpha parameter (eg. accuracy  / auc). The score metric depends on your estimator / choice

Documentation:

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html


I guess if you know what you want to extract you can just find it using regular expressions for integers or car specifications.


What is the purpose of Logit function? At what stage of model building process this logit function is used?

Has Logit function (i.e. Logit equation LN(P/1-P)) being derived
  from Logistic Regression equation or its the other way around?

Neither, really. It comes from the probability distribution that you assume in a logistic regression model. First, logistic regression as you probably know is a specific case of a generalized linear model (often abbreviated as "GLM"). A GLM is just that; a generalization of the famous "standard" linear regression model. The "standard" model assumes the conditional distribution of your response, given your predictors (denote this as $Y|X$) is normally distributed. However, in a classification problem we clearly do not have a target variable that can take on any value in the interval $[-\infty, \infty]$, but rather, only two possible outcomes (0 or 1).

A GLM requires three components; a specified probability distribution of $Y|X$ from the exponential family of distributions (important in the context of your question), a linear predictor (i.e. $B_0 + B_1X_1 + B_2X_2 + ...$, denote this as $XB$ in matrix notation), and finally, a specified link function that "links" the linear predictor to the mean (denote this as $\mu$) of the specified probability distribution.

First, let's discuss the probability distribution as this answers the question in quotes. In any logistic regression, you are assuming that $Y|X$ follows a Bernoulli distribution. Recall that a Bernoulli distributed random variable takes on 1 with probability p and 0, with probability 1-p. The mean/expected value of a Bernoulli random variable is therefore $1 *p + 0 * (1-p) = p$. This is clearly much more appropriate to our binary classification problem then say, a normal.

Also recall that I mentioned the importance of the exponential family of distributions. Well, it turns out that all members of this family of distributions have density functions that can be further factored into a very specific form with isolated terms in an exponential. One of these terms is a function that is only an argument of the scale parameter of the distribution. For the Bernoulli distribution, it turns out that this function of the scale parameter is exactly the logit link function, $ln(\frac{p}{1-p})$. Therefore, using the logit link function allows us to map our linear predictor to the exact form of the density function of the Bernoulli distribution.

There are theoretical reasons as to why we in general prefer to use the canonical link function (this is concerned with sufficiency) but there are also many reasons as to why we don't use canonical link functions. For example, the Gamma distribution has support on $[0, \infty]$ but its canonical link function is the negative inverse which runs on $[-\infty, \infty]$. Most prefer to use the log link instead for the Gamma distribution for this reason.

There are other legitimate link functions for classification problems as well. The probit link function, for example, or $ln[-ln(\mu)]$.

  What is the purpose of Logit equation in logistic regression equation? How logit function is used in Logistic regression algorithm? Reason for asking this
  question will get clear after going through point no. 3 & 4.

I basically explained this above. The logit equation, like you correctly state, is used to map the linear predictor $XB$ which runs on $[-\infty, \infty]$ to an interval on $[0,1]$, a probability. That's basically the main purpose of the function but again, there are others you can use that also have supports that run on $[0,1]$, but may have less desirable statistical properties when compared to the logit.

  Upon building a logistic regression model, we get model coefficients. When we substitute these model coefficients and respective predictor values into the logistic regression equation, we get probability value of being default class (same as the values returned by predict()). a) Does this mean that estimated model coefficient values are determined based on the probability values (computed using logistic regression equation not logit equation) which will be inputed to the likelihood function to determine if it maximizes it or not? If this understanding is correct then, where the logit function is used in the entire process of model building.

The estimated model coefficient values ARE determined based on what link function you choose. In the fitting process, the link function is what provides the map between the linear predictor with fitted model coefficients and the mean of the fitted distribution (in our case for logistic regression, the parameter $p$ as explained above). The objective function = log likelihood will end up looking like this:

$$\sum_{i=1}^{n} (y_i * ln(p_i) + (1-y_i) * ln(1-p_i))$$

where $y_i =$ 0 or 1, $p_i =$ predicted probability of class 1 for the ith observation, and $n =$ number of observations.

Notice how there is no linear predictor and no fitted coefficients in this expression. But this is where the link function comes into play; namely,

$X_iB = ln(\frac{p_i}{1-p_i}) \rightarrow p_i = \frac{exp(X_iB)}{1 + exp(X_iB)}$. So the fitted coefficients in the linear predictor are influenced by your choice of link function. There is no way to relate the linear predictor and its coefficients to the loglikelihood function otherwise. We find the most optimal $B$ that maximize the expression above.

  Assume that - "Neither logit function is used during model building not during predicting the values". If this is the case then why do we give importance to logit function which is used to map probability values to real number values (ranging between -Inf to +Inf). a) Where exactly the logit function is used in the entire logistic regression model buidling process? Is it while estimating the model coefficients?

See the above to (hopefully) be convinced that the logit link function is used during model fitting and is not arbitrary. This isn't a very rigorous explanation but I hope it clears up some of your doubts.

  The model coefficient estimates that we see upon running summary(lr_model) are determined using linear form of logistic regression equation (logit equation) or the actual logistic regression equation?

In general, you can typically specify whether you want just the linear predictor (i.e. $X_iB$) or the predicted probabilities (i.e. $p_i = \frac{exp(X_iB)}{1+exp(X_iB)}$). In R this is done via. an argument to the predict function, not sure how it is done in Python but I assume it is similar.


NLP Transformers: How to get a fixed sentences embedding vectors size?

I don't think there will be a definitive answer, but I suspect that you'll get better results using the averaging method rather than the padding method.

One big problem with the padding method is that it's sensitive to word order. For example, the sentences "Gibbons are one type of ape" and "One type of ape is the Gibbon" look very different if we do a word-by-word comparison.  "Gibbons" is very different than "One", "are" is very different than "type", etc.  This problem will be offset slightly because RoBERTa embeddings are context-sensitive, but you get the idea.

Another big problem is that the padding can dominate the similarity comparison for sentences with different lengths.  Suppose we want to compare the following two sentences which have very similar meaning:

Sentence1: "Very furry" would be an apt description for most mammalian species.

Sentence2: Mammals have hair.

We would add 8 padding tokens to sentence2 in order to give it the same length as sentence1.  But then the similarity computation is dominated by the padding tokens and not by the actual content of the sentence.

To me, the averaging approach seems superior because it avoids these two problems.  Of course it also has drawbacks.  The "sentence embeddings" obtained by averaging the word embeddings will be noisy because each word is given equal weight.

You could try running a keyword detector over the sentences you wish to compare.  Then compare the means of the top-k keywords instead of the whole sentence.


How do you identify whether your RMSE score is good or not?

Note that RMSE is an easy to understand metric. Its the Root of the Mean Squared Error. So this is just how is the typical error.
If your target is something like how big is a building, and the mean of the target its 100m, then having an error of 0.3m its nothing. On the other hand if you predict the size of insect, and your target mean is around 0.1m then an error of 0.3m is huge.
For your case it seems like a good result, an RMSE in test of 0.3, in a range of [-3,3], but then this depends on your problem.


The Indico.io API supports Spanish (and
Chinese (Mandarin), Japanese, Italian, French, Russian, Arabic, German, English).

eg in Python:

>>> import indicoio
>>> indicoio.config.api_key = <YOUR_API_KEY>
>>> indicoio.sentiment("¡Jamás voy a usar esta maldita aplicación!  No funciona para nada.")
0.02919392219306888
>>> indicoio.sentiment("¡Es patrón!  La mejor que he visto.  Punto.")
0.8860221705630639

If this isn't your area, then that's probably the easiest sort of solution to integrate.


Neural Networks - Strategies for problems with high Bayes error rate

You have a high Bayes error rate and it means that you almost can not learn anything. You have to add extra features and investigate whether your data has a small Bayes error or not. Currently, it is worse than a disaster. This large Bayes error illustrates that you have patterns, input vectors, that have completely same components as each feature but different labels. Take a look at here.


Stanford NER - Increase probability for a certain class

Your data set will influence the labeling results. If it is focused on organizations, the NER should favor them simply by virtue of the data it's fed. So you might need to do anything. But if you do observe undesirable behavior in the resulting NER, you can adjust the weights.

It's the same with any machine learning algorithm. A humorous demonstration was provided by Google's Deep Dream: it kept seeing dogs every where. Why? Because the data set they used for training had an abundance of dogs.

(And Carl Zeis should be labeled as a person. The company is Carl Zeiss.)


Simple question about prediction classes of item in question vs not item in question

It depends if the percentages are required to sum to 100%. Typically when training on only 2 classes, the model will make predicts that sum to 100% for 2 classes. There will be no chance for out-of-class predictions.


Support Vector Classification kernels ‘linear’, ‘poly’, ‘rbf’ has all same score

You will love the answer to this one...

Take a look at your code and notice that you are calling the scoring function and each time you are passing in the exact same values i.e. they are all spitting out the lin_svc.score().  Try interweaving the four scoring calls below the four respective fit calls and you should see the desired variation in the result.

# we create an instance of SVM and fit out data. We do not scale our
# data since we want to plot the support vectors
#rbf is for gaussian
C = 1.0  # SVM regularization parameter

svc = svm.SVC(kernel='linear', C=C).fit(X_train, y_train)
print svc.score(X_test, y_test)

rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(X_train, y_train)
print rbf_svc.score(X_test, y_test)

poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(X_train, y_train)
print poly_svc.score(X_test, y_test)

lin_svc = svm.LinearSVC(C=C).fit(X_train, y_train)
print lin_svc.score(X_test, y_test)

Similarly, you are doing the same thing below also.

Hope this helps!


I have added a dense layer to the network:

model = Sequential()

model.add(LSTM(32, return_sequences=True, input_shape=(None, 5)))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(32, return_sequences=True))
model.add(Dropout(0.2))
model.add(TimeDistributed(Dense(5)))  # <--- This is new :)
model.add(Activation('linear'))
model.add(Lambda(lambda x: x[:, -50:, :])) # Grab only the last 50 values

This has provided some improvement :

These are two sine waves and the predictions of the last 50 data points of them.
As can be seen, the results have improved, with some of them being okay approximations of the sine waves. Other times... not so much.

Still a lot better than the previous results.

I then tried another approach: Instead of having one (multilayered) LSTM, in which I ignore all but the 50 inputs (in a sense trying to predict from each one of the 50 last values the value located 50 places after it... which might not be an ideal approach) I split the network into 2 LSTMs, with one receiving the input and "injecting" its results via a dense layer into the second LTSM layer which generates the output.

features_num=5

model.add(LSTM(40, return_sequences=True, input_shape=(None, 5)))
model.add(LSTM(40, return_sequences=False))

model.add(Dense(80))
model.add(Activation('tanh'))
model.add(RepeatVector(50)) # 50 output points

model.add(LSTM(40, return_sequences=True))
model.add(LSTM(40, return_sequences=True))

model.add(TimeDistributed(Dense(features_num)))
model.add(Activation('linear'))

Results are, for the most part, better:


How can I arrange data in different columns (for different constituents) from one?

It is passing 6 days from your question and I didn't want to waste your chance of getting an appropriate response by answering. So, my solution might be naive.

Step 1: Changing the categorical data of column Sample to numerical

dictionary = {'SITE_NAME':['SITE_7_1', 'SITE_7_1','SITE_7_1'],
'SAMP_DATE_TIME':['10/04/1988 00:00:00', '10/04/1988 00:00:00', ' 10/04/1988 00:00:00'],
 'SAMPLE':['Aluminium','Aluminium','Beryllium'],
 'Value':[150, 150,5]}
import pandas as pd
dataset = pd.DataFrame(dictionary)
X = dataset.iloc[:,:].values

from sklearn.preprocessing import LabelEncoder
labelencoder_X = LabelEncoder() #Creating an object from LabelEncoder class
X[:,2] =labelencoder_X.fit_transform(X[:,2])

Result so far:

array([['SITE_7_1', '10/04/1988 00:00:00', 0, 150],
       ['SITE_7_1', '10/04/1988 00:00:00', 0, 150],
       ['SITE_7_1', ' 10/04/1988 00:00:00', 1, 5]], dtype=object)

Step 2 Adding the dummy variables as columns

dummies = pd.get_dummies(dataset['SAMPLE'])
df = pd.concat([dataset, dummies], axis=1)  #concating the dataset and the dummies

The result so far:

  SITE_NAME        SAMP_DATE_TIME     SAMPLE  Value  Aluminium  Beryllium
0  SITE_7_1   10/04/1988 00:00:00  Aluminium    150          1          0
1  SITE_7_1   10/04/1988 00:00:00  Aluminium    150          1          0
2  SITE_7_1   10/04/1988 00:00:00  Beryllium      5          0          1

Step 3: filling the dataframe

The logic for filling the values in their corresponding columns is as follows :

for i in range(len(df)):
    if df.iloc[i]['SAMPLE'] == 'Aluminium':
         df.at[i, 'Aluminium'] = df.iloc[i]['Value']

If we want to apply this to the whole dataset, we can change that Aluminium each time for different samples
After this, you can groupby your dataframe to have the rows based on SITE_NAME or whatever you want.
feel free to ask if there is any vague point here.

Filling in the whole table

samples = list(df['SAMPLE'].unique()) #storing all the unique values of the SAMPLE column

Creating a dictionary of samples as keys and their values as values:

dictionary = {}
df_new = df.groupby(['SAMPLE'], as_index=False).mean() #dataframe in which `SAMPLE` is grouped.
for i in range(len(df_new)):
    dictionary[df_new.iloc[i]['SAMPLE']] = df_new.iloc[i]['Value']

now dictionary equals:
{'Aluminium': 150, 'Beryllium': 5}
Now we can fill in the table:

for sample in samples:
    for i in range(len(df)):
        if df.iloc[i][str(sample)] == 1:
            df.at[i, str(sample)] = dictionary[str(sample)]


Joint spatial clustering: How to force clusters to minimally contain datapoints from all datasets

I don't understand the purpose of imposing a condition that requires any cluster to contain at least some (let's say even 1) point from every dataset but at the same time find a solution that also allows that condition to be broken by outliers (I assume these are specific cluster outliers or data set outliers?).

Have you considered implementing an overlapping cluster algorithm?  You don't get the full distinctness of a traditional single membership but your might better fit your data to an algorithm that achieves the desired condition.  It will probably require some exploration of the data and testing for cluster membership after the fact.


Multiple solutions with same minima in MLP with same weights

If one permutes the connections of the hidden layer ($d!$ ways to do that), and move and rename connections appropriately, then one effectively has the same MLP with the exact same minima, yet the configuration has changed (in a trivial sense). Thus there are (at least) $d!$ configurations only trivialy different with the exact same minima.
To see it in your notation, effectively the hidden layer output is the following sum:
$$O_{\text{hidden}} = \sum_{i=1}^d w_{\pi_i} \cdot x + b_{\pi_i}$$
Where $\pi_i$ is some order of the connections. For example $\pi_i = i$
But for $d$ items there are $d!$ permutations thus $d!$ order functions $\pi(i) = \pi_i$. Yet the difference is only in re-ordering the configuration and is trivial. The rest follow from that.
This is one reason why Neural Networks are non-convex models.
See also: Explanation of why Neural Networks are non convex


Accelerate deep learning model training on several GPUs

What kind of Framework are you using? If you are using Keras, the newest Version of it supports multi-gpu How can i run a keras model on multi gpus

If you have an RNN network, using GPU wont give you much more performace. Most of the times CPU learning is faster in my experience.

Please provide more informations to your problem.

Update

If you have a more complex architecture in Tensorflow or Keras you can give each process Step in your Model a specific GPU/CPU which this part of the Model will be working on. You can do it with

with tf.device('/gpu:1'):

I think you just have to try it out. It depends on the Model you have( which you did not provide, so the complexity is not known) and on the volume of the data. Is your whole dataset in the RAM or do you generate each batch individually?
If you have several branches in your architecture and they are concatenating at a certain point. Which branch is the slowest?

First of all I would try to train your Model on the CPUs before doing anything else.

This post gives a quite good overview how you can use multi GPU.


Seq2Seq Model training: Encoder vs. Decoder

You can check medium page, https://link.medium.com/iolQKIWawX.
You can also get more detailed pages on medium itself, shared link includes the references of those.


In a machine learning context, a dichotomy is simply a split of a set into two mutually exclusive subsets whose union is the original set. The point being made in your quoted text is that for four points, a linear boundary can not form all possible dichotomies (i.e., it does not shatter the set). For example, if the four points are arranged on the corners of a square, a linear boundary can be used to create all possible dichotomies except it cannot produce a boundary that splits the two points lying along one diagonal from the other two points (and vice versa), as you indicated in your own answer.


regarding the inclusion of missclassified data in final training

Since these examples can't be learned, why not remove them and retrain the final model with only the 92% of the data that can be learned?

In general, I think this is a bad idea for the reasons below. This being said, the only sure way to know with a particular dataset is to experiment.

This would modify the distribution of the data. If the errors tend to happen more often with a particular class or a specific combination of features (this is quite likely), these cases would not be seen anymore by the model. Sometimes modifying the distribution can lead to better performance (e.g. when using resampling), but it can also do the opposite. So this is a bias with unknown effects on the performance of the model.
A more serious problem is that these error cases are likely to be important for the model. Typically errors happen with borderline cases, and these cases are usually very important for the model to learn the optimal way to separate the classes. By removing them it might be easier for the model to separate the classes during training, but that's not a good thing since it won't have all the information it needs in order to minimize errors. As a consequence it's likely that it won't find the right optimum and therefore will make more errors.


What is the type of each variable i.e. Categorical or Continuous?

Name attribute cant be continuous, if you really want to use it then take it as a categorical. It depends on what is your use case.
Gender is categorical.
Date of Birth/Annual Income/Tax Paid are in general continuous variable. But you may change them in categorical depending on your use case.


Do you run ModelCheckpoint on its default parameters (besides monitor)?

ModelCheckpoint has a parameter called mode which specifies the type of metric to be used. mode can take 3 values 'min' 'max' and 'auto' (which is the default):

min: means that you want to minimize the metric (e.g. the loss function).
max: means you want to maximize the metric (e.g. accuracy).
auto: attempts to figure what to do on its own. If you look at the code, it checks if the metric's name contains 'acc' or if it starts with 'fmeasure'. If yes it sets the mode to max, if not it sets it to min.

In your case, you monitor the jaccard index, which is a metric you want maximized, so you want the mode set to max. Normally because "jaccard" contains the string "acc", even if the mode is set to auto it should work fine.

If however you named your metric something arbitrary (e.g. my_metric), the default mode will be set to min, which means that it will store the weights that achieve the least performance on your metric, which should be the weights of the first epoch.

Suggestion: next time try with mode='max' to be sure.


Trying to understand encoder-decoder sequential models in Keras?

First of all, I personally think that the code you provided is unnecessary since you are looking for architectural comparison between models. It would have been easier (and less bulky) to include the graphs of your vanilla and stacked LSTM networks for simplicity. You can do that by keras.utils.plot_model() as explained here.

Answering your questions:

  I don't see how we are using only the decoder in the predict step...

Uber's model does not use the decoder component of the autoencoder. It uses a separate sequential LSTM model, judging by the graph provided.

  ...which variable or method in Keras corresponds to the embedding that
  we would be using as an input for predicting new time series?

The input to the forecasting model is simply the latent space of the autoencoder. In particular, the latent space constitutes a condensed representation of the past observations, so that you don't need to include the complete timeseries because it might be very long and include redundant information, repetition, noise, etc. Instead, you compress it into something similar to a "zip" file, that has smaller size but still includes all the essential information. Then, you train the "forecasting" network to predict future values based on the "zipped" past information instead of training it on the original timeseries.

  How can I implement the code for a model similar to the one in the
  illustration using Keras?

Check the 4th model of this autoencoder example. First, train the autoencoder. Only after it is properly trained and optimized, train a simple LSTM forecasting model, like the one in the code you provided.

The key is to use the latent space of the autoencoder to train the forecasting model, which in the current example is the output of the 1st LSTM layer. To do that, you need to first encode your complete dataset into a latent representation using the already trained encoder, and then use these encodings to train the forecasting model in the second step.


Grid seach is unavailable for Keras in case of multiple outputs?

It does not appear possible  as stated here.  The problem is basically how sklearn expects it's input.  Currently it only accepts a 2d array.  I think it'll be rather straight forward to go to sklearn wrapper and modify is so that it can accept multiple arrays.


There are multiple possiblities from dusk, to others model etc.
Here are my 2 favorites, not to loose you in the number of possibilities:

www.h5py.org/ "It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want."

Try online learning with Cousin models of random forest (light-gbm). He has online learning capabilities.


How to binary encode multi-valued categorical variable from Pandas dataframe?

If [0, 1, 2] are numerical labels and is not the index, then pandas.DataFrame.pivot_table works:

In []:
data = pd.DataFrame.from_records(
    [[0, 'A'], [0, 'B'], [1, 'B'], [1, 'C'], [1, 'D'], [2, 'B'], [2, 'D']],
    columns=['number_label', 'category'])
data.pivot_table(index=['number_label'], columns=['category'], aggfunc=[len], fill_value=0)

Out[]:
              len
category      A      B      C      D
number_label
0             1      1      0      0
1             0      1      1      1
2             0      1      0      1

This blog post was helpful.

If [0, 1, 2] is the index, then collections.Counter is useful:

In []:
data2 = pd.DataFrame.from_dict(
    {'categories': {0: ['A', 'B'], 1: ['B', 'C', 'D'], 2:['B', 'D']}})
data3 = data2['categories'].apply(collections.Counter)
pd.DataFrame.from_records(data3).fillna(value=0)

Out[]:
       A      B      C      D
0      1      1      0      0
1      0      1      1      1
2      0      1      0      1


Estimating average daily consumption with samples randomly scattered in time

If you only want the average daily consumption over the whole period of time, you can simply calculate the difference between the last and first reading and divide by the total number of days.
As far as I understand your explanation:

your method 1 would not give you the true average over the period, unless you multiply each individual average by a weight corresponding to the number of days in the period.
your method 2 gives the true average, since it would represent every day individually (if I understand correctly)

However if you want to be able to observe the variations across time while smoothing some of the irregularities, you could:

calculate the daily average for every day like in method 2.
calculate a rolling average over a fixed number of days for every day. This is not perfect if there are long periods with no reading, but it should show the general trend better.
Plot a graph based on rolling average.


When being in a perfect "Long Valley" situation, does momentum help?

The answer is simple: in a perfect situation never. These optimizers are 1st order methods (unlike for example Hessian-Free optimization). The reason for they escaping from the stationary point above is that there is some small gradient (too small for ordinary SGD) in the "correct" direction. Momentum and Nesterov gradient can build up the velocity over time in that direction and Adagrad/Adadelta/RMS can immediately scale the gradient (increase it in the direction for which the past gradients were small).


Machine learning Classification model for binary input and output data

Your problem is one of "sequence classification" for which Recurrent Neural Networks (RNN) e.g. Long short-term memory (LSTM) are generally used.

See here for a good example: https://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/

and here for a technical paper: https://www.cs.sfu.ca/~jpei/publications/Sequence%20Classification.pdf

Here is a specialized package for sequence classification which uses convolutional neural network (CNN): https://github.com/aajanki/sequence-classification

CPT algorithm, an accurate method for sequence prediction, can also be used here: https://www.analyticsvidhya.com/blog/2018/04/guide-sequence-prediction-using-compact-prediction-tree-python/ A continous output can easily be rounded to 0 or 1 to get binary result.


Selecting a boundary on a binary classifier to optimal precision and recall

The intersection of the precision and recall curves is certainly a good choice, but it's not the only one possible.
The choice depends primarily on the application: in some applications having very high recall is crucial (e.g. a fire alarm system), whereas in some other applications precision is more important (e.g. deciding if somebody needs a risky medical treatment). Of course if your application needs high recall you'd choose a threshold before 0.6, if it needs high precision you'd choose a threshold around 0.85-0.9.
If none of these cases apply, people usually choose an evaluation metric to optimize: F1-score would be a common one, sometimes accuracy (but don't use accuracy if there is strong class imbalance). It's likely that the F1-score would be optimal around the point where the two curves intersect, but it's not sure: for example it might be a bit before 0.8, when the recall decreases slowly and the precision increases fast (this is just an example, I'm not sure of course).
My point is that even if it's a perfectly reasonable choice in this case, in general there's no particular reason to automatically choose the point where precision and recall are equal.


The API allows two types of operations: Learn and transform a matrix under analysis. Or alternatively, learn the factorization on one matrix (myfile / X) and then apply the same transformation on a new matrix (X_new), having the same number of columns (but potentially different number of rows).

Think of it as learning the basis transformation on one set of input data (.fit method) and then apply the same basis transformation on new set of data (.transform method).

If you are interested in transforming the same matrix (myfile in your case), use only the fit_transform method instead of transform.


Exogenous Variables in Time Series Models

If your error is already very low, you don't have to use xreg, otherwise you may increase the error, but it's worth to try. But if your error is high because your model can't capture the seasonality that you want, try to use an exogenous variable (xreg) for forcing an seasonality.

You can use weekends/holidays dates as an xreg, recurring 0/lowest values is an example of the very thing you want your model to learn. But in my experience, simply try to use the X histogram (binned X) as an xreg, before you try anything. I found that fact after my months of confusion.

The intuition is you can see the histogram/bin as a product of the seasonality itself.


It's hard to say without your data.  I'd comment this, but it's a bit long and better formatted as an answer.
The part of the source code that's relevant is quite short, so you can go through step by step to see what's wrong:
bins = np.linspace(0., 1. + 1e-8, n_bins + 1)
binids = np.digitize(y_prob, bins) - 1
bin_sums = np.bincount(binids, weights=y_prob, minlength=len(bins))

digitize can return 0 if an element is to the left of all the bins; that shouldn't happen here, but maybe some computational precision error has crept in?  Check what values you get for binids above.
See also this StackOverflow question, where NaNs were the culprit.


If your objective is to find clusters of users, then you are interested in finding groups of "similar" reviewers.

Therefore you should:

Retain information which relates to the users in a meaningful way - e.g. votes_for_user.
Discard information which has no meaningful relationship to a user - e.g. user_id (unless perhaps it contains some information such as time / order).
Be mindful of fields which may contain implicit relationships involving a user - e.g. vote may be a result of the interaction between user and ISBN.


What is the best way to feature engineer features which have more than one repeated values?

Depending on the kind of ML problem you're facing, there might be more or less suitable methods.
Have you tried one-hot-encoding ? It should actually answer your question, by defining people as dense vectors filled with 0s excepted at a given company's coordinates, where they'll show a 1.
Well it's kinda bruteforce and the drawback is that it will for sure increase the dimension of your feature vectors, but it's a good starting point.


Is it possible to create a rule-based algorithm to compute the relevance score of question-answer pair?

could we invent an algorithm like BM25 to compute the relevance score of question-answer pair?

It depends:

BM25 (actually cosine with BM25 weighted vectors) is a simple similarity measure, ultimately based on counting words in common. Proposing a different similarity measure is easy, for instance there are various measures used for MT evaluation (including some quite sophisticated ones) which could be used as well. Of course, these measures don't actually measure the relevance, they just offer a crude approximation.
However if there was such a rule-based algorithm which would be able to actually measure the relevance of an answer in any context, then for all means and purposes we would have solved AI: judging the semantic relevance is much more subtle than counting words in common. In particular if there is such an algorithm, then the problem of question answering is solved: you can just generate all the possible answers and loop until one is found relevant to the question.

People have tried to do "intelligent" rule-based algorithms in NLP for decades, before realizing that ML is more efficient and performs much better in most tasks. So it's extremely unlikely that a rule-based algorithm would suddenly outperform ML on a non-trivial task like this.


Welcome to Datascience.SE!

Like you said, you can extract the day of the week. Also extract the hour of the day, then encode these two variables using sines and cosines with their respective periodicities (7 and 24). Also create a column for the UNIX/epoch time. If there are "special days", such as holidays or sales, create boolean columns for them too.


TL;DR: Array-tidy is extending tidiness across multiple data frames that are indexed by an additional shared dimension, such as time.

From what I understand, array-tidy and dataframe-tidy refer to the underlying data models that are used to achieve your tidy data. It seems to be a reference to how plyr and pandas differ in their data structure logic

Pandas was developed around the idea of panel data, which is your typical two dimensional data frame of observations and variables, but with samples accumulated over a period of time. Pandas name even stems from this particular type of data (panel data structures, per this McKinney presentation and the pandas docs). The inclusion of a third dimension, thinking about your data over time, seems to have informed some thinking around how pandas would evolve. I recommend checking out the slides (specifically the one titled "The pandas killer feature: indexing").

Relating this back to the Tidy Data paper, creating multiple data frames across an additional dimension (like you would with a Panel in prior versions of pandas) seems to qualify as the "high-dimensional arrays" Wickham discusses in the section you quoted.

I hope this helps!


What can I use to post process an NLP tree generated from the python library `spaCy`?

What do you want to do with the chat bot? How you parse it will depend on the final use case and, believe it or not, many people get the job done by simply collecting the POS they want and using some filtering.

If you want to try to maintain more of the data and perhaps abstract it makes sense to try clustering of some kind, perhaps using hierarchical methods, such as the (relateively new) hdbscan. The features on which you cluster will again depend on what you want to achieve.

If you haven't already, check out the spaCy examples for some inspiration!

Once you have a corpus with word all tagged, you can try training models that might be able to answer questions, based on user input. This will involve steps such as creating encoding of the words (or entire user questions), using embeddings such as Word2Vec, GLoVe, or simple sparse one-hot encodings. You basically need to transform words into numerical input somehow.

I hope this gives you some keywords to help you on your search  :-)


What is the difference between reconstruction vs backpropagation?

Reconstruction is used for the concept Restricted Boltzmann Machine (RBM), it describes a phase where the structure reconstructs (generates) visible samples from the hidden states of the layers. For more detail, you can refer to:
https://stackoverflow.com/questions/4105538/restricted-boltzmann-machine-reconstruction

Backpropagation is something different entirely; you can find backpropagation in Deep Neural Nets, Convolutional Nets, RBMs (in some way) and so on. Assume a Deep Neural Net with N hidden layers. Upon training we feed the input forward through the randomly initialized weights up to the last neuron. From the output of the last neuron, we calculate our loss using a cost function that calculates the error between predicted and the true output. This is forward propagation.

Knowing the loss and the loss function, we start to take derivatives backwards to find gradients of weights and biases of each layer using the chain rule, all back to the input side. This is called backpropagation. After backpropagation, we update all of the weights and biases of N layers with the gradients for each layer we calculated. Then we do forward propagation again, backpropagation again, update again and all again until we achieve our goal (which is usually minimizing the error as solving hopefully a convex optimization problem). You can also refer to:

https://www.youtube.com/watch?v=x_Eamf8MHwU for backpropagation

Also a final note: Gradients of RBM algorithm are not able to be calculated via classic backpropagation, a method called as contrastive divergence is used for the calculation. You can also have a look at the RBM paper on this matter and more, by Geoffrey Hinton:

https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

Hope I could help, good luck!


Comparing dataframe object with string value in django

I don't see anything glaringly wrong with your code, but you can try this instead:

df['classification'].value_counts()['RS']

If you omit the indexing at the end there, the call to value_counts() will concretely show you what values appear with what frequency in the classification column, which might help you debug what's going on if this still doesn't work.


Python implementation of cost function in logistic regression: why dot multiplication in one expression but element-wise multiplication in another

In this case, the two math formulae show you the correct type of multiplication:

$y_i$ and $\text{log}(a_i)$ in the cost function are scalar values. Composing the scalar values into a given sum over each example does not change this, and you never combine one example's values with another in this sum. So each element of $y$ only interacts with its matching element in $a$, which is basically the definition of element-wise.
The terms in the gradient calculation are matrices, and if you see two matrices $A$ and $B$ multiplied using notation like $C = AB$, then you can write this out as a more complex sum: $C_{ik} = \sum_j A_{ij}B_{jk}$. It is this inner sum across multiple terms that np.dot is performing.

In part your confusion stems from the vectorisation that has been applied to equations in the course materials, which are looking forward to more complex scenarios. You could in fact use cost = -1/m * np.sum( np.multiply(np.log(A), Y) + np.multiply(np.log(1-A), (1-Y))) or cost = -1/m * np.sum( np.dot(np.log(A), Y.T) + np.dot(np.log(1-A), (1-Y.T))) whilst Y and A have shape (m,1) and it should give the same result. NB the np.sum is just flattening a single value in that, so you could drop it and instead have [0,0] on the end. However, this does not generalise to other output shapes (m,n_outputs) so the course does not use it.


nltk measure the accuracy of the new features

If you don’t have a classified testing set - which allows to measure a performance score, then it is useful to use part of your training data as a validation set, meaning that you test the performance of your model on it (because you have the true labels/values).

The percentage of the split is arbitrary and depends on the degrees of freedom you have (number of individuals vs number of variables) - 0.1 to 0.25 is a common choice.


How is BCELoss counted in PyTorch? [different result comparing to mathematical implementation]

Take a look at the documentation for BCELoss

Our solution is that BCELoss clamps its log function outputs to be
greater than or equal to -100. This way, we can always have a finite
loss value and a linear backward method.

If you amend the definition to include that it should yield the correct results:
def binary_cross_entropy(pred, y):
    log_pred = torch.clamp(torch.log(pred), -100, 100)
    log_pred_rev = torch.clamp(torch.log(1-pred), -100, 100)
    return -(log_pred*y + (1-y)*log_pred_rev).sum()

a = torch.linspace(0, 1, 10)

w = np.array([F.binary_cross_entropy(x,torch.tensor(0.2)).item() for x in a])
q = np.array([binary_cross_entropy(x,torch.tensor(0.2)).item() for x in a])

print(w)
print(q)

Gives
[20.          0.53367138  0.501867    0.54409462  0.63241535  0.76630157 0.95998287  1.2535249   1.78133631 80.        ]
[20.          0.53367138  0.501867    0.54409462  0.63241535  0.76630157 0.95998287  1.2535249   1.78133631 80.        ]


Keras: X and Y are the same, yet validation accuracy is 50%, what is wrong?

You should think about how the initial values impact the ReL Units. If, for example, you use init='one' for the activation='relu' layer you'll get the desired result (in this simple setup).


color compression using k-means algorithm

As already pointed out in a comment, the 16 million is the number of colors in the RGB space, $256^3=16\ 777\ 216$.  (The image itself only contains $96\ 615$ unique colors.)

As for the reduction, the author is generating 16 clusters; while the center points are still given in RGB format, there are only 16 of them.  (If you're following along in Colab or your own clone, run np.unique(new_colors, axis=0); that gives the 16 unique colors remaining in the compression.)  Now, to actually save the color-compressed image with smaller disk usage, you'd probably need to convert those 16 RGB codes with simple label encoding, and be able to reference their colors upon loading.  Anyway, as the author points out this isn't really meant to be an effective way to reduce image size, just an interesting example of clustering.


What is NLP technique to generalize manually created rules in text?

You have many options depending on the level of complexity and creativity you may have. Among all, I would suggest going through "Natural Language Understanding" techniques.

Introduction

Not so precise but practically so to speak, NLP is about processing text and taking each token/word/phrase as an object and learning different models based on appearance of these objects and answering different questions. Here you do not really work with what they mean. Even when you do sentiment analysis (which seems to be a semantic concept) what you really do is counting tokens and seeing their labels in training data and predict the sentiment of the new one.

In NLU, you mainly deal with semantics, meanings and relation between words. For example imagine a Google search query. When I search "who was the president of US when Italy won the world cup last time?". If Google wants to rank pages for me based on NLP then many things might come up including these words and their close words (which of course helps me find the answer in related webpages). But if google needs to deliver the precise answer (which is still not easy!), it needs to understand what I say as you see an example here. For this, they use a Knowledge Base (for Google, it's called Knowledge Graph) which is the answer I have to your question.

Suggested Idea

You can make a knowledge base for your corpus if there is enough text. For this, you can extract patterns of desired phrases manually and through this patterns search all occurrences of your desired sentences (e.g. "|MEDICINE| is used for |DISEASE|") and all occurrences of your desired objects (e.g. "Tumor"). Hearst Patterns can be used for such purpose as you see in this paper. Then connecting things though a graph gives you an option for understanding that the similar pattern is happening. For synonyms and related words you can easily use existing Knowledge Bases which are a lot.

Hope it helps :)


Leave one out Cross validation using sklearn (Multiple CSV)

Leave One Out Cross Validation is just a special case of K- Fold Cross Validation where the number of folds =  the number of samples in the dataset you want to run cross validation on.

For Python , you can do as follows:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(classifier , X = input data , y = target values , cv = X.shape[0])

Here , cv = the number of folds . As cv = number of samples here , we will get Leave One Out Cross Validation.
The length of list(scores) will be equal to number of samples in input data.


Help on designing data science curriculum from "traditional" offerings

Of the traditional majors, I think computer science + statistics is the most obvious. You can substitute applied mathematics or even physics for the statistics major, if you want. Always keep the computer science courses.

If the choice is between information management and business analytics, given that you're already studying maths, it's hard to say. They're both pretty light-weight, and will add little to your math major. I would not consider them a plus per se as a prospective employer reading your resume. Unless they're cheap, I'd take online classes and get some practice instead.


Implementing PatterNet in Python as it is in MATLAB

Consider reusing an existing framework and adding the missing activation functions. For example, here you can see how to do that in sklearn.

However, notice that there might be other differences (consider this example) in how the algorithm is implemented, as with the rise of deep learning and convolution networks, neural networks have hugely grown in variety and complexity.


How to choose inputs to maximize reward for an ordered dataset?

It seems like a reinforcement learning question with continuous action space. (ob1, ob2) is the observation, (x, y) the action, and reward the reward.

You may refer to this paper from DeepMind, which provides a general deep Q-learning way solving this type of question, including some physics tasks.

However, as I don't have your data, I'm not sure if you have sufficient instances. ~70,000 instances with 4 features is (roughly) sufficient for general ML problems, but in reinforcement learning world, people use a simulation environment which is able to generate infinite samples, thus ~70,000 instances may be the bottleneck.

If you encounter such problem, you may try classic reinforcement learning (without "deep") techniques.


Is it possible to change the input columns of a trained ML model while making predictions from it without affecting the accuracy?

Is this approach a correct approach, or logical with respect to machine learning principles?

It will affect the performance of the model in the sense that your algorithm learned to separate the clusters based upon distance according to all the features. I have read discussions about how to calculate feature importance on unsupervised problems like yours, so you could do some research on that and figure out a way to measure the importance of your features so that you have an idea of how influential is a feature on your model and therefore the impact on removing one. In this case removing will imply filling the features you are not using with NaNs so your model must be prepared for such scenario (sklearn pipelines are the best way of doing this)

Will it affect the model accuracy and if yes then how?

First of all, you are referring to an unsupervised model (K-means) so the metrics you mention does not apply, rather there must be metrics on the separateness of the clusters you formed (silhouette score, etc) and according to my first answers, you may adapt a version of Permutation importance using a metric according to your problem to see how it impacts to remove a feature the general performance.

If I have to provide features B and C, then can I populate them with zeros and then provide it to trained model for making predictions

Remember you are using an algorithm that is based on euclidian distance so imputing with zeros may have an undesired result

Will the action taken in step 3) affect the model accuracy, if yes and then why and how?

Sure it will, imputing with zeros will take the features not present to the origin on the euclidean space, so be careful with that


What is the procedure to realize zero mean and unit variance?

To generate zero mean and unit variance, you are looking at standardization. Where a feature $x$ becomes scaled feature $z$ like this:

$$z = \frac{x - \mu}{\sigma}$$

Where $\mu$ is the mean of that feature and $\sigma$ the standard deviation.


Pandas: Assign back to table from grouping by column and index

EDIT:
Couldnt find a clean solution but this function should do the trick
def calc_ema(cur, *prevs):
    rele_prevs = [x for x in prevs if not pd.isnull(x)]
    rele_prevs_cnt = len(rele_prevs)
    if rele_prevs_cnt == 0:##First item
       res = cur
    else:
       x0 = rele_prevs[rele_prevs_cnt-1]
       prev_items_sum = sum([x * ((1-a) ** (i+1)) for i, x in
       enumerate(rele_prevs)])
       st = a * (cur + prev_items_sum)
       res =  st + ((1-a) ** (rele_prevs_cnt+1)) * x0
    return res

You should use it as follows:
grpd = data.groupby('group')
data['EMA3'] = list(map(calc_ema, data['price'], *[grpd['price'].shift(i) for i in range(1, n + 1)]))

Original Answer:
-- Dosnt answer OP's needs --
You can use the shift function on a grouped by object.
This will let you calculate on $Y_t$ and $Y_{t-1}$
data.sort_values(by='bin', inplace=True)
a = 0.1
data.loc[:, 'ema'] = map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1))

##python 3
 data.loc[:, 'ema'] = list(map(lambda x, y: x if pd.isnull(y) else x*a + (1-a) * y,
data['price'], data.groupby('group')['price'].shift(1)))


Is there a term for "this month last year" in a report?

Same period last year is what you want.

BTW, your question does not appear to be about data science.

References

https://msdn.microsoft.com/en-us/library/ee634972.aspx

http://www.wiseowl.co.uk/blog/s2477/same-period-previous-year.htm


As this question highly overlaps with a similar question I have already answered, I would include that answer here (linked in the comments underneath the question):

In images, some frequently used techniques for feature extraction are binarizing and blurring

Binarizing: converts the image array into 1s and 0s. This is done while converting the image to a 2D image. Even gray-scaling can also be used. It gives you a numerical matrix of the image. Grayscale takes much lesser space when stored on Disc.

This is how you do it in Python:

from PIL import Image

%matplotlib inline

#Import an image
image = Image.open("xyz.jpg")

image

Example Image:

Now, convert into gray-scale:

im = image.convert('L')

im

will return you this image:

And the matrix can be seen by running this:

array(im)

The array would look something like this:

array([[213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 176, 176, 176],
       [213, 213, 213, ..., 175, 175, 175],
       ...,
       [173, 173, 173, ..., 204, 204, 204],
       [173, 173, 173, ..., 205, 205, 204],
       [173, 173, 173, ..., 205, 205, 205]], dtype=uint8)

Now, use a histogram plot and/or a contour plot to have a look at the image features:

from pylab import *

# create a new figure
figure()
gray()
# show contours with origin upper left corner
contour(im, origin='image')
axis('equal')
axis('off')

figure()

hist(im_array.flatten(), 128)

show()

This would return you a plot, which looks something like this:

Blurring: Blurring algorithm takes weighted average of neighbouring pixels to incorporate surroundings color into every pixel. It enhances the contours better and helps in understanding the features and their importance better.

And this is how you do it in Python:

from PIL import *

figure()
p = image.convert("L").filter(ImageFilter.GaussianBlur(radius = 2))
p.show()

And the blurred image is:

So, these are some ways in which you can do feature engineering. And for advanced methods, you have to understand the basics of Computer Vision and neural networks, and also the different types of filters and their significance and the math behind them.

The entire analytics is done with the PIL package. I wouldn't claim that it's a one-stop shop for Image analytics, but for a starter to novice level, it is pretty much it.


Sklearn train_test_split() error: Found input variables with inconsistent numbers of samples

You need the shape of your training data to be (num_samples, num_features) or in your case (10000000, 2). An easy way to fix this is to transpose X before feeding it to the train test split:

X_train, X_test, y_train, y_test = train_test_split(X.T, Y, test_size=0.2, random_state=0)
regr.fit(X_train, y_train)  # <-- also 'y_train' not 'Y_train' here


Your workflow is correct and you are doing a type of transfer learning where only the last layer is fine-tuned. Image embedding widget calculates features that are the activations of a penultimate layer of Inception-v3, VGG or Painters network. All of those networks are trained on the different set of images. VGG and Inception-v3 are trained on ImageNet dataset and Painters network is trained on the set of artworks. In the transfer learning you are implementing with the workflow you fine-tune all of the connected classifiers for your images.


Can I use the Softmax function with a binary classification in deep learning?

For binary classification, it should give the same results, because softmax is a generalization of sigmoid for a larger number of classes.


how can I convert tensor shape to numpy array?

Assuming that these are pytorch tensors, you can convert them to numpy arrays using the .numpy() method. Depending on whether your tensors are stored on the GPU or still attached to the graph you might have to add .cpu() and .detach().


Feature extraction is basically reducing the amount of resources required to describe a large set of data. Analysis with a large number of variables (like images) generally requires a big amounts of memory and computation power.
You can start from the simplest and slowly work your way up (in computational requirements) until you reach your desired accuracy.

Start from:

ORB (oriented FAST, rotated robust independent features) -> SURF (speed up robust feature) -> SIFT (scale invariant feature transform) -> OverFeat (Convolutional Network) -> vgg16 ... (There are many more, any neural network model can be used for this)


Confusion in Policy Iteration and Value iteration in Reinforcement learning in Dynamic Programming

In policy iteration, you define a starting policy and iterate towards the best one, by estimating the state value associated with the policy, and making changes to action choices. So the policy is explicitly stored and tracked on each major step. After each iteration of the policy, you re-calculate the value function for that policy to within a certain precision. That means you also work with value functions that measure actual policies. If you halted the iteration just after the value estimate, you would have a non-optimal policy and the value function for that policy.

In value iteration, you implicitly solve for the state values under an ideal policy. There is no need to define an actual policy during the iterations, you can derive it at the end from the values that you calculate. You could if you wish, after any iteration, use the state values to determine what "current" policy is predicted. The values will likely not approximate the value function for that predicted policy, although towards the end they will probably be close.


Choosing a suitable learning rate based on validation or testing accuracy?

You cannot select a parameter based on test accuracy, because the moment you do that, it becomes a validation accuracy as it has affected the final model. Therefore, you are always choosing based on validation accuracy.

As a result, the best result comes from learning rate 0.001, with the highest validation accuracy 0.6900. We have ignored Testing accuracy. If we select based on Testing accuracy, it becomes a validation accuracy.

Generally, a learning rate that is a looser at epoch 50, might be a winner at epoch 200. In other words, a slower convergence may lead to a higher accuracy. Therefore, this issue is worth considering too.


Can anyone explain the reasoning behind this post?

"Weight of $-4$" seems to be the key thing you're missing.  Translating a little, the neuron's value is $\operatorname{relu}(5-4x_1-4x_2)$.

Oh, and there seems to be a typo in the post:
"Then, if X2 also gets switched to 0" should be "switched to 1"


RandomForest and tree feature importance in scikit-learn

_feature_importance of a random forest calculates the average feature importance across all trees in the forest. While tree.feature_importances_ is the feature importance for a single tree.

Since feature importance is calculated as the contribution of a feature to maximize the split criterion (or equivalently: minimize impurity of child nodes) higher is better.

You can see how it works in the source code:

The property _feature_importance of random forests is defined as following (see here for the complete source code):

@property
    def feature_importances_(self):
        """
        Return the feature importances (the higher, the more important the
           feature).
        Returns
        -------
        feature_importances_ : array, shape = [n_features]
            The values of this array sum to 1, unless all trees are single node
            trees consisting of only the root node, in which case it will be an
            array of zeros.
        """
        check_is_fitted(self)

        all_importances = Parallel(n_jobs=self.n_jobs,
                                   **_joblib_parallel_args(prefer='threads'))(
            delayed(getattr)(tree, 'feature_importances_')
            for tree in self.estimators_ if tree.tree_.node_count > 1)

        if not all_importances:
            return np.zeros(self.n_features_, dtype=np.float64)

        all_importances = np.mean(all_importances,
                                  axis=0, dtype=np.float64)
        return all_importances / np.sum(all_importances)

As you can see it returns the average feature importance across all tress in the forest and thereby uses the tree class. The tree class implements this as following:

def feature_importances_(self):
        """Return the feature importances.
        The importance of a feature is computed as the (normalized) total
        reduction of the criterion brought by that feature.
        It is also known as the Gini importance.
        Returns
        -------
        feature_importances_ : ndarray of shape (n_features,)
            Normalized total reduction of criteria by feature
            (Gini importance).
        """
        check_is_fitted(self)

        return self.tree_.compute_feature_importances()

And here compute_feature_importances is defined:

cpdef compute_feature_importances(self, normalize=True):
        """Computes the importance of each feature (aka variable)."""
        cdef Node* left
        cdef Node* right
        cdef Node* nodes = self.nodes
        cdef Node* node = nodes
        cdef Node* end_node = node + self.node_count

        cdef double normalizer = 0.

        cdef np.ndarray[np.float64_t, ndim=1] importances
        importances = np.zeros((self.n_features,))
        cdef DOUBLE_t* importance_data = <DOUBLE_t*>importances.data

        with nogil:
            while node != end_node:
                if node.left_child != _TREE_LEAF:
                    # ... and node.right_child != _TREE_LEAF:
                    left = &nodes[node.left_child]
                    right = &nodes[node.right_child]

                    importance_data[node.feature] += (
                        node.weighted_n_node_samples * node.impurity -
                        left.weighted_n_node_samples * left.impurity -
                        right.weighted_n_node_samples * right.impurity)
                node += 1

        importances /= nodes[0].weighted_n_node_samples

        if normalize:
            normalizer = np.sum(importances)

            if normalizer > 0.0:
                # Avoid dividing by zero (e.g., when root is pure)
                importances /= normalizer

        return importances


What are the differences between SVC, NuSVC, and LinearSVC?

In scikit SVC and nuSVC are mathematically equivalent with both methods based on the library libsvm. The main difference is that SVC uses the parameter C while nuSVC uses the parameter nu.

LinearSVC is based on the library liblinear. As the documentation says, LinearSVC is similar to SVC with parameter kernel='linear', but liblinear offers more penalties and loss functions in order to scale better with large numbers of samples. Please check out this question and this question for more details.


Not able to restore attention model properly

you have to use entire things like ckpt.restore("./tf_ckpts/ckpt-10"). please check the https://www.tensorflow.org/guide/checkpoint.


LSTM - How to prepare train from a dataset which contains multiple observations for different events

It's just several time-series with same data. You should make preprocessing (if it's neccessary) and can provide as an input just for one RNN layes.
for example

model.add(GRU(units=512,return_sequence=True,input_shape=(None,number_of_inputs))


Data Visualisation for Dataframe having 3 columns

use matplotlib:

import matplotlib.pyplot as plt
%matplotlib inline
f, (ax1, ax2) = plt.subplots(1, 2)
ax1.plt(Data_X_err.min, Data_X_err.err)
ax2.plt(Data_X_err.max, Data_X_err.err)

Alternatively you could use seaborn's pairplot

import seaborn as sns
%matplotlib inline
sns.set(style="ticks")

sns.pairplot(df)


sub-setting based on categorical variable in R

This question seemed to be more suitable for general stackoverflow since it's a R question.

I'm using data.table. The syntax is a little bit different from data.frame, but the time spent on learning data.table is well worth the effort. See more tutorials on wiki, and this good sheet.

library(data.table)
s = "rowname    group   marks   upd
2   C   3   up
3   C   4   down
6   C   7   up
1   T   2   up
4   T   5   up
5   T   6   down
7   T   1   middle
8   T   0   down"
# fread is the fastest csv reader
a = fread(s)
# the i in data.table[i,j] subset rows, empty j select all columns
upd_subset = a[upd %in% c('down','middle'),]

> upd_subset
   rowname group marks    upd
1:       3     C     4   down
2:       5     T     6   down
3:       7     T     1 middle
4:       8     T     0   down

> upd_subset_bygroups = split(upd_subset, upd_subset$group)
> upd_subset_bygroups
$C
   rowname group marks  upd
1:       3     C     4 down

$T
   rowname group marks    upd
1:       5     T     6   down
2:       7     T     1 middle
3:       8     T     0   down


Gradient Descent does not always converge to Global minima. It only converges if function is convex and learning rate is appropriate.

For most real life problems, function will have local minimums and we need to run training multiple times. One of the reason is to avoid local minima.


How to implement a POS tagger using Word co-occurence and Clustering concepts?

First, you are right that word co-occurrence graphs have been used in various applications of NLP. More precisely in applications related to the meaning of the words, typically for topic modelling or word sense induction. These applications follow the linguistic principle that words which occur in similar contexts tend to have a similar meaning. This is the basis of distributional semantics.
However this principle is unlikely to help finding words which have a similar syntactic function, simply because the co-occurrences in a sentence don't correspond to their syntactic role: a sentence usually contains many distinct POS tags, it doesn't have a specific POS category. Basically, word co-occurrences are useful for semantics, not syntax.
If the goal is to implement a POS tagger, why not use state of the art methods known to work very well for POS tagging? There is a lot of training data available, for instance the Universal Dependencies corpora.
Note that POS tagging is normally a supervised task (based on annotated data), whereas your idea appears to be about finding groups of similar POS in an unsupervised way. This would be another story entirely, probably involving grammatical inference.


What is word embedding and character embedding ? Why words are represented in vector with huge size?

Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words.

  Semantic understanding enables a system to say that,  "I am happy" and "It's joyful", have the same meaning.

To incorporate this feature to a system, we present words of a particular language in form of vectors. Often called as embeddings, they help in establishing similarities between words and phrases.

  For instance, a vector representing the word "happy" will lie in the vicinity of the vectors representing the words "joy", "pleasure", "sad" etc. These vectors are high dimensional but using PCA or other dimensionality reduction techniques they are brought down to 3 dimensions where they could be visualized.

That's why we encode words in the form of vectors. We often use cosine similarity to determine the closest vector to a given vector in analysing sematic similarity.

  For an intuition, the 3D space which contains vectors for all possible English words could be thought of our knowledge base. We tend to keep similar words together in our mind. If we are talking about fast food, for instance, our brain would capture the region of the knowledge base to retreive words related with fast food like "burgers", "chicken" etc.


In my understanding, when you are fine-tuning for any task you use additional data (not used during pre-training) and those examples will change the weights on lower levels so that your model is better prepared for the context in which you will use it. A good example is a Twitter sentiment classifier.


My answer is that I would just normalize 0 = 00:00:00, 1 = 23:59:59+1sec by counting seconds and divide by the number of seconds in a day.

I disagree with the sine/cosine transformation, it might treat 11:30 and 12:30 the same if you pick the wrong transformation.

Also, there might be patterns like 'last minute' or 'early rise' that do not correspond to the cyclic nature of the day.

It should be the responsibility of the machine learning algorithm to detect which transformation should be made, e.g. if your machine learning algorithm would be a neural network, it should/could have nodes with activation functions that may look like sine/cosine, amongst others. Those would detect and respond to such cyclic behaviors.

The normalized time data is also easy to pick up by tree based/svm algorithms. Maybe even easier than 'Hour' as category, e.g. if you have a shop that is open from 9:30 to 17:30, the hour category "9" would be ambiguous.


Assigning new word by creating a new column when finding match

You are not getting the results you're expecting because you're explicitly comparing the value in the Answer column to 'Coca-Cola' (i.e x == 'Coca-Cola') but the value in that column is (I'm assuming) a list or a string. In either case comparing it to 'Coca-Cola' would result in False.
What you need to do is change your condition to check in 'Coca-Cola' appears anywhere in the list or string.
Here's one way of achieving this (I'm assuming the Answer column contains lists):

import pandas as pd

data = {
          "ID": ["101005", "026458", "045987"],
          "Question": ["what brands did you purchase the past 5 months",
                       "what brands did you purchase the past 5 months",
                       "what brands did you purchase the past 5 months "],
          "Answer": [["Coca-Cola" ,"Pepsi" ,"fanta"], [None], ["Coca-Cola"]]
}
deux = pd.DataFrame(data)
deux['Buyer_Type'] = deux['Answer'].apply(lambda x: 'existing buyer' if 'Coca-Cola' in x else 'new buyer')


How to change the names of the layers of deep learning in Keras?


Item Based Collaborative Filtering with No Ratings

There might be different ways to do that, like considering implicit ratings like views or clicks.

But basically, you can consider a rating of 1.0 for each user-item pair you have.

This way, your prediction will be between 0 and 1 which you can consider similar to a click prediction probability.

0 being the probability on not having a click or the item by a user
1 being the probability that a click would be made.

You can also set a threshold where you can drop a recommendation at a certain limit of your prediction level.

E.g. Let's say an user alpha has the following recommendations with a threshold of 0.7

i1 , 0.87
i2, 0,75
i3, 0,6

You can drop the i3 recommendation since you don't consider it good enough.

Nevertheless, the parameter of your recommendation engine must be determined of course with the help of your evaluation metrics.

Considering software solutions, I use Apache Spark MLlib with Scala as a base for my recommendation engine algorithms where you can compute item cosine similarity easily per example where you are using in-house implementation or an approximation with the DIMSUM algorithm.

I hope this helps!


Replace Missing Values with Most Frequent number under Condition

This solution works well but I don't know why above don't work!
Imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
for i in range(2):
    for j in range(1,4):
        ls = np.array(df.Age[((df.Sex==i) & (df.Pclass==j))]).reshape(-1,1)
        df.Age[((df.Sex==i) & (df.Pclass==j))] = Imputer.fit_transform(ls)[:,0]
df.Age.isnull().sum()


Bias simply takes care of variables which are unaccounted for i.e you did not include it in X. All Machine Learning and Neural Nets do is approximate a function. All functions can be represented as a series of exponentials using Fourier Transform which basically means a function can be expressed as a sum of many other special functions.

In this series there is a constant term to adjust the elevation of the curve above the x-axis. This constant can be thought of as the bias. Thus it basically takes care of the fact that when all variables are 0, a function need not be 0.


GridSearch returns worse results than default configuration

Technically: Because grid search creates subsamples of the data repeatedly. That means the SVC is trained on 80% of x_train in each iteration and the results are the mean of predictions on the other 20%.

Theoretically: Because you conflate the questions of hyperparameter tuning (selection) and model performance estimation. GridSearch is used for selecting a combination of hyperparameters, performance estimation has not yet happened. The only comparison you should be making is between the parameter combinations within the CV itself (grid_results.cv_results). In my opinion, the reported CV train accuracy is within acceptable boundaries from non-CV training (meaning your SVC is able to extract a lot of generalization from subsamples). See e.g Cawley 2010

It would be interesting to see the reported performance on x_test on the retrained estimator grid_results.best_estimator_.predict(x_test), if it was kept separate from x_train, and how it is different from the first results.


How would you build a big production ready image training dataset from scratch?

I'm not expert in images classification, I'm just going to give some general advice here.
The strategy should be progressive, for instance:

Proof of concept: devise a first draft of annotation process with some general guidelines, then take a random subset of a few hundreds images and try annotating them following the guidelines. Observe carefully the problems found at the annotation stage: any ambiguous cases, any problems with the granularity of the classes (for instance a class which is too general), potential error cases. Devise a training/testing setup to train a mockup system, test it with the small dataset, fix any bugs in the process, possibly try different methods. The goal of this stage is to iron out the different stages, possibly rule out some options which turn out to be unworkable.
Prototype: have a team of a few laypeople annotate a few thousands images following the updated annotation guidelines. Again pay attention to any problem, especially human errors in the process. At least a subset of the images should be annotated by several annotators, in order to detect differences. Ask annotators for feedback: were there any difficulties, was it fun, would they do it as a game or not, etc. At this stage you can start having a real ML system, albeit not optimal yet. Analyze the performance of the system on the different classes, possibly try different methods, estimate the minimum number of images per class to obtain decent performance. At this stage a lot of other tests could be done, and it should become clear how accurate the manual annotation is and whether annotators need to be compensated or not.

It's only around this stage that the final annotation process can be fully designed. Depending on the strategy, you could consider devising a strategy for iterative manual annotation: some classes are going to be quickly learned by the model, so it could make sense to use the model in order to propose for annotation at the next round the images for which the model fails. Be careful to avoid bias, keep evaluating and refining the model at every round of annotations.


Which book is a standard for introduction to genetic algorithms?

Most of the "standard textbooks" (e.g., Goldberg, Mitchell, etc.) are pretty dated now. If you just want to have some confidence that you understand how the basic algorithms work, they're fine, but they tend to emphasize material that's doesn't necessarily match the more modern way of understanding and talking about things like theoretical issues.

I've used the Eiben and Smith (https://www.amazon.com/Introduction-Evolutionary-Computing-Natural/dp/3642072852) book to teach evolutionary algorithms to my students over the past few years. It's a bit more concise than some of the others, but if you're OK with that, I think it does a better job of covering the stuff that's relevant about evolutionary computation today.


Convolution and Pooling as Infinitely strong priors

A prior distribution expresses your assumptions about the model without observing any data. E.g. when doing linear regression, you a priori assume that the slope is close to zero. Now you start measureing data points and it turns out that the slope should probably be close to one, so you compromise and pick a value somewhat inbetween. If your belief in your prior assumption of zero slope is weak, it does not take much data to convince you to pick a slope closer to one. If your belief is strong, you will pick a slope close to zero anyway and demand to see many data points until you move it slowly towards one.

In case of regression the strength of your belief would be parameterized by the weight of the regularization term in the optimization objective. The connection between regularization and priors is the same for neural networks (see here for a little more detail).

So you can think of the strength of the prior as a measure for how much evidence you would need to see until you deviate from your prior assumptions about the model.

When you train a CNN, you assume a priori that this network structure is the best for your problem, i.e. that the model should internally compute convolutions. Since the CNN structure is build into the model, no amount of data will convince you to abandon this structure in favour of e.g. a fully-connected NN. Hence the prior is infintely strong.

The argument for pooling etc. would be similar. I would even claim that most if not all non-trainable parameters can be seen as an infinitely strong prior.


Decision Trees change result at every run, how can I trust of my results?

how can I trust of my results?

You probably shouldn't trust your results, because the large variation is likely caused by overfitting. Basically your model is not very reliable.

My guess is that you have either too many features or not enough instances, or any combination of these two issues.

  I am not sure If it's correct to take the best result on the test set, or maybe I should consider an average of the results

It is definitely not correct to take the best result. The average result is more representative of the true performance, but you should also provide the standard deviation.

  Second, would not it be better to use CV on all data, instead of just splitting at the beginning the dataset in test and train just one time?

Yes, that's a good idea: as far as I understand, currently you're just manually running the program several times and obtaining different performance each time. You can indeed use Cross-validation instead.

  I read about random_state but the problem is that the results depend on which value I use.. example random_state = 2 --> accuracy 0.6 (on test set) random_state = 6 --> accuracy 0.79 (on test set) so basically this does not resolve my problem. How can I validate my model if I dont know which one to use?

That's normal: the split function separates the training and test set randomly according to a random sequence. Setting the random seed to a particular value guarantees the exact same sequence every time, but you don't want that since it makes the splitting non-random.

Instead you should work on reducing the variation. Usually it's not possible to add instances, so you should probably try to reduce the number of features, or simplify them.


How to match up categorical labels in training and evaluation

You are going about it all wrong. You shouldn't transform your labels into numerical values and match them to the output; that's not how classification is done. Instead, for each class (category) you will add a node in the output layer; the predicted class is then the argmax of the output layer.

For example, let's say you have three classes: simple, compound and complex. We'll assign an integer for each class starting from $0$.
Your output layer will have three nodes, one for each class, and for each training example $x$, the true label $y$ is a three dimensional "hot-encoded" vector. (i.e $y = (y_0, y_1, y_2)$ where $y_i = 1$ if $x$ belongs to class $i$ otherwise $y_i = 0$ for $i \in \{0,1,2\}$).

Now, let's suppose you feed your (trained) network an input value x and it outputs the following vector:
$y' = (0.4, 1.4, 2.5)$. The predicted label in this case is $\arg\max(y') = 2$. So the prediction for the sentence x is: complex.

Note: since argmax is not differentiable, we use softmax during training to convert the output vector y' into a vector of probabilities that sums up to $1$ before applying the loss function. The prediction in this case is the class with the highest probability. For example, if we apply softmax to the same y' from the example above, we get: $\text{softmax}(y') = (0.08, 0.23, 0.69)$. It's clear that the highest probability ($69\%$) belongs to the class $2$ (complex) and that these probabilities sum up to $1$.


Reason behind the sum of rate factors for calculating cost function derivative

First, disclaimer: you're presenting fairly hairy computations and it's a little hard to read your question. I'll try answering nonetheless.

i don't understand exactly how can we calculate that slope of the cost with respect to that weight by summing [...]

If I interpreted the sums correctly then the sums in your question seem to deal with the Jacobian matrix that is used when applying the chain rule in multiple dimensions.
Intuitively, the weight $w_{j,k}^{l}$ will be referenced by all the paths originating from the neuron utilizing $w_{j,k}^l$, and will then be expressed by nodes in the "final" expression of $C$ via multiple linear combinations (sums). I'm fairly certain that if you actually followed the gradient $\partial C\over\partial w_{j,k}^l$ starting at the sum $C=a^L_{1}w_{2,k}^L+a^L_{2}w_{2,k}^L$ it would work out to be the sum you show. This also, I believe, would apply to the second sum you present.


How much data is created daily (source of IBM estimation)

The McKinsey Global Institute published an interesting report on the topic of Big Data.
You can find it here.

I don’t know where IBM go the numbers from, but this report is a good source.


Approaches to A/B testing when you can't randomize on the user level

I would recommend against switch back as the purchase data is likely to exhibit seasonality (e.g. 2pm - 6pm / tuesdays exhibit lower volume, December exhibit higher volumes, company running advertisements during certain months etc.), which would become an unnecessary exogenous factor that complicates the post implementation analysis.

I am inclined towards the causal impact approach. However, there are certain things that should be checked before selecting the stores as control vs. test (it is a kick-start list, please feel free to adapt to your business domain).

Decide on the KPI to be measured as the objective, is that revenue per customer? Profit per customer? Or total profit per store? Is the algorithm designed to increase per customer KPI or more customers?
Check that the stores selected have some history to give assurance that they were similar in terms of the selected KPI before the implementation of the tests. Remember to remove outlier customers / transactions.
Check that enough stores are selected in control vs. test buckets to detect a desired threshold increase that is statistically significant.

Hope this helps! :)


Should I prevent individuals mating with themselves in EAs?

When talking about Evolutionary Algorithms, you will have to decide between diversity or specialization of the possible solutions.

You should not let individuals crossover themselves because you will be loosing diversity in your algorithm.

But, there is the exception to this rule: You need to make sure you are not punishing the success of this individual if he is the best, you could make sure this individual passes to the next generation by preserving the best parents.

The answer to your question is: Prevent auto-crossover but make sure you don't loose this solution when selecting the ones passing to the next generation (Elite group, passing n of the best parents)

In tournament selection, is not common to select the same individual as a participant of the tournament twice, so this possibility should not be one of your worries.

In roulette selection, be sure you don't eliminate successful individuals.


Sklearn classification report is not printing the micro avg score for multi class classification model

The micro-averaged precision, recall, and F1-score are all the same, and are all equal to the overall accuracy (!) when including all the classes.

That fact is hinted at in the documentation for classification_report, in the "Return" field ("it is only shown for multi-label or multi-class with a subset of classes because it is accuracy otherwise").  It's explained in a little more detail at https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1 (near the end).


How to quantify a tokenized user agent string for a neural network?

iOS, Windows and Android should not be viewed as strings but as categories. Categories should be one-hot encoded, and should not share the same input as they are not related to each other.

So let's assume we work with two operating systems: iOS and Windows.

So for iOS, input is: 1 0 and for Windows input is: 0 1.

But now we want to add another OS: Android.

iOS becomes: 1 0 0, Windows 0 1 0, Android: 0 0 1. Note that by adding android, you do not actually change the input! You only add an extra 0 for iOS and Windows, and 0's don't get computed in neural networks 0*weight = 0.

So by adding an OS, you do not affect what the network has learned for the previously trained OS's.


For the first question, it is important to recall that RMSE has the same unit as the dependent variable. It means that there is no absolute good or bad threshold, however you can define it based on your DV. For a datum which ranges from 0 to 1000, an RMSE of 0.7 is small, but if the range goes from 0 to 1, it is not small.

I would do some feature engineering (Create more variables: Time of Day, Day of Week, Month etc...) and run it through a Neural Network and then check for accuracy.  You may want to check if there is any correlation between the 4 junctions as well but if you run a NN, you don't have to.


What is the hypothesis space used by this AND gate Perceptron?

As far as I understand:

A hypothesis is a model which is capable of predicting outputs from inputs, hence the $x_1 + x_2 - 1.5$ is a hypothesis but not the only one. The whole models that have the same capability are regarded as hypothesis space.

We know that in AND gate:

     x1       x2     output
|---------|--------|--------|
|     0   |    0   |    0   |
|     0   |    1   |    0   |
|     1   |    0   |    0   |
|     1   |    1   |    1   |
|---------------------------|

and we have $w .\cdot x + b$, based on which, either $0$ or $1$ turns out as output.
$$w \cdot x + b$$
$$w_1 \cdot x_1 + w_2 \cdot x_2 + b$$

trying all the inputs in this expression:

$w_1 \cdot 0 + w_2 \cdot 0 + b <= 0$ (because the output should be 0)  $b < 0$

$w_1 \cdot 0 + w_2 \cdot 1 + b <= 0$ ---> $ w_2 + b <= 0$  So $w_1 < |b|$

$w_1 \cdot 1 + w_2 \cdot 0 + b <= 0$ ---> $ w_1 + b <= 0$  So $w_2 < |b|$

$w_1 \cdot 1 + w_2 \cdot 1 + b > 0$ ---> $ w_1 + w_2 + b > 0$

Firstly, we initialize the weights and bias parameters and then if needed, change them.

Here, since $b < 0$ we set it as $-1$

Since $w_1 < |b|$,  $w_2 < |b|$ and weights are not negative, we set them as 1.
So we would have:

$w_1 \cdot 0 + w_2 \cdot 0 + b = -1 < 0$ is right, returns 0 because is negative.

$w_1 \cdot 0 + w_2 \cdot 1 + b = w_2 + b = 1 - 1 = 0$  wrong, it returns 1 while it should return 0

$w_1 \cdot 1 + w_2 \cdot 0 + b <= 0$ ---> $ w_1 + b <= 0$  So $w_2 < |b|$ wrong, it returns 1 while it should return 0

$w_1 \cdot 1 + w_2 \cdot 1 + b > 0$ ---> $ w_1 + w_2 + b > 0$

So we set b for a smaller value like -1.5 (Note), then all the expressions would work appropriately. Hence $x_1 + x_2 - 1.5$ is a hypothesis for this problem.

Note: we know that in using perceptron algorithm, when reaching at any point that is not following the current model, weights and bias are updated as follows:

w = w + yx
b = b + y

Here, in the source that you referred to, maybe for simplicity they haven't done so and have just found a sample of a plausible model (hypothesis)

The other hypothesizes should follow the previously mentioned rules, thereby, $x_1 + x_2 - 2$ can also be another hypothesis for this problem, etc.


What is the benefit of using Max pooling in convnets as opposed to just using convolution layers? (from Francois Chollet's Deep Learning with Python)

Max pooling layers are useful for two things:

undersampling the dimensions of their input, to reduce model parameters and also reduce overfitting
increasing the robustness of the model to small changes in the input image (e.g. if I change the input image slightly the model should predict the same thing).

However their usefulness is limited and are starting to be replaced by convolution layers with large strides. You could read this article on the matter.


You said:

  All the algorithms I've checked so far e.g. n-grams variants, bag of
  words etc. are only used for clustering strings that are part of a
  context but here is not the case.

However, those algorithms can still be used if you don't treat tokens as full words, but as combinations of characters instead.

Here is an example of such algorithm in Python using Sklearn.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans

data = [
    'alex@examplemail.com',
    'bernard@examplemail.com',
    'chris@newmail.com',
    'dorothy@nemail.com',
    'emily@examplemail.com'
]

X = TfidfVectorizer(analyzer='char').fit_transform(data)

kmeans = KMeans(n_clusters=2).fit(X)

for cluster in set(kmeans.labels_):
    print('\nCluster: {}'.format(cluster))
    for i, label in enumerate(kmeans.labels_):
        if label == cluster:
            print(data[i])

> Cluster: 0
> alex@examplemail.com
> bernard@examplemail.com
> emily@examplemail.com
>
> Cluster: 1
> chris@newmail.com
> dorothy@nemail.com

The major trick is TfidfVectorizer(analyzer='char'), where you don't treat n-grams of words, but of characters instead.

P.S. If you at some point want to use word n-grams (if you have generally well written sentences) you can use TfidfVectorizer(analyzer='word').


Why would Pytorch be non-deterministic for batch only?

The cudnn implementaton of the LSTM has determinism issues that appear to be fixed in the 7.6.1 release. Check your cudnn version.

https://github.com/pytorch/pytorch/issues/18110


The developers docs for torch specify that the arguments can be tensors, so you can just pass the tensors directly, but you need to reshape your output so that j is the last index. First, rewrite your code so that j is the last index and in your for-loop you have

l1 = torch.autograd.grad(output[0][0][j], ...)

And then drop the for-loop and remove the j-indices.


Is adding the embedded words of a sentence to represent the sentence a good approach?

So the question asks how to represent a series of words a uniform vector representation, which is not dependent on sequence.
The idea you suggested is definitely not a bad idea, you should try that out.
You should also try Doc2Vec, which works on the same principle as Word2Vec but this time it will output a vector which represents the meaning of a section of text that is longer than 1 word.
The main problem with this sort of representation is that you lose sequential information of the text and you treat words in sentences as a “bag of words”. If you happy to make that assumption, then continue using your approach of Doc2Vec.
Otherwise, you might better off using a sequential model architecture, such as RNN/LSTM, etc. Here you can input each word at a given time step initially as a one hot encoded vector and then you add an embedding layer before it goes into the sequential model to transform the one hot encoding into a word embedding.


How to handle a regression problem with skewed target and only few high values?

You can generate symmetrical distribution(s) by suitable transformation:
Your distribution is mainly right-skewed, therefore a log10 transformation is required.
You can also use an auto-binning method and combine it with dummy variables for the spikes.
If you then perform a feature selection, the learner will automatically choose the most significant features. Learners like XGBoost automatically take care of multicollinearity.


Intuition behind the fact that SVM uses only measure of similarity between examples for classification

You can try running SVM just on this similarity matrix. But you'll then need to provide the sikikaritirs also for new data points.

Furthermore, SVMs rely on the similarities bring dot products in some vector space. If they aren't you may get inconsistencies.
They may rely on the triangle inequality being satisfied for the distance function d(x,y)=sqrt(2k(x,y)-k(x,x)-k(y,y)). Although I cannot find a clear reference on whether or not this is needed. If k is a scalar product in some vector space, this obviously is satisfied.

Last but not least, SVMs are good for larger amounts of data, where you cannot afford to keep the entire similarity matrix in memory! By reducing the data set to the support vectors only, the resulting classifier will need much less memory and much less time. Much of the challenge of learning a SVM is to manage memory during training.


Forecasting time series: Method Selection

You can try two different approaches:

1) Kalman filter, the method is battle-tested and has proven useful in many areas.

Resources:

Understanding the Basis of the Kalman Filter
kalman filter in pictures

2) Recurrent Neural Networks, the LSTM and GRU architectures are particularly interesting for time series predictions.

Resources:

RNN effectiveness
Understanding LSTMs

To do regression and predict future data points, you would need to build a training dataset consisting of a sequence of events. Let's say a value $x$ for every timestamp $t$.

Your data seems to have 1 dimension, so both the network input layer and the output layer would consist of 1 unit. You would then train your model to predict $(x_{t+1})$ given $(x_{t})$.

Let $M$ be our trained model and let's say you want to forecast a data point at time $k$ and you know the current value at time $t$.

$M(x_{t}) = (x_{t+1})$

$t = t+1$

$M(x_{t}) = (x_{t+1})$

$...$ increment $t$ and keep predicting until $t+1 = k-1$

$M(x_{t+1}) = (x_{k})$

Put in pictures this corresponds to:

(picture from Udacity lecture about Deep Learning)


high root mean square error in regression model

RMSE doesn't work that way. An RMSE of 13 might actually great, it completely depends on how your target variable is scaled. For example, if your target variable was in the range [0,1e9], than an RMSE of 13 is spectacular. On the other hand, if your target is in the range [0,1], an RMSE of 0.5 is terrible. If you want to try a metric that can be more readily interpretable as having a "good" or "bad" score, try Mean Average Percent Error (MAPE).

As far as why you get a lower MSE when you cross validate: you don't show us how you constructed your training and test sets, but my guess is that you basically just got unlucky and ended up with a training/test split that performed poorly on your holdout set. Your CV-MSE is clearly better than your single holdout MSE, but you should also check the spread of CV scores as well. In any event, for a dataset as small as yours I'd recommend using bootstrap cross validation instead of k-fold.


I think that before anything else it's important to explore the data to identify features that aren't bringing anything worthwhile (v. low variance, duplicates, collinearity) and to see if you've got any issues with missing values/outliers in your numeric data.

With that said, I think you'd want to keep your numeric variables as such (initially at least), but how many do you have? You may wish to normalise them so that the scale of one variable doesn't dominate (eg. is income included?). Or if you have time, try both? Look at the distribution and see what bins might be appropriate to discriminate by:
https://en.wikipedia.org/wiki/Normalization_(statistics)

Ordinal data is tricky; in your case you can't have 1.5 beds, but that's what you might end up with if you treat them as numeric. I would initially run with one hot encoding, and when you get some results you can look at the impact of your features to decide if you want to keep them that way:
https://www.quora.com/What-is-one-hot-encoding-and-when-is-it-used-in-data-science

Without knowing the full range of your grade values or how granular/continuous your prediction needs to be, you could easily do either - I'd say think about what the ideal outcome would be and build towards that. If you've got time try both, try different bins for a multi-class classifier; but if you have a massive range of target outputs then regression is probably preferable.

Finally - start with something simple and interpretable, and build from there. If you run with classification, can a decision tree deal with your dataset, and how does it do/where does it fall down? You might get lucky, but if not you can try an SVM or random forest. Given the number of variables you might want to consider some dimensionality reduction first; you might get some good results from clustering.

Hope this helps!