Imbalanced dataset: how to deal with test data?

You should use the testing set without any change, as answered by others. But it is very important to understand the difference between average accuracy and overall accuracy.
In overall accuracy you find ( number of samples predicted correctly/ total number of samples) in average accuracy, you find the overall accuracy per class and then you find the average of these overall accuracies.
When you know that you are working with imbalanced database, where all classes are important, you should use the average accuracy

To understand what  this means: imagine you have two classes, class A and class B , and the ratio is 90 to 10 .
If you are sampling randomly for the training and testing, then the ratio is still 90:10 in the testing set. If your model is very biased , that predicts all the samples to be class A , then:
Overall accuracy = 90%
Average accuracy = 50 %
( 100% for class A + 0% for class B) / 2

The overall accuracy is really high but it does not reflect the actual quality of the model. The average accuracy gives you a better indication of the quality


Based on the current details about your question, I'm not sure what you seek, but I will give it a try.

I assumed you already tried Pearson's coefficient. There are other metrics, which are more or less difficult to implement if not already existing in matlab.

Spearman's rank
Coefficient of determination
Hilbert-Schmidt independence criterion

All those will provide some insight about dependencies within your dataset.


Good question, probably a good handful of ways to look at this as it sounds like a classic classification problem.

Define the label

First step, before you choose a model, would be to define the label. As you suggested, I would go for email opened = 1 and email not opened = 0. I would then think about throwing the email not sent (-1) data away, as I'm not sure you can learn anything about a probability of opening when it wasn't sent.

It would then be important to check the balance of the label: are they mostly opened or mostly ignored? This will influence any decision to add a class weight if it is imbalanced towards one class.

Define features

So time series could work, but I think another good angle might be to bin the send times into half hourly slots (or shorter, depending on your needs) and treat them as a categorical variable. You would then one-hot encode this feature. Anything else you can add to this will probably help, such as the kind of person it was sent to and what country they're in etc.

Choose model

So it's classification, so have a play around with either logistic regression or something tree based (random forests, gradient boosted trees) to get something going quickly. It's usually a case of trial and error to find the best one.

You can then use traditional metrics of precision/recall/accuracy to measure how good it is. Make sure you know which metric is most important to your problem when optimising the model.


Why do cost functions use the square error?

Your loss function would not work because it incentivizes setting $\theta_1$ to any finite value and $\theta_0$ to $-\infty$.

Let's call $r(x,y)=\frac{1}{m}\sum_{i=1}^m {h_\theta\left(x^{(i)}\right)} -y$ the residual for $h$.

Your goal is to make $r$ as close to zero as possible, not just minimize it. A high negative value is just as bad as a high positive value.

EDIT: You can counter this by artificially limiting the parameter space $\mathbf{\Theta} $(e.g. you want $|\theta_0| < 10$). In this case, the optimal parameters would lie on certain points on the boundary of the parameter space. See https://math.stackexchange.com/q/896388/12467. This is not what you want.

Why do we use the square loss

The squared error forces $h(x)$ and $y$ to match. It's minimized at $u=v$, if possible, and is always $\ge 0$, because it's a square of the real number $u-v$.

$|u-v|$ would also work for the above purpose, as would $(u-v)^{2n}$, with $n$ some positive integer. The first of these is actually used (it's called the $\ell_1$ loss; you might also come across the $\ell_2$ loss, which is another name for squared error).

So, why is the squared loss better than these? This is a deep question related to the link between Frequentist and Bayesian inference. In short, the squared error relates to Gaussian Noise.

If your data does not fit all points exactly, i.e. $h(x)-y$ is not zero for some point no matter what $\theta$ you choose (as will always happen in practice), that might be because of noise. In any complex system there will be many small independent causes for the difference between your model $h$ and reality $y$: measurement error, environmental factors etc. By the Central Limit Theorem(CLT), the total noise would be distributed Normally, i.e. according to the Gaussian distribution. We want to pick the best fit $\theta$ taking this noise distribution into account. Assume $R = h(X)-Y$, the part of $\mathbf{y}$ that your model cannot explain, follows the Gaussian distribution $\mathcal{N}(\mu,\sigma)$. We're using capitals because we're talking about random variables now.

The Gaussian distribution has two parameters, mean $\mu = \mathbb{E}[R] = \frac{1}{m} \sum_i h_\theta(X^{(i)})-Y^{(i))}$ and variance $\sigma^2 = E[R^2] = \frac{1}{m} \sum_i \left(h_\theta(X^{(i)})-Y^{(i))}\right)^2$. See here to understand these terms better.

Consider $\mu$, it is the systematic error of our measurements. Use $h'(x) = h(x) - \mu$ to correct for systematic error, so that $\mu' = \mathbb{E}[R']=0$ (exercise for the reader). Nothing else to do here.
$\sigma$ represents the random error, also called noise. Once we've taken care of the systematic noise component as in the previous point, the best predictor is obtained when $\sigma^2 = \frac{1}{m} \sum_i \left(h_\theta(X^{(i)})-Y^{(i))}\right)^2$ is minimized. Put another way, the best predictor is the one with the tightest distribution (smallest variance) around the predicted value, i.e. smallest variance. Minimizing the the least squared loss is the same thing as minimizing the variance! That explains why the least squared loss works for a wide range of problems. The underlying noise is very often Gaussian, because of the CLT, and minimizing the squared error turns out to be the right thing to do!

To simultaneously take both the mean and variance into account, we include a bias term in our classifier (to handle systematic error $\mu$), then minimize the square loss.

Followup questions:

Least squares loss = Gaussian error. Does every other loss function also correspond to some noise distribution? Yes. For example, the $\ell_1$ loss (minimizing absolute value instead of squared error) corresponds to the Laplace distribution (Look at the formula for the PDF in the infobox -- it's just the Gaussian with $|x-\mu|$ instead of $(x-\mu)^2$). A popular loss for probability distributions is the KL-divergence.
-The Gaussian distribution is very well motivated because of the Central Limit Theorem, which we discussed earlier. When is the Laplace distribution the right noise model? There are some circumstances where it comes about naturally, but it's more commonly as a regularizer to enforce sparsity: the $\ell_1$ loss is the least convex among all convex losses.

As Jan mentions in the comments, the minimizer of squared deviations is the mean and the minimizer of the sum of absolute deviations is the median. Why would we want to find the median of the residuals instead of the mean? Unlike the mean, the median isn't thrown off by one very large outlier. So, the $\ell_1$ loss is used for increased robustness. Sometimes a combination of the two is used.

Are there situations where we minimize both the Mean and Variance? Yes. Look up Bias-Variance Trade-off. Here, we are looking at a set of classifiers $h_\theta \in H$ and asking which among them is best. If we ask which set of classifiers is the best for a problem, minimizing both the bias and variance becomes important. It turns out that there is always a trade-off between them and we use regularization to achieve a compromise.

Regarding the $\frac{1}{2}$ term

The 1/2 does not matter and actually, neither does the $m$ - they're both constants. The optimal value of $\theta$ would remain the same in both cases.

The expression for the gradient becomes prettier with the $\frac{1}{2}$, because the 2 from the square term cancels out.

When writing code or algorithms, we're usually concerned more with the gradient, so it helps to keep it concise. You can check progress just by checking the norm of the gradient. The loss function itself is sometimes omitted from code because it is used only for validation of the final answer.

The $m$ is useful if you solve this problem with gradient descent. Then your gradient becomes the average of $m$ terms instead of a sum, so its' scale does not change when you add more data points.

I've run into this problem before: I test code with a small number of points and it works fine, but when you test it with the entire dataset there is loss of precision and sometimes over/under-flows, i.e. your gradient becomes nan or inf. To avoid that, just normalize w.r.t. number of data points.

These aesthetic decisions are used here to maintain consistency with future equations where you'll add regularization terms. If you include the $m$, the regularization parameter $\lambda$ will not depend on the dataset size $m$ and it will be more interpretable across problems.


Problem encountered while updating a particular cell of a pandas data frame in Python

Use a combination of string processing and type casting to process the whole column in one swoop:

  excelDF[datecol] = pandas.to_datetime(excelDF[datecol].str.replace('*',''))

I tried using str.rstrip('*') but I could not get it to work.


A random subset of features than using the best split logic as done in normal Tree

The Random Forest algorithm introduces extra randomness when growing trees; instead of searching for the very best feature when splitting a node, it searches for the best feature among a random subset of features. The algorithm results in greater tree diversity, which (again) trades a higher bias for a lower variance, generally yielding an overall better model.

$\hspace{3cm}$Image credit - SE( https://stats.stackexchange.com/a/438384/256691) - Ashish Anand
Extra-trees -
More randomness by getting the feature using a random Threshold instead of searching the best split.

When you are growing a tree in a Random Forest, at each node only a random subset of the features is considered for splitting. It is possible to make trees even more random by also using random thresholds for each feature rather than searching for the best possible thresholds (like regular Decision Trees do)

I used Extra-tree to explain the point to have a contrast. It helps in catching the point.
Please also read the SE link of the image
[Blockquote ref] - Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition


First what comes to my mind: "the only statistics you can trust are the ones you have falsified yourself" - Churchill

However, to answer your question.

The correlation coefficient is defined as:

You can find data points that are negatively correlated by looking at the covariance, that is defined as:

First, you calculate the mean of your dependent and independent variable. Then you subtract the mean from the corresponding data points. After that, you multiply those two series. Negative values are rows that contribute to a negative correlation coefficient.

However, as a statistician, I am not supporting this approach (data deletion on purpose is not scientific working).


Turning Histogram values into Numerical format ( Excel-xslx, Pandas-DataFrame, etc.)

The data is available in json format through this link, which contains the data for all countries that can be selected from the list. I found this link by going to the link you provided,  opening the Network tab in the Chrome developer tools and reloading the page to see  all the resources that are loaded by the webpage.


What is the stochastic part in stochastic gradient descent?

You sample a random mini-batch as opposed to the full dataset. This means that you get a stochastic approximation of the true gradient of your loss function with regards to your dataset and weights.


You can just skip the Embedding layer and use a normal input layer with n input nodes where n is the dimensions of your word2vec embeddings. The rest is the same as you would with an embedding layer, just pass a sequence of n dimensional vectors as the input, potentially padded or truncated depending on your model.


Since it's a master's thesis, how about writing something regarding decision trees, and their "upgrades": boosting and Random Forests? And then integrate that with Map/Reduce, together with showing how to scale a Random Forest on Hadoop using M/R?


One uses a frequency or TFIDF representation in the features when the target directly depends on specific words. For example for spam classification words like "cheap, free, viagra, exclusive..." are direct indicators of the target label.
In your case the target doesn't directly depend on specific words, it depends whether the same words appear in both the customer and product descriptions. This is an indirect relationship and most regular ML algorithms cannot really deal with that. So your design is unlikely to work in my opinion.

Until now, I only got to build a similarity pairwise coefficient (using this), but there is no classification, and I know using labelled data can help. In particular, similarity measure gives the same weight to any text coincidence, but some coincidences should be more important than others.

This makes more sense for your purpose: use only the similarity score as a feature and train a model to predict the label. Technically the model will only learn the optimal threshold to separate the labels, so you can just use linear regression for instance. You could improve this method by calculating different types of similarity measures and provide all of them as features.
Note: if you use TFIDF vectors for measuring similarity, it doesn't give the same weight to every word. However don't expect perfect result: a lot depends on the data itself, are you sure that the customer description gives useful indications about the products they're interested in? For example if a customer description contains the word "computer" it doesn't mean they're interested in every possible type of computer.


I think the professor might have meant the closed-form solution of linear regression.


A quick look (this post, for example) suggests that to use SVM for image classification, you need to extract features beforehand, and not run it directly on the image data.


Can Reinforcement learning be applied for time series forecasting?

Yes, but in general it is not a good tool for the task, unless there is significant feedback between predictions and ongoing behaviour of the system.

To construct a reinforcement learning (RL) problem where it is worth using an RL prediction or control algorithm, then you need to identify some components:

An environment that be in one of many states that can be measured/observed in a sequence.
An agent that can observe current state and take actions in the same sequence.
The evolution of state in the sequence should depend on some combination of the current state and the action taken, and may also be stochastic.
There should be a reward signal that the RL agent can observe or  measure. The value of reward should depend on the same factors as the evolution of the state, but can depend on them in a different way.

The general case of time series forecasting can be made to fit with this by treating the prediction as the action, having the state evolution depend on only the current state (plus randomness) and the reward based on state and action. This will allow RL to be applied, but causality only flows one way - from the environment into your predictive model. As such, the best you can do for rewards for instance is to use some metric about the correctness of the predictions. Consequences for good or bad predictions do not affect the original environment. Essentially you will end up wrapping some predictive model for the sequence (such as a neural network) in a RL layer which could easily be replaced by basic data set handling for a supervised learning problem.

One way you could meaningfully extend series forecasting problems into RL problems is to increase the scope of the environment to include the decisions made based on the predictions, and the state of the systems that are affected by those decisions. For instance, if you are predicting stock prices, then include your portfolio and funds in the state. Likewise the actions stop being the predictions, becoming buy and sell commands. This will not improve the price prediction component (and you are likely better off treating that as a separate problem, using more appropriate tools - e.g. LSTM), but it will frame the issue overall as a RL problem.


Use LSTM to predict the proportion of steps with nonzero feature values

First, you have too little data. 50 samples. Think of it. Of all possible 2001 sequences of that kind, you are only feeding 50 (less than 2.5%). Your question is actually a very good case to show the importance of big data for training a neural network.

And second, it is the problem stated in this arxiv article made by Uber empolyees - even a deep neural network with nonlinear units often cannot generalize to solution of a simple linear problem.

Your problem still can be approximately solved with

Reducing batch size

It is well known that increasing batch size leads to faster but poorer convergence. Particularly in your case, you should use line:

model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1,
    batch_size=1,
    shuffle=True)

instead of

model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1)

even knowing that it will be slooooow...

Modifying LSTM layer: removing unit_forget_bias=True and adding orthogonal initializer with small gain - explained in this article:

from keras.initializers import orthogonal
model.add(LSTM(10, activation='tanh', #unit_forget_bias=True,
    kernel_initializer=orthogonal(gain=.01),
))

Increasing coverage of the space of all possible samples

In the code below SEQ_LEN=2000 and N_SAMPLES=50 is your case. Here we generate random sequences like ones that your code above (although not the same sequences as yours). Now, try experimenting with values SEQ_LEN=200 and N_SAMPLES=80 and then SEQ_LEN=200 and N_SAMPLES=130. See if with N_SAMPLES<SEQ_LEN/2 your net converges at all, while in second case it will!

import numpy as np

SEQ_LEN = 2000
N_SAMPLES = 50
n_test_samples = int(.1*N_SAMPLES)
if N_SAMPLES>SEQ_LEN+1: raise ValueError("Can't create more samples of this type than SEQ_LEN+1 is.")
ns = np.random.randint(0,SEQ_LEN,size=(N_SAMPLES,))
all_Xs = np.triu ( np.ones(shape=(SEQ_LEN+1,SEQ_LEN)) )
np.random.shuffle(all_Xs)

Xs = all_Xs[:N_SAMPLES, ...]
Ys = Xs.mean(axis=1)
Xs = Xs.reshape(N_SAMPLES, SEQ_LEN, -1)
trainX, testX, trainY, testY = Xs[:-n_test_samples], Xs[-n_test_samples:], Ys[:-n_test_samples], Ys[-n_test_samples:]

import tensorflow # To prevent
import tensorflow as tf # to declarations to prevent cryptic errors on my Windows laptop

from keras import Sequential
from keras.layers import Masking, LSTM, Dense
from keras.optimizers import adam
from keras.initializers import orthogonal
from keras.layers import Flatten

model = Sequential()
model.add(Masking(mask_value=0.00, input_shape=(trainX.shape[1],trainX.shape[2])))
model.add(LSTM(10, activation='tanh', #unit_forget_bias=True,
    input_shape=(trainX.shape[1],trainX.shape[2]),
    kernel_initializer=orthogonal(gain=.01),
))
model.add(Dense(1,activation='relu'))
model.compile(optimizer=adam(),loss='mse',metrics=['mae'])
model.fit(trainX,trainY,epochs=50,validation_data=(testX,testY),verbose=1,
    batch_size=1,
    shuffle=True)

testY_PRED = model.predict(testX)
print (np.concatenate(( testY_PRED, testY.reshape(-1,1)), axis=1) )

I haven also tested case with 300 samples and sequence length of 500 (converges). However, it is rather long to test if network converges with SEQ_LEN=2000 and let's say N_SAMPES=1200 (don't hope it will with N_SAMPLES=50) - it is up to you :)


Difference between CBOW and SKIP Gram word vectors

Here is a example for what @Christos Karatsalos is describing:
If you take a statement like "The cat jumped over the puddle",
In CBOW's case, the neural network's word embeddings are trained by passing a input set of words to the neural network and making the network to predict an output word as shown below:
$\{ cat, jumped, over, the, puddle\} \rightarrow  The $
$\{ The, jumped, over, the, puddle\} \rightarrow  cat $
$\{ The, cat, over, the, puddle\} \rightarrow  jumped $
$\{ cat, jumped, the, puddle\} \rightarrow  over $
$\{ The, cat, jumped, over, puddle\} \rightarrow  the $
$\{ The, cat, jumped, over, the\} \rightarrow  puddle $
In skip-gram's case, the neural network's word embeddings are trained by passing an input word to the neural network and teaching the network to predict a set of words as shown below:
$The \rightarrow  \{ cat, jumped, over, the, puddle\} $
$cat \rightarrow \{ The, jumped, over, the, puddle\}$
$jumped \rightarrow  \{ The, cat, over, the, puddle\}$
$over \rightarrow  \{ cat, jumped, the, puddle\}$
$the \rightarrow  \{ The, cat, jumped, over, puddle\}$
$puddle \rightarrow \{ The, cat, jumped, over, the\}$
For usage, if you are thinking of a fill-in-the-blanks kind of problem CBOW Word2Vec would be a suitable vector to use, on the other hand if you have a word and you are trying to come up with a new sentence with it then   skip-gram Word2Vec will be useful.
Take a look this lecture notes for more information.


What is meaning of zip(kfold.split(X, Y) in sklearn


Why linear regression feature coefficients become super large?

Large coefficients in linear regression are not necessarily a problem. They can be large becuase some variable was rescaled. You mentionned that you do some rescaling, but provide no details. Therefore it is not possible to tell what exactly is going on.

Here is a (general) example that explains how coefficients can get "large" (in R). Assume we want to model "visits" ($y$) contingent on "rating" ($x$):

# Data
df = data.frame(c(1,3,5,3,7,5,8,9,7,10),c(34,54,31,45,65,78,56,87,69,134))
colnames(df)<-c("rating","visits")

# Regression 1
reg1 = lm(visits~rating,data=df)
summary(reg1)

The regression results are:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   19.452     15.273   1.274   0.2385
rating         7.905      2.379   3.322   0.0105 *

This tells us, that visits increase by about 7.9 when rating increases by one unit. This is basically a linear function with intercept 19.45 and slope 7.9. Since our model is
$$ y = \beta_0 + \beta_1 x + u ,$$
the corresponding (estimated) linear function would look like:
$$f(x) = 19.45 + 7.9 x .$$

We can predict and plot our model. The results are just as expected, a positive linear function.

# Predict and plot
pred1 = predict(reg1,newdata=df)
plot(df$rating,df$visits,xlab="Rating",ylab="Visits")
lines(df$rating,pred1)

Now comes the interesting part: I do a linear transformation on $x$. Namely, I divide $x$ by some "large" number and I run the same regression as before:

# Transform x
large_integer = 10000000
df$rating2 = df$rating/large_integer
df

   rating visits rating2
1       1     34   1e-07
2       3     54   3e-07
3       5     31   5e-07
4       3     45   3e-07
5       7     65   7e-07
6       5     78   5e-07
7       8     56   8e-07
8       9     87   9e-07
9       7     69   7e-07
10     10    134   1e-06

# Regression 2 (with transformed x)
reg2 = lm(visits~rating2,data=df)
summary(reg2)

The results are:

Coefficients:
             Estimate Std. Error t value Pr(>|t|)
(Intercept) 1.945e+01  1.527e+01   1.274   0.2385
rating2     7.905e+07  2.379e+07   3.322   0.0105 *

As you see, the coefficient for rating is rather large now. However, when I predict and plot, I get basically the same results as before. The only thing that has changed is the "scale" of $x$ (the way $x$ is expressed).

Let's compare the coefficient for rating in both regressions.

In the first case it was:

# Relevant coefficient "rating" from reg1 (the "small" one)
reg1$coefficients[2]

  rating
7.904762

In the second case it was:

# Relevant coefficient "rating2" from reg2 (the "large" one)
reg2$coefficients[2]

 rating2
79047619

However, when I divide the coefficient rating2 by the same "large" number as I did to "rescale" the data, I get:

# "Rescale" large coefficient
reg2$coefficients[2]/large_integer

 rating2
7.904762

As you can see, the "rescaled" coefficient rating2 is exactly the same as the original coefficient for rating.

What can you do to check your regression:

Run the regression without any rescaling and see if the results make sense
Make a prediction from the regression
Rescale your data (i.e. "standardise"), which should contribute to get better predictions because data are less "wonky" in this case. However, coefficients have no natural interpretation any more
Compare standardised data to non-standardised to see how your data changed. Based on the discussion above, you should get a good idea if very small or large coefficients can make sense after standardisation
Make a prediction, compare to the prediction from above


Why is shuffling timeseries a bad thing?

Because the different observations in a timeseries by definition have an order, i.e. Jan 1st comes before Jan 2nd. If you then shuffle your observations this inherent order will be lost and you might be leaking data, meaning that your model will see data that is actually in the future since Jan 31st might suddenly be before Jan 1st.


Text classification into thousands of classes

In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of categories (for instance discard the least common ones).

In any case I'm not aware of any specific model to deal with a high number of classes, it's regular text classification. I'd suggest to start with a robust method such as decision trees, but there are many options.


Examining a DocumentTermMatrix in RTextTools

a documenttermmatrix is a simple_triplet_matrix. You can turn this into a simple matrix with the as.matrix command and then use all matrix functions.

# turn into simple matrix
mat <- as.matrix(doc_matrix)

# vector of the words
word_vector <- colnames(mat)

# Dataframe containing words and their frequency
df_words <- data.frame(words = colnames(mat), frequency = colSums(mat), row.names = NULL)


The answer depends on how your data is distributed, and what kind of error you can accept. If the distribution is close to uniform, and you care about having an absolute error of, say, $1000, data does not necessarily need scaling, or you could use standard scaling (min to max).

In any other case, you may want to apply a non linear transformation to your target variable. For instance, if the observations are evenly distributed among 0.01, 0.1, 1, 10, 100, etc., or most specifically when you care about relative error (a certain percentage of the target value) I would recommend to use a log transformation. This is because an error of 10% will always represent 0.1 unit after being log-transformed, whatever the target variable value is.

Most cases, if not all, will lie in the second case.


How to maintain accuracy with more classes?

There can be multiple explenations.

But here are some:

Images in train test and valid that are poor for some classes are not representative in train (poperties of these objects are not to be found hence you cant predict on something that you did not atleast partially learn on)- covariate shift

You say adding data is problematic and I understand but just note that more data beats everything else by a mile. Seriously. Hence augment the data, A LOT!. And make sure that for every class properties are representative and to be found in every train test and valid set.

I can start talking about architecture but these are marginal benefits at this point, make sure you have this point right. Without that every discussion about architeture if futile.


The ^M looks like the file came from a computer with a different OS from yours - maybe Unix and you're on Windows? Look on Google for a Unix2DOS utility, or maybe you can open it in Excel and then re-export to fix it.


How to get spike values from a value sequence?

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply:

data_df[data_df > 15000]

If this data is sitting in a particular column, you can use this instead:

data_df[data_df['column_name'] > 15000]

These will return the peak values.

Updated Answer:

If you want local extreme points (e.g. maximum or minimum ) around each peak, check scipy.signal.argrelextrema in Scipy. A concrete example:

Let's make a artificial random data with random spikes:

import numpy as np
import matplotlib.pyplot as plt

random_number1 =np.random.randint(0,200,20)
random_number2=np.random.randint(0,20,100)
random_number=np.concatenate((random_number1,random_number2))
np.random.shuffle(random_number)
plt.plot(random_number)

Now using argrelextrema function you will find the index of relative extreme values (either minimum or maximum)

c_max_index = argrelextrema(random_number, np.greater, order=5)

Please make sure you understand the "order" option. It basically look around 5 neighbouring points, and it returns the maximum in this case. And you can see how it works by pinpointing the found points on the actual graph like this:

plt.plot(random_number)
plt.scatter(c_max_index[0],random_number[c_max_index[0]],linewidth=0.3, s=50, c='r')

Note that you can retrieve peak points via random_number[c_max_index[0]], and c_max_index are just indexes of the extreme points.


As far as I can imagine, its multiple feed-forward neural networks with one neuron at each layer put next to each other, and connected from left to right, where each neuron is connected not just with the neuron below it, but the one at the left from the previous time.

Not really. Each cyan box in your image represents the exact same cell. Now this cell can be a lot of things just take a look at LSTM cell (h and c represents your A) but it can also be a network which takes $A_i$ and $X_{i+1}$ as input and returns $A_{i+1}$ and $Y_{i+1}$ as output.

It may be true if the RNN tries to predict e.g. a time series. To train such a net you would provide the series as a training input and the same time series but in the next step as an output (so it would try to predict $X_{i+1}$ based on $\forall_{j \in [1;i)} X_{j}$. But in general it's not true. The output may be in a completely different format and represent completely different thing exactly like in your example. In your example your $X_i$ is an encoded i-th frame and $Y_i$ is what the network thinks the distance the horse has traveled is up until i-th frame.
$A_0$ is the starting state of the RNN. What it is exactly depends on the exact architecture used but it's common to just set it all to zeros. We need this starting state because as I mentioned the same cell is used at each recurrent step so there has to be something to provide as network state in the beginning. There is no $X_0$ missing. Also there is nothing stopping you from making a prediction based on a sequence of length 1. It's just that it's not useful to use a RNN in such a situation.
The number of timesteps taken depend on the data - not the network. You can use the same network on sequence of length 512, 2 and 2 million. That's why they're commonly used to solve problems with varying length like speech recognition. LSTMs have weights just like a normal neural network does. You can think of these 512 hidden nodes as the size of a hidden layer in the cell. Using two layers of LSTM means using two LSTMs with 512  nodes, and using output of the first as the input of the second one. The output of the second LSTM is the output of 2-layer LSTM.


How to determine the weights of features when calculating the weighted average?

Retweet and Favourite are totally different features and although they are correlated with the popularity of the tweet, the exact correlation will be subjective to the training dataset you are using.

One way to come up with the weights is: Information Gain

This paper tries to come up with a weighted average using the user and tweet features for the prediction of retweet counts for a given tweet: http://www.nlpr.ia.ac.cn/2012papers/gjhy/gh154.pdf

Steps by step procedure:
1) Give every tweet a popularity score between 0 to 5 (You have to label your training data)
2) Calculate information gain for different weights of Retweet and Favourite count (refer: https://homes.cs.washington.edu/~shapiro/EE596/notes/InfoGain.pdf)


Genetic algorithms: what connection to support vector machine / naive bayes

A fast search on Google for the term "Genetic Algorithms" will return you this answer:

"A genetic algorithm is a search heuristic that is inspired by Charles Darwin's theory of natural evolution"

So, the term is usually associated with a "search heuristic", for example the search for a local optima.

By that common definition: Genetic Algorithms are not listed among these 7 classifiers since it is not a classifier, it would be in a list of optimization methods like Gradient Descent (and it's variation), Grid Search, Random Search, etc.


How to use a dataset with only one category of data

The Model learns to match the weights as per the image and feedback from label data.
If you will feed a few Image classes as "Not Cat", it will learn to classify similar features as "Not Cat". But might fail for a new Class.
 e.g. if it is trained on "Car/Furniture/Dog" as "Not Cat", then chances are high that a Wild Cat will be classified as Cat.
 Dumping all the Imagenet dataset will definitely provide quite a good variance to the "Not Cat" class and may work most of the time but that is not the appropriate solution for the problem.
Such type of problem will fall under One-Class-Classification.
Core idea is to use CNN to extract features then use some specialized models e.g. one-class SVM, Gaussian Mixtures, etc. to define a boundary for "Cat"

This problem, as defined by the one-class SVM approach, consists of identifying a sphere enclosing all (or the most) of the data. The classical strategy to solve the problem considers a simultaneous estimation of both the center and the radius of the sphere.

You may start with these links(In the specified order) -
Hackernoon blog
Arxiv
Researchgate
There are other approaches too i.e. based on Auto-encoder. Here, we try to put a Threshold on reconstruction error.
References-
Quora
SO
Keras blog
Also, may look here to check an idea which generated Random images for "No Cat" Class.Here


You could always write two functions (one with the decorator and one without) and call whichever suits you...

For example

@tf.function()
def graph_function()
    # This function will operate in graph mode
    ...

def eager_function()
    # This function will operate in eager mode
   ...

if tf.executing_eagerly()
    my_function = eager_function
else:
    my_function = graph_function

# You proceed to my_function from now on

I don't know if there is a better way but I've seen this a lot being used in the tensorflow official repository on github.


How to chose training sample for neural network in fraud detection?

I think you are facing an unbalanced dataset problem (fraud), and here are some Q&A associated with it.

Poor performance shown on Rare event modeling

https://stats.stackexchange.com/questions/134599/how-to-classify-a-unbalanced-dataset-by-convolutional-neural-networks-cnn


How to aasign a new data point to a cluster?

Suppose if  you use kmeans clustering then you can
1.train and  save the model using pickle
2.loa the model using pickle
3.pass your new sample as a vector to the predict function of the loaded model  object
model.predict([[0, 0], [12, 3]])
this will give you only one cluster label
4.if you want to get  top n clusters that the sample might belong to then take the cluster centers of your model as save it in a variable this will be list of list or matrix .
cc=kmeans.cluster_centers_
compute the similarity between your new sample and the cc matrix , rank the distances and yo will get n nearest clusters


What is difference between Multi-class One vs All and Multilabel Classification?

In multiclass classification each class is mutually exclusive, but in multilabel classification each class basically represents a different binary classification task.

An example.

Multiclass: Images that could contain a dog, a cat or a frog. Each image contains only one of the animals.

vs

Multilabel: Movie Genre Classification based on poster images. You have a poster image from a movie and want to determine whether the movie is a drama, action, thriller etc. A movie could belong to multiple of the these genres.

So to answer your question, the one-vs-all strategy in multilabel classification basically separates the k binary classification tasks. So using the above example, you would have k binary classifiers, where each one would basically represent each genre.
So you would have a binary classifier for drama, one for action, one for thriller etc.


Combinatoric system which using in bookies "S3" system

This can be interpreted as an unusual error correcting code with a non-binary signal.

Essentially, you want the tuples to be very different, i.e. there are no two tuples that agree in two positions. That is also a simple strategy for producing such codes (the more interesting question is what (x,y) combination is best.)

Think of it this way: assuming you only know part of the tuple. Say the first is unknown: (?, 2, 4) then you want to be able still conclude the correct triple.

https://en.wikipedia.org/wiki/Forward_error_correction


How to create a multi-dimensional softmax output in Tensorflow?

You are confusing "dimensions" with "order of tensor". A softmax with 256 different categories is a 256 dimensional vector, but is also a tensor with order 1 (whilst a matrix is a tensor of order 2). The paper is using the technical terms correctly, so the 256 dimensional vector is just a normal vector with 256 scalar entries.

Therefore a 256-dimensional softmax in TensorFlow is typically an output layer that looks something like this:

y = tf.nn.softmax(tf.matmul(h, W) + b)

where h is the last hidden layer, W is the weight matrix n x 256, and b is the bias 1 x 256 vector.

  In the paper, the candidate generation neural network model outputs a softmax with 256 dimensions, which acts as an "output embedding" of each of the 1M video classes

That is a description of the training process that compresses 1M different inputs to 256-dimensional output for use as an embedding for recommendation matches. The softmax is at the output, and as far as I can see is just a normal softmax classifier output as seen in many other classifier networks (except the result is not technically being used to classify anything). I am not clear on what supervision data was used or on what the input representation was. However, I don't think it likely that 1M "classes" ever appear as e.g. 1-hot encoding, because that would not scale out usefully to the many other millions of videos - the point of the embedding is to turn disparate features of the videos into something that be used as a similarity measure, that can be run on any video stored in YouTube.


The short answer is yes.

While your classes are imbalanced, model will be more likely to "learn" class 2 during the training, especially when it comes to mini-batch updates. (even though each mini-batch is unlikely to have only class 2 samples)

The common solution is to use weighted loss function or feed weighted samples to mini-batch.

In my practice, I always use weighted samples if I have sufficient data, even when my classes are not that skewed. Weights always help, at least it never ruins the model.


How can I fill NaN values in a Pandas DataFrame in Python?

You can use the DataFrame.fillna function to fill the NaN values in your data. For example, assuming your data is in a DataFrame called df,

df.fillna(0, inplace=True)

will replace the missing values with the constant value 0. You can also do more clever things, such as replacing the missing values with the mean of that column:

df.fillna(df.mean(), inplace=True)

or take the last value seen for a column:

df.fillna(method='ffill', inplace=True)

Filling the NaN values is called imputation. Try a range of different imputation methods and see which ones work best for your data.


Google Sheets: how to find max value in column B, corresponding values in column A, and max among these

Suppose you have data in the columns A, B from row 2 to 4. Put MAX(B2:B4) somewhere (in this example, I put it in B6). Then for the corresponding max in A you can use the formula MAX(ARRAYFORMULA(IF(B2:B4 = $B$6,A2:A4))).


Neural network with multiple layer: learning function

Here are some questions to help with the implementation of the algorithm:

1- You did the sum, but where/what is the activation function (sigmoid, hyperbolic tangent, Gaussian, etc.)?

2- Is multilayer? How did you forward? Did you use the sum along with activation of each neuron past for the other layers?

3- Have you stored the derivatives of activations to the backpropagation?

4- Will you use the traditional backpropagation algorithm or genetic algorithm to adjust the network weights?

5- If you choose the sigmoid and backpropagation, here is an example algorithm (this algorithm the derivative of the sigmoid is already calculated $O_{j}$ $(1-O_{j})$.


You may want to use something like the following snippet:

with h5py.File("some_path.h5") as f:
   f["data1"] = some_data

import h5py #added
with h5py.File('../images.h5', 'w') as f:
    h5_file = tables.open_file("images.h5", mode="w")
    # do what ever you want to do, it will be closed by itself.


Guiding principle of Neural Network structure building

You would use a sigmoid if you want your activations to be between zero and 1 and tanh between -1 and 1, this is important for the final layer, but for a hidden layer there is not much difference between them.

Here is an article on how to build a neural network in a more systematic way.

The basic principle is to start with only one layer, and make it bigger and bigger until you see satisfactory results, if you don't, add another layer and start over. Oh and make sure you use batch normalization, and don't use sigmoids or tanh's, use PreLUs or ELUs:
https://www.linkedin.com/pulse/keras-neural-networks-win-nvidia-titan-x-abhishek-thakur?trk=prof-post

In general:

Sigmoids: use for output layer which are probabilities with logloss loss function, and for gates in RNNs.

Tanh: use for updating the state in RNNs (linear-like activations might make it unstable), or for an output layer with variables between -1 and 1.

Rectifiers: use for activations in hidden layers of feed-forward network.


best NN architecture for point prediction

Since your outcome $y$ is restricted, you should check „beta regression“ in which the outcome is „squeezed“ into the interval $ \hat{y} \in [0, 1]$. See an example in R here: http://r-statistics.co/Beta-Regression-With-R.html

To my best knowledge, there is no "off the shelf" NN equivalent of beta regression.

You could of course use a "normal" NN for regression with dense layers with a sigmoid output layer. I.e. use a normal regression setup and change that last layer to have a sigmoid output (which is between 0,1).

What could be interesting as well is a boosted beta regression: https://cran.r-project.org/web/packages/betaboost/betaboost.pdf

Beta regression is also available for GAM (generalised additive models), which are very flexible in handling non-linearities (https://stat.ethz.ch/R-manual/R-patched/library/mgcv/html/Beta.html).

Ultimately, you need to check which approach works for you given your objectives and second order restrictions. It could (probably) also be okay to use a normal boosting approach (e.g. LightGBM or Catboost) - so without a beta link function. In this case some of the $\hat{y}$ could be $>1$ or $<0$. If you do not care too much about few "overshooting" predictions, you could also achieve good results with "normal" regression.


Understanding and find the best eps value for DBSCAN

KMeans and DBSCAN are two different types of Clustering techniques.
The elbow method you used to get the best cluster count should be used in K-Means only.
You used that value i.e. K=4 to assign colors to the scatterplot, while the parameter is not used in DBSCAN fit method.
Actually that is not a valid parm for DBSCAN
You will have to control "esp" to control the number of Clusters
Fit with esp=6 resulted in 112 Clusters. You only need these few lines of code
dbscan2 = DBSCAN(eps=6, min_samples=10).fit(data3)
fig = plt.figure(figsize=(12,7))

# max(dbscan2.labels_) # This is the number of Cluster
plt.scatter(data3[:,0], data3[:,1], s=10, edgecolors='none', c=dbscan2.labels_, alpha=0.5, cmap='hsv')


Exploring variables to guide xgboost tuning

A tip: Dont
A trick:Dont
The reason? Machine learning scientific methodology is based on cross-validation. Almost all papers (and i put the almost because of yes) select everything based on cross-validation and not in previous knowledge.
Xgboost is particularly more complicated because it has a lot of math involved.
For a simpler case, lets say that you have a problem with 3 features and you want to use a Lasso regression. Which hyperparameter will you choose? Even in this really simple case you don't know. You will need to do CV and select them there.
For xgboost the only tip that I can provide based on my experience is that the only important hyperparameter is the number of trees on the ensembling. The rest seems not to be so important.
My recommendation: only tune number of trees in the xgboost and select the best in cross-validation. Unless you want to find the absolute minime, that then, you are going to explore the whooooole hyperspace. Good luck!


How are samples selected from training data in Xgboost

In Gradient Boosting the simple tree is built for only a randomly selected sub-sample of the full data set (random without replacement).
While on the other hand, Random Forest the samples for each decision tree are selected via bootstrapping; sampling a dataset with replacement.
Particularly for xgboost (see paper here)the ratio of sampling of each tree can be modified with the following hyperparameter:

subsample [default=1]

Subsample ratio of the training instances. Setting it to 0.5 means that XGBoost would randomly sample half of the training data prior to growing trees. and this will prevent overfitting. Subsampling will occur once in every boosting iteration.

There are more parameters such as colsample_bytree, colsample_bylevel, colsample_bynode.
Bootstrapping techniques can vary with the implementation and. Normally I use Catboost implementation for several reasons.
You can read about the sampling of cb that you can use in the documentation here
Moreover, a paper that I found interesting a bit ago is Minimal Variance Sampling in Stochastic Gradient Boosting . I will quote one of the sentences from the abstract

Different sampling approaches were proposed, where probabilities are not uniform, and it is not currently clear which approach is the most effective...it leads
to a new sampling technique, which we call Minimal Variance Sampling (MVS)...The superiority of the algorithm was confirmed
by introducing MVS as a new default option for subsampling in CatBoost

Here the Catboost development team explains what is the new sampling technique that is the default in their algorithm.


80-20 better than full dataset for LightGBM

I cannot think about any theoretical reason. However:

It may be the case that the neg_mean_squared_error is not correctly implemented. Try another metric, like Median or Mean Percentage Error and see if that continues to happen.
It may be related to how the data is distributed, but still that's not common.

Lastly, I wouldn't put a model in production (if by that you mean deploying into a product, for example) without testing its performance in a test set, even if in another instance of the model, trained using cross validation, has good performance. One reason: it may be the case that a small part of the training data breaks your model, and does not generalise.


Convert CSV from an api to a dataframe format in python

Try the following code:

import requests
import pandas as pd
import io

params = {
  "api_key": "abc",
  "format": "csv"
}
r = requests.get('https://www.parsehub.com/api/v2/runs/ttx8PT-EL6Rf/data', params=params)
r = r.content
rawData = pd.read_csv(io.StringIO(r.decode('utf-8')))


Data model and algorithm for recommending "related" interests

I think you could use these category interests as implicit ratings, and use the Implicit package to do some collaborative filtering and recommend interests based on what other users have selected.

I'd suggest you do that instead of manually filtering out interests using your own domain knowledge. A lot of graphic designers use Apple products, and could be interested in "ios" and "painting". Its best to use the data instead of building a model based on our own understanding, or what we think is right.


Are decision tree algorithms linear or nonlinear

A decision tree is a non-linear mapping of X to y. This is easy to see if you take an arbitrary function and create a tree to its maximum depth.

For example:

if x = 1, y = 1
if x = 2, y = 15
if x = 3, y = 3
if x = 4, y = 27
...

Of course, this is a completely over-fit tree and won't generalize. But it demonstrates why a decision tree is a non-linear mapping.


Neural Network - Sparsity of collaborative based filtering and modelling the prediction problem

You don't have to use full input and output vectors to predict the ratings for a user.

Use the embedding idea and make pairs of (user_id,movie_id) for training. So your input could be like this: (112,1456)=7, it means the user with id number 112 watched the movie number 1456 and the rating was 7.So the size of embedding for the User is  |unique users|  and movies are |unique movies| you call it |M|.

With this trick, we changed the structure of the problem to what we already know. It's regression!

Now you can add all the bias(and whatever you want) to movies and users embeddings. Use the known loss functions for regression like
RMSE and you all set. For sure you can add other linear layers, relu, and dropouts on top of this embeddings and make it just like fully connected NN and get even better results.


tensorflow simple regression nan after >29 observations

The way you calculate the 'error' function is wrong.
You aggregated error value whenever you train the model. Therefore, the error value increases over time and reaches infinity (you can print the error value at each epoch to check that).
The error function should be computed as follows.

  error = tf.reduce_mean((y_hat - y_label)**2)

By the way, you can check the error value at each epoch.
Choose the hyperparameters of the model to make sure that the error value decreases in the time span.
Here is the error curve over the epoch span.


XGBoost evaluation metric unbalanced data - custom eval metric

I feel you are mixing few things:

1:20 is not exactly a class imbalanced data. Classification algorithms work well with this ration also. If you still feel so, you can oversample your positive class or undersample negative class.
AUROC generally to compare different models. Like one from logistic and other from Xgboost.( It also ensures robustness of models), Generally AUROC is not used for model accuracy/ getting cut-off values.
Logloss/ misclassification rate/ F1 score/ MCC any of these can be used based on your requirement of classification. If you are predicted cancer , then you might not want to miss any patient and might be Ok to false positive, then focus should be on recall. f1 gives balance of precision and recall. Log loss is used (Log Loss heavily penaltises classifiers that are confident about an incorrect classification) when you don't want to have false positive. These links might help.

https://machinelearningmastery.com/classification-accuracy-is-not-enough-more-performance-measures-you-can-use/
https://www.r-bloggers.com/making-sense-of-logarithmic-loss/
https://towardsdatascience.com/evaluation-metrics-for-classification-409568938a7d


Why does this paper say that 0-1 loss is insensitive to scaling of weights in a neural network?

Zero-one loss only really makes sense for (hard) classification, so the output of the network needs to already include a discretization.  Presumably the authors have a particular system in mind, and scaling weights doesn't affect the discretized predictions.

Two possibilities (though very closely related) come to mind, though I can't tell whether the authors had either of them in mind.  If the final output of the model is on a linear scale, and the decision is based on the sign of the output, then scaling weights will not change the sign and hence not the decision.  Similarly, if the final output of the model is on a probability scale via a sigmoid activation, then scaling the weights will not change the log-odds's sign, hence will preserve whether the output is above or below 0.5, hence not change the final decision (assuming a 0.5 cutoff).


How to write formula inside the loop to run this code in every hour continously in every day in panda python

you could do something like

import time
while True:
    <your code>
    time.sleep(3600)

this would execute your code every hour.
I hope this answer your question.


Should I modify my data to reflect time dependence of my rules?

Maybe I misunderstood something, but if you have feature vector with N features, it would be ok to add one more feature reflected the time. Then there could be two instances with the same first N features, but the different (N+1)-th and as a result the answer will differ too: "yes" for one year and "no" for another. However, for some instances answer will stay the same, which is also possible. So, I guess your idea might work, if there's actually some connection and meaningful "pattern" in real-life data.


Large variation in cross-validation scores

Based on the description my guess is that any or all of the following problems occur:

the data is not randomized, which could explain why the first fold get a very different performance than the others
perhaps the dataset is very small, causing massive variations in performance and overfitting.
in any case the model is very unstable, possibly due to overfitting caused by the model being too complex and/or dataset too small.


A question on realizable sample complexity

We want to prove:

If H is PAC learnable, then $\forall \epsilon, \exists C, \forall m \geq m_2:=Clog(1/\epsilon)(m_1+1/\epsilon^2), E[L] \leq \epsilon \mbox{ (a)}$
where $m_1:=m(\epsilon/2,1/2)$

Since $L \leq 1$, we have $E[L] \leq 1$. So proof is trivial for $\epsilon \geq 1$. Let $\epsilon \in (0, 1)$.

Find an equivalence for $(a)$:

$\begin{align*}
E[L] &= \int_{l \geq \epsilon / 2}ldP + \int_{l< \epsilon / 2} ldP \leq \int_{l \geq \epsilon / 2}dP + \int_{l< \epsilon / 2} \frac{\epsilon}{2} dP\\
&= P(L \geq \epsilon/2) + \frac{\epsilon}{2} P(L <\epsilon/2)\\
&= (1 - \epsilon/2)P(L \geq \epsilon/2) + \epsilon/2 < \epsilon
\Leftrightarrow P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)
\end{align*}$

Therefore, if

$\forall \epsilon, \forall m \geq m_3:=m(\epsilon/2, \epsilon/(2 - \epsilon)), P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)\mbox{ (b)}$ holds,

$\forall \epsilon, \forall m \geq m(\epsilon)=m_3, E[L] \leq \epsilon \mbox{ (c)}$ holds too (and vice versa)

Prove $(b) \Rightarrow (a)$:

Using The Fundamental Theorem of Statistical Learning for PAC learnable H with VC dimension $d$, we have:

$\begin{align*}
(&\epsilon/2, 1/2)\mbox{-learnable H with } m_1  \Leftrightarrow \exists C_1 > 0, m_1 \geq C_1\frac{d + log(2)}{\epsilon/2}\\
&\Leftrightarrow  log(1/\epsilon)(m_1 + 1/\epsilon^2) \geq \frac{log(1/\epsilon)(C_1d + C_1log(2) + 1/(2\epsilon))}{\epsilon/2}
\end{align*}$

which uses $1/\epsilon > 1$ and $log(1/\epsilon) > 0$.

Now we use an inequality without proof (plot the function here)

$\forall x > 1, \forall d, C_1 \geq 0, \exists C_2 > 0, f(x)=\frac{log(x)(C_1d+C_1log(2)+x/2)}{dlog(2x)+log(2x-1)} \geq C_2$

Setting $x=1/\epsilon$, we continue as:

$\begin{align*}
&... \overset{\exists C_2}{\geq} C_2\frac{dlog(2/\epsilon) + log((2-\epsilon)/\epsilon)}{\epsilon/2} \overset{\exists C_3}{\geq} \frac{1}{C_3} m_3
\Leftrightarrow (\epsilon/2, \epsilon/(2-\epsilon))\mbox{-learnable H with }m_3
\end{align*}$

By setting  $m_2 := C_3log(1/\epsilon)(m_1 + 1/\epsilon^2)$, we have $m_2 \geq m_3$, thus

$\begin{align*}
&\forall \epsilon, \forall m \geq m_3, P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)\\
&\Rightarrow \forall \epsilon, \exists C, \forall m \geq m_2 := Clog(1/\epsilon)(m_1 + 1/\epsilon^2) \geq m_3, P(L \geq \epsilon/2) < \epsilon/(2 - \epsilon)\\
&\Rightarrow \forall \epsilon, \exists C, \forall m \geq m_2 := Clog(1/\epsilon)(m_1 + 1/\epsilon^2), E[L] < \epsilon
\end{align*}$.

Proof is complete.


Passing the output of a model as input to an other model , computer vision?

I find your question a little hard to understand. If I understand correctly you know how to obtain a bounding box for a person.
Using these coordinates, you can select this rectangle using open-cv or matplotlib or any image processing library.
If your second model can handle multi-resolution or multiple sizes you can just use this selection as input.
If, on the other hand, your second model requires a fixed size you can resize the selection and use padding if necessary.


Basic doubt regarding "training" of a YOLO model

About Pre-trained model : This is a very common practise (especially in image recognition) and here is how we use it.
Let's imagine you want to recognize different types of food (beef, pork, vegetables, ...). You know some networks already exist that recognize all types of objects (boats, cars, food, sofas, ...). This objective of transfer learning is to use these models and adapt it to your task, so you pick an already trained model, and train it again, but only with your data (in our example types of food). Being able to recognize and distingish different objects is not useless for your task, so even if your network doesn't do the the same thing as the initial model, what the initial model has learned is used as foundations to learn your task.
We call this fine-tuning the model, it is basically using a complicated network that would be way too heavy to train because the dataset is not big enough, and just adjusting it so it performs our specialized task.
In your tutorial, the dataset Coco128 is made of 128 images, which is not enough for the training of a model as big as YOLOv5. So you use the pretrained weights and fine-tune it with your dataset. As far I as understand, the objective of the tutorial is to see how the model overfits these 128 images.

About validation, we usually splits datasets into 3 parts :

Training set : used to train the models. (~80% dataset)
Testing set : used to test the trained models, this part is the one used to estimate the accuracy of the model. (~15% dataset)
Validation set : used at the very end, to validate the fact that our model works correctly. (~5% dataset)

In many cases, Testing and validation datasets are the same (because validation dataset often is not necessary).
In your tutorial, what they call validation set is what I usually call the testing dataset, and it is the same as the training set, so bascially all your sets are the same. (it is very very very uncommon to do so, but since the goal of the tutorial is to see how we overfit this set, it makes sense here).


Are there any activation functions which on inputting integer data will produce the output as integers?

There are quite a lot of activation functions $f : \mathbb{R} \rightarrow \mathbb{R}$ such that their restriction to integers $f|_\mathbb{Z} : \mathbb{Z} \rightarrow \mathbb{Z}$ produce integer values. $f(x) = max\{0,x\}$ is quite famous.


Influence of the data generating distribution
To see this, first we have to mention that, neither by using Batch gradient descent (using the whole dataset to compute the gradient) nor using mini-batch gradient descent, we are computing the true (exact) value of the gradient.
To compute the true value of the gradient we would have to use the set of all possible values of the features, $x$, (and thereby the outputs $y$).
More formally, and refering to the quantity that we want to minimize as the expected value of the per-example loss function ($J(x,y,\theta)$, where $\theta$ are the parameters) w.r.t all the possible $x,y$ values, the true gradient $g$ is given by:
$$g = \frac{\partial}{\partial \theta}\mathbb{E}_{x,y\sim p_{data}}(J(x,y,\theta)) $$
And if we assume certain conditions We have that:
$$g = \mathbb{E}_{x,y\sim p_{data}}\left(\frac{\partial}{\partial \theta}J(x,y,\theta)\right) $$
Where $p_{data}$ is the data generating distribution (the distribution from which the values of $x$ and $y$ are drawn). However, this data generating distribution is usually unkown. We just know the dataset that we are given.
Because of this, to update the parameters using all the information given (the training set), we instead use the empirical ditribution defined by the training data ($\hat{p}_{data}$) which puts a probability of $1/m$ on each of the $m$ samples $(x^{(1)}, y^{(1)}), \,(x^{(2)}, y^{(2)}),\,...\,,(x^{(m)}, y^{(m)})$ of the dataset. So the gradient is approximated by:
$$ \begin{aligned}
\hat{g}&=\frac{\partial}{\partial \theta}\mathbb{E}_{x,y\sim \hat{p}_{data}}(J(x,y,\theta))\\&=\frac{\partial}{\partial \theta}\left(\sum_{i=1}^m \frac{1}{m}J_i(x^{(i)},y^{(i)},\theta)\right)\\
&= \frac{1}{m}\sum_{i=1}^m\frac{\partial }{\partial \theta}J_i(x^{(i)},y^{(i)},\theta)
\end{aligned}
$$
Ending up with the Batch gradient descent.
But what happens with mini-batches?
By using mini-bath updates, we are continuously seeing new data (assuming that we compute just one epoch). So in this case, using mini-batches, we are using the data generating distribution.
This means that on each mini-batch update, by sampling this data generating distribution, we end up with an estimation ($\hat{g}$) of the true gradient ($g$) that is unbiased i.e. $\mathbb{E}_{x,y\sim p_{data}}(\hat{g})=g$. To see this, and considering $\text{s-sized}$ mini-batches:
$$\begin{aligned}
\mathbb{E}_{x,y\sim p_{data}}(\hat{g})&=\mathbb{E}_{x,y\sim p_{data}}\left(\frac{g^{(1)}+...+g^{(s)}}{s}\right)\\
&=\frac{1}{s}(\mathbb{E}_{x,y\sim p_{data}}(g^{(1)}+...+g^{(s)}))\\
&=\frac{1}{s}s\,\,g=g
\end{aligned}
$$
Thereby, making succesive mini-batch updates we would be tending in average (as shown by $\mathbb{E}_{x,y\sim p_{data}}(\hat{g})$) to updating our parameters with the true value of the gradient. And this is what I think the authors refer to in the quote of the question.

Great references:
Deep Learning book, Ian Goodfellow et. al Chapter 8.1
Answers from here


ImageDataGenerator - trained with model.fit instead of model.fit_generator

The data augmentations are defined when you instantiate your data generators. An example is as such:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

Other augmentation techniques can also be used by setting the correct parameters. Refer to this link: https://keras.io/preprocessing/image/


What are the advantages of Skip-Gram methods?

Basically, skip-gram aim for computing the probability of a context given a word (it is CBOW that does the opposite). BUT, it does not simply learn a co-occurrence matrix, it compresses the information in a low dimensional space (for example 300 for a 100000 original dimension). By doing this, it learns continuous low dimensional representation for the words.

It is proved that word2vec actually factorizes a word-context pointwise mutual information matrix (close to the co-occurrence matrix). I think this article might give you a better understanding of the underlying processes.


There is no such function.

A convex function defined on all of $\mathbb{R}$ with bounded range is constant.


How does the trainable projection layer used in PRADO and pQRNN work?

The key idea in both the papers are word embeddings are replaced by trainable activations that are computed using a nueral network. The network (referred to as bottleneck layer) takes word projections as input. The parameters of this network is shared across tokens. This results in a trainable token representation. The projection itself is not trainable, but the representation derived from projection is trainable. Hope that clarifies.


With the Start Date and End Date given , finding date counts in Pandas

Try this. It is a bit ugly, as I couldn't manage to get a solution without iterating through one of the dataframes. It could be done by iterating over the original data, or the new df with the list of all dates.

I had a similar problem, any cleaner ways of doing this would be appreciated.

import pandas as pd

df = pd.DataFrame(data={'Item':['A','A','B','A'],
                        'StartDate':['2018-01-01','2018-01-07', '2018-01-03','2018-01-03'],
                        'EndDate': ['2018-01-04','2018-01-08','2018-01-05','2018-01-05']})
df['StartDate'] = pd.to_datetime(df['StartDate'])
df['EndDate'] = pd.to_datetime(df['EndDate'])

index = pd.MultiIndex.from_product([['A','B'],pd.date_range(start = df.StartDate.min(), end = df.EndDate.max())], names=['item', 'date'])
new_df = pd.DataFrame(index = index, data = {'counts':0})

dates_list = new_df.index.get_level_values(1)
items_list = new_df.index.get_level_values(0)
for row in df.itertuples():
  new_df.counts[(items_list == row.Item) & (dates_list>=row.StartDate) & (dates_list<= row.EndDate)] += 1

print(new_df)

Which gives this. You can remove the multi-index later if needed

                 counts
item date
A    2018-01-01       1
     2018-01-02       1
     2018-01-03       2
     2018-01-04       2
     2018-01-05       1
     2018-01-06       0
     2018-01-07       1
     2018-01-08       1
B    2018-01-01       0
     2018-01-02       0
     2018-01-03       1
     2018-01-04       1
     2018-01-05       1
     2018-01-06       0
     2018-01-07       0
     2018-01-08       0


Order SparseVectors by the closest distance to given SparseVector

it appears that problem was solved with BucketedRandomProjectionLSH. After fit && transformation, and approxNearestNeighbors resulted dataset contains distCol, which (as per LSH.scala):

  @param distCol Output column for storing the distance between each result row and the key.


Why is my prediction using ARIMA better if I'm using less historic data?

Electricity prices are essentially the same as stock prices: best modelled by a random walk, where the best prediction for tomorrow is the price today. Therefore I am not really surprised that you get worse results using more historical data.
 Some versions of ARIMA will also include regularisation, which will punish your model for including more and more data - inclusion of new data must be justified by contributing to a lower residual error to be "worth" inclusion.

Other models that tend to be a little more robust use features other than the actual target, here the price. For example, trying to predict the volatility of the price might  prove to be more accurate. For this there are GARCH models (Generalised AutoRegressive Conditional Heteroskedasticity).

Another thing you might consider is to include external data... for example, electricity consumption is heavily influenced by the weather - if it is cold outside, a lot of people heat their homes using electrical heaters, they also drink more hot drinks etc.


How to apply MultiOutputClassifier to a dataset for Naive-Bayes algorithm

The examples's make_classification, shuffles, and stack are just to generate a multilabel dataset. You've already processed to y_train and don't need any of that.
Then, you want a base estimator to be Naive Bayes, not a random forest as in the example.  And all sklearn models should be fit as model_instance.fit(X, y, sometimes_optional_parameters); the error being raised is because you passed a model object instead of y (and also are using the wrong model object).  So the following should do what you want:
nb = GaussianNB()
multi_target_nb = MultiOutputClassifier(nb, n_jobs=-1)
multi_target_nb.fit(X_train, y_train)


Difference between output of probabilistic and ordinary least squares regressions

They both solve the exact same objective, which is minimizing the mean squared error. However the second method can answer "how confident it is that slope is not zero, i.e. $Y$ is correlated with $X$?" via p-value.

In detail

Lets denote the data as $(X, Y) = \{(x_n, y_n)|x_n \in \mathbb{R}^D, y_n \in \mathbb{R}\}$. And the regression as $\hat{y} = Ax+B$.

Extra quantities returned by scipy.stats.linregress(X,Y) are: rvalue ($r$), and pvalue ($p$).

In statistics, $r^2$ (known as r-squared) measures the "goodness-of-fit" . That is, as regression $\hat{y}=Ax+B$ gets closer to observation $y$, $r^2$ gets closer to $1$. Since it is a function of $y$ and $\hat{y}$, it can be calculated for the first method too. So no difference here.

However, $p$ is specific to second method. scipy.stats.linregress(X,Y) adds a normality assumption to noise, i.e. assumes $\epsilon \sim  N(0, \sigma^2)$ where $$\epsilon = y - \overbrace{Ax+B}^{\hat{y}}$$
On the basis of this assumption, it can answer an additional question: "how confident it is that the slope is not zero?". The first method cannot answer this question.

For example, suppose the estimated slope is $2.1$ for both methods, we still cannot tell whether this slope is significant or $Y$ is actually independent of $X$. Unless we look at the value of $p$. For example, for $p < 0.01$ we are confident (at significance level $0.01$) that $Y$ is correlated with $X$, but for $p > 0.1$ we cannot be confident, i.e. slope $2.1$ could be due to chance and $Y$ might be independent of $X$.

This link gives more details on how p-value is actually calculated in second method.


When I would use a specific similarity coefficient over another?

Jaccard - measures similarity of assymetric, binary attributes. For example, if you have insurance claims with binary attributes ("poor driving record", "premium paid in cash") you can compare claims with those attributes.
Cosine - measures similarity between vectors, like feature vectors. Could be used in a recommender system where a user asks to see items similar to some selected item.
Dice - equivalent to F1-score. Often used in image segmentation, comparing a model's output with reference masks.


Do I need to correct predict_proba by training fraction?

It depends on the model, whether your training data is representative of the testing data, and possibly on how "easy" the classification problem is.  But the short answer is that (probably) no adjustment is necessary.

The model is exposed to the imbalance in the training data; most models will have that baked into their probability scores, so to trust the given probabilities requires that the testing data have a similar distribution.  This goes to what exactly we mean by "the probability" that the new car is blue; I think in most cases, you would want the training data to be representative of the data in your use-case, so that we really do want the probability to be aware of the imbalance.  [Now, sometimes in the presence of extreme imbalance, certain models might learn better with some adjustment, in which case you'll want also some "probability calibration" post-processing, but that's generally more complex than the simple multiply-or-divide method you suggest.  In logistic regression for example, there's a well-known additive adjustment to the constant term, so that turns into a multiplicative adjustment to the log-odds, which does not translate into such a nice transformation on probabilities.]

Finally, some models implicitly skew their scores, so predict_proba should be taken with a fair bit of skepticism.  Again, look up "probability calibration."


Since the time-series are annual, the data points you have for each time-series are limited and also quite distant (the values are 1 year apart). So I wouldn't use Dynamic Time Wrapping on your data.

If you are interested in comparing the patterns, a very simple approach would be Pearson's correlation. Keep in mind that this will not compare the actual values but the patterns (i.e. if the values have similar fluctuations with the years, so for example time-series [1 2 3 4] would have higher correlation with [5 6 7 8] than with [1 1 2 2])

If you are interested in both values and pattern, I would use a distance-based metric: Euclidean Distance, Manhattan distance etc. I believe you will find this post interesting, where the mathematical background of similarity is explained. Also, python implementations of several distance metrics in python (including cosine-similarity) can be found in this blog-post.


Spark DataFrame "Limit" function takes too much time to display result

The reason is the way limit and the show is implemented under the hood. Show just reads the first 20 (first n) rows, which limit reads the whole data before showing it. Refer this answer on StackOverflow - link


Shuffle the data before splitting into folds

I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each iteration of hyperpatameter tuning re-shuffle the data before splitting into batches/folds (i.e., the shuffle argument in the KFold function)?

No, its no needed, shuffling is needed before split.

  I assume that if the outcome depends on shuffling then the model is not robust. Is this correct?

You are right, good model has great performing on every combination of data.

  However, this also may not be 'fair' to the model because the result is not reproducible since the data for each fold is different every time I run cross-validation (i.e., each hyperparameter combination is evaluated on totally different folds. For example, the training/validation dataset in fold #1 of the 1st tuning iteration is different than fold #1 dataset of the 2nd tuning iteration.)

You make cross-validation to check performance on every chunk of data. Your task is achieved most generous model.


How to standarize feature vector with data in different scales?

we've to use standardization(substitute by z-scores or subtract mean divide by standard deviation) in such cases:

#find mean of each feature
mean = data.mean(axis = 0)
#subtract mean
data - = mean
#find standard deviation of each feature
std = data.std(axis = 0)
#divide by it
data /= std


Why we use transposed filter as the deconvolution operation instead of the pseudo inverse of filter?

First, transposed convolution isn't the inverse operation of convolution.
It only takes the output shape of origin convolution as its input shape and takes the input shape of origin convolution as its output shape.

Second, transposed convolution can be seen to a learnable upsampling operation. It just likes bilinear or bicubic upsampling but all of the parameters are learned by gradient descent.

https://www.youtube.com/watch?v=nDPWywWRIRo&index=12&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&t=1632s


How to deal with string labels in multi-class classification with keras?

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the original data set, you now have a simple consistent way to represent each. It doesn't do one-hot encoding, although as you correctly identify, it is pretty close, and you can use those ids to quickly generate one-hot-encodings in other code.

If you want one-hot encoding, you can use LabelBinarizer instead. This works very similarly:

 from sklearn.preprocessing import LabelBinarizer
 encoder = LabelBinarizer()
 transfomed_label = encoder.fit_transform(["dog", "cat", "bird"])
 print(transfomed_label)

Output:

[[0 0 1]
 [0 1 0]
 [1 0 0]]


Can `k=1` be a good choice for K neighbors classification?

Having k = 1 is not inherently unreasonable. This just means that all new observations will be predicted to be the class label of the first closest neighbour you have in your training set. Predicted class probabilities should just be 1 or 0 since the the estimator will take on the form 1/1 or 0/1. This also makes it clearer as to why log loss will probably not favor a k = 1 model; the predicted probabilities will be far too extreme in the cases where the model favors the wrong class label. Consider the case where the the true label is 0, i.e. $y_i = 0$. If the closest neighbour in your training set is of class label 1, then your model will incorrectly predict $Pr[Y = 1|X] = 1.$ Hence, $ (1-y_i) * ln(1-Pr[Y = 1|X])$ will be infinity. On the other hand, if the true label is 1, i.e., $y_i = 1$, and the closest neighbour to this observation ends up being of label 0, now $Pr[Y = 1|X] = 0$ and we have $y_i * ln(Pr[Y = 1|X])$ infinity as well. Thus, log loss will either be 0 (where your model predicts correctly) or -infinity for all of your observations; clearly very extreme.

However, I would just like to discuss the validity of even using accuracy as a scoring measure in the first place. Accuracy is simply not a proper scoring rule and often leads to unoptimal, bogus models. The problem is due to the arbitrary nature of the threshold you use to classify a "1" or a "0", and also that a correct classification of "1" with predicted probability 51% is not the same as a correct classification of "1" with predicted probability 90% (for instance).

Logloss is a proper scoring rule that is minimized when the predicted probabilities are close to the true, population probabilities. Accuracy is clearly influenced by what threshold you choose to use and I would not be surprised at all to see the accuracy seemingly "higher" if you choose the right threshold that maximizes accuracy for the model fit on log loss (assuming you are just using the very arbitrary 50% cutoff). Thresholds chosen should depend on the context of how your model is being used and perhaps one's own personal belief in risk; in particular, the costs of making incorrect predictions.

If you are absolutely forced in classifying objects as 1 or 0, then I suggest tuning the threshold as part of your model validation process and then assessing the accuracy on a held out test set (using the threshold found in your validation). See this thread here for more discussion on this topic.


The same link shows how these features are extracted, with a deep look into the cited article "Image-based recommendations on styles and substitutes":

  Features are calculated from the original images using the
  Caffe deep learning framework [11]. In particular, we used a
  Caffe reference model with 5 convolutional layers followed
  by 3 fully-connected layers, which has been pre-trained on
  1.2 million ImageNet (ILSVRC2010) images. We use the
  output of FC7, the second fully-connected layer, which results
  in a feature vector of length F = 4096.

The reference neural network that was mentioned is the BAIR Reference CaffeNet at the Caffe Model Zoo, which is a slightly modified version of AlexNet.

Since the model was trained over ImageNet, which contains a wide variety of photographs of various categories (1000 of them, if I recall correctly), retrieving the neural codes of one of the layers (obtained just by forward propagation) will give you visual features with a fair representation of the images, even if the network was not specifically trained for Amazon's tasks (such as product recommendation). What these values actually mean is not something that tangible: it is the outcome of multiple 2D convolutions and other normalization and regularization functions, the parameters of which were adjusted specifically for classifying photographs from ImageNet.

The FC7 layer has a rectified linear unit activation (ReLU), which means that they are all non-negative numbers (potentially with several zeros). And since it's a fully connected layer that follows several convolutions, there is no intuitive mapping between a feature index and a certain characteristic of the image. You may picture the network as a highly complex function that yields a high-level representation of the image, under the form of a vector of numbers.

See also the paper Neural Codes for Image Retrieval, where the authors retrieve features from a pre-trained neural network in this fashion, for retrieving images in a different image domain.


Role of mode in bottleneck function in unet network

That mode parameter is simply referring to how (what way) the bottleneck should be, what type of bottleneck to use.

There is nothing fancy to it. There is at least one usage in statistics of mode that I know of, as a way to describe some distributions, but this doesn't have anything to do with it.

As to the difference between cascade and parallel, in the former the layers pass the input to each other (like in the diagram), while in the latter the layers operate on the same input.

In general, cascade refers to the application of operations sequentially (e.g. in a cascade classifier). Parallel means that operations are performed independently and could, theoretically, be ran simultaneously (think about parallel operations for threads). Here, those layers are parallel since they all operate on the same input, and their output is then collected by subsequent layers.


Your text has some characters that the editor does not show.  Using an online UTF-8 encoder, you can see that the end of the line has:

\x29\xef\xbb\xbf\x0a

When it should simply be a close paren and a new line like:

\x29\x0a

You need to delete the offended characters.  Best way to handle these situations, is to select an area of the text on either side of the compiler complaint and delete it all as a block and then retype the text.

After the cleanup your code produces:

[1]

PS: That error output is called a stack trace, and it was exactly right, there was an invalid character present at the location indicated.


First of all, dimensionality reduction is used when you have many covariated dimensions and want to reduce problem size by rotating data points into new orthogonal basis and taking only axes with largest variance. With 8 variables (columns) your space is already low-dimensional, reducing number of variables further is unlikely to solve technical issues with memory size, but may affect dataset quality a lot. In your concrete case it's more promising to take a look at online learning methods. Roughly speaking, instead of working with the whole dataset, these methods take a little part of them (often referred to as "mini-batches") at a time and build a model incrementally. (I personally like to interpret word "online" as a reference to some infinitely long source of data from Internet like a Twitter feed, where you just can't load the whole dataset at once).

But what if you really wanted to apply dimensionality reduction technique like PCA to a dataset that doesn't fit into a memory? Normally a dataset is represented as a data matrix X of size n x m, where n is number of observations (rows) and m is a number of variables (columns). Typically problems with memory come from only one of these two numbers.

Too many observations (n >> m)

When you have too many observations, but the number of variables is from small to moderate, you can build the covariance matrix incrementally. Indeed, typical PCA consists of constructing a covariance matrix of size m x m and applying singular value decomposition to it. With m=1000 variables of type float64, a covariance matrix has size 1000*1000*8 ~ 8Mb, which easily fits into memory and may be used with SVD. So you need only to build the covariance matrix without loading entire dataset into memory - pretty tractable task.

Alternatively, you can select a small representative sample from your dataset and approximate the covariance matrix. This matrix will have all the same properties as normal, just a little bit less accurate.

Too many variables (n << m)

On another hand, sometimes, when you have too many variables, the covariance matrix itself will not fit into memory. E.g. if you work with 640x480 images, every observation has 640*480=307200 variables, which results in a 703Gb covariance matrix! That's definitely not what you would like to keep in memory of your computer, or even in memory of your cluster. So we need to reduce dimensions without building a covariance matrix at all.

My favourite method for doing it is Random Projection. In short, if you have dataset X of size n x m, you can multiply it by some sparse random matrix R of size m x k (with k << m) and obtain new matrix X' of a much smaller size n x k with approximately the same properties as the original one. Why does it work? Well, you should know that PCA aims to find set of orthogonal axes (principal components) and project your data onto first k of them. It turns out that sparse random vectors are nearly orthogonal and thus may also be used as a new basis.

And, of course, you don't have to multiply the whole dataset X by R - you can translate every observation x into the new basis separately or in mini-batches.

There's also somewhat similar algorithm called Random SVD. I don't have any real experience with it, but you can find example code with explanations here.

As a bottom line, here's a short check list for dimensionality reduction of big datasets:

If you have not that many dimensions (variables), simply use online learning algorithms.
If there are many observations, but a moderate number of variables (covariance matrix fits into memory), construct the matrix incrementally and use normal SVD.
If number of variables is too high, use incremental algorithms.


Predict whether or not a user will visit the library tomorrow using historical data

First and foremost, you need to reformat your data into what's called a balanced panel structure. For each day in your training data, each user should have a record for that day associated with an indicator variable for whether or not they visited. If every record in your training data corresponds to a visit, you're not giving those classifiers much to work with. You're model needs to know what a user's status looks like on days when they don't visit to be able to discriminate between days when they will and won't visit. The indicator variable, denoting "this user visited today", is the target of your prediction. Not the date or time column, if that's what you were trying.

Additionally, you're going to need to do some feature engineering. This process comprises the bulk of time and creativity expended on most data science projects, and the amount of effort you will put in will make or break your project. For each record in the data set, you should calculate things like:

What is the day of the week/month/year?
how many times did the user visit on this day of the week in the last k days?
how many times did the user visit on weekdays/weekends in the last k days?
How many days has it been since the users last visit?
How many times has the user visited the library in the last k days?
how many times has the user visited the library and paid a fine in the last k days?
How many users visited in the last k days?
Is the library open this day (e.g. it could be a national holiday)?
Are the public schools currently in session? On holiday? In finals?
How many days since the beginning of the most recent semester?
How many days until the semester ends?

Better yet, if you're not limited to what you've shown us...

How many books are currently in the users possession? How many are late?
If the user visited today, how much would they owe in fines?
how many books did they check out in their last visit?
how many books have they checked out in the last k days?
how long has this person been a member at the library?
has their membership expired?
in the last k years, have they allowed their membership to expire?
have they recently updated their address?
How much have they donated in the last k years?
What fraction of books checked out in the last 5 years by this user have belonged to each dewey decimal class?

These are just some ideas off the top of my head. Don't let yourself be limited by my suggestions.

If you really only have the data you've shown us, a project you can try that would be more likely to give you good results would be to predict just the number of people who will visit the library on a particular day, rather than which actual people. If this sounds like it might be useful to you, check out Poisson models.


RNN input data must follow this pattern:

  ( Number of observations , Number of input series , Window size )

Keep the number of observations to None when you define the LSTM input_shape. The number of input series in this case is 2, since you are using two time series to make predictions. Window size is the length of your time series row. Your input shape would then be:

input_shape = (2, window_size)


Types of maps in Interpretable Machine Learning

Difference between saliency and heat maps - https://www.reddit.com/r/EyeTracking/comments/hhy2u5/what_is_the_difference_between_saliency_maps_and/

Activation maps - https://computersciencewiki.org/index.php/Feature_maps_(Activation_maps)

These two links should answer your question!
Regarding types of maps, they could be many but activation maps are the most used. People can create various maps so i will not be stressing on the question. I am no expert but I have written one paper on iml. Smoothed Score-CAM. Do look into it :)


Finding a range of values for each feature that contribute to positive classification

It's impossible in general, simply because a particular value or range for feature A might correspond to class 'good' if feature B has a certain value/range but correspond to class 'bad' otherwise. In other words, the features are inter-dependent so there's no way to be sure that a certain range for a particular feature is always associated with a particular class.

That being said, it's possible to simplify the problem and assume that the features are independent: that's exactly what Naive Bayes classification does. So if you train a NB classifier and look at the estimated probabilities for every feature, you should obtain more or less the information you're looking for.

Another option which takes into account the dependency between variables is to train a simple decision tree model: by looking at the conditions in the tree you should see which combinations of features/ranges lead to which class.


How to group by multiple columns in dataframe using R and do aggregate function

library(dplyr)
library(tidyr)
df %>% group_by(country, gender) %>%
       summarise(total_loan_amount =sum(loan_amount)) %>%
       spread(gender, total_loan_amount) %>%
       ungroup() %>%
       transmute(country = country,  female_percent = F / (F+M), male_percent = M /(F+M))

results in

 Source: local data frame [1 x 3]

      country female_percent male_percent
       (fctr)          (dbl)        (dbl)
    1  Austia      0.2673267    0.7326733


Is there a method that is opposite of dimensionality reduction?

The short answer is yes.

Essentially you will be performing some sort of feature engineering. This means constructing a series of functions of your data, often:

$\phi_j(x): \mathbb R^p \rightarrow \mathbb R \ ,\ \  j=1,...,K$

Which, strung together, define a transformed data vector $\boldsymbol \phi(x)$ of length $K$.

There are a number of ways, better and worse, of doing this. You may want to look up terms like:

Splines and generalised additive models.
The kernel trick (how to make a model where $K\rightarrow \infty$).
Feature engineering (of the manual variety, e.g. adding an $x^2$ column to your data).
Deep learning, representation learning

As you might guess from such a varied bag of techniques, this is a large area. It goes without saying really but care has to be taken to avoid overfitting.

This paper Representation Learning: A Review and New Perspectives deals with some of the issues around what makes a particular set of features 'good', from a deep learning perspective.


I am getting different mean_absolute_error when i retrain my model everything same

I think you need this one too:

from tensorflow import set_random_seed
set_random_seed(123)


xgboost performance with predicted values as input

You could try   the next thing:,   get some   top  k predictions made with   xgboost, and use  only those as features  and   feed them  into  an LogisticRegression model.
I am currious if that will help  in case  you want  to  give a  feedback   on my proposed solution


training data in supervised models, case of linear regresion

no. of features $p$
no. of observations (training examples) $n$

In your example, $p = 10$ and $n = 3000$

So, what you said is right, you have to perform the following equation 3000 times.
$$h(x) = \sum_{i=0}^p \theta_{i}x_{i
}$$

and, see that the summation runs from $0$ to $p$, so a total of $p+1$ multiplications, with the first one $\theta_{0}$ being bias, and $x_{0}$ being $1$ always.

If you have all $\theta_{i}$ values, at inference time for getting prediction for each test example, you perform the equation above, but the objective of linear regression is to find the values of $\theta$. This is nothing but called as fitting the model.

And the equation for finding $\theta_{i}$ values is

$$\theta_{i} := \theta_{i} - \frac{\alpha}{n} \sum_{j=1}^{n}(h^{j}(x)- y^{j})x^{j}$$

You can observe that calculation of $\theta$ also has the first equation $h(x)$,

where $\alpha$ is the Learning rate.


Machine Learning algorithm for predicting number of cases in pandemic

It will depend completely on your feature engineering so I can think that in this case your model is maybe only predicting the mean or median of your target.

Also, it might help try using other kinds of models since you are trying to predict the counts of an event on a determined period of time, so it might be useful to use Poisson models that are in experimental phase in sklearn, nonetheless, the documentation might help to understand how the model works


What are your thoughts on SKLearn's dismissal of GPUs for machine learning?

GPU doesn't inherently fit naturally into all machine learning algorithms.  A natural contender is one that inherently takes a myriad of matrix multiplication.  This makes sense since graphic processors inherently were design for this.  However, for an algorithm like a Random Forest this may not be so important.  Also there exist a cost to transfer information to a GPU.  Any algorithm that is O(n) should not be computed on a GPU because it takes O(n) to communicate the data.  There's a few other issues that GPU present dealing with RAM and Threading, each of which often render making a GPU variant of a project more of a hassle than it's worth.

Furthermore, adding GPU to the sklearn framework inherently adds a hardware dependency and complexity that seems needless for shallow algorithms.  Odds are, if you're needing access to your GPU, you are dealing with a neural network, which has it's own unique architecture challenges.  I think it makes far more sense to separate deep learning into it's own module (look at how huge tensorflow/pytorch/etc) project are) than force Sklearn to add hardware dependencies for marginal computational gains.


Passing variables and values from an R script to a shell script

The Bash feature you want is called command substitution, e.g:

echo $(./test.r)

That will echo the output of your Rscript. Alternatively you could capture it in an environment variable.


Cross-validation of a cross-validated stacking ensemble?

I posted this same question on Reddit and someone was kind enough to answer

By https://www.reddit.com/user/patrickSwayzeNU

  This

    When I do CV B, is it valid to do CV A just once and use those out of fold predictions for the entire CV B process?

    Normally, I'd think that there'd be some data leakage in the first method, but one could also reason out that since the out of fold predictions are taken, well, out of fold, that issue is taken care of.

  Yes, your data set created by CV A is now "good as new".