Is it always better to use the whole dataset to train the final model?

Interesting question. I personally haven't seen that for products going into production, but understand the logic.

Theoretically, the more data your deployed model has seen, the better is should generalise. So if you trained the model on the full set of data you have available, it should generalise better than a model which only saw for example train/val sets (e.g. ~ 90%) from the full data set.

The problem with this (and the reason we split data into train/val/test sets in the first place!) is that we want to be able to make statistical claims as to the accuracy on unseen data. As soon as we re-train a model again on all the data, it is no longer possible to make such claims.

[Edit]

Here is a related question on Cross-Validated, where the accepted answer makes similar points to me and mentions other ways of doing things.

We loop over:

train a model
assess performance on validation set $\rightarrow$ if satisfactory, go to step 5
change model
go to step 1
assess performance on test set
Present model with test accuracy found in step 5

Eventually, if you manage to get a great score on the test set, you can claim it generalises well.  So the question as to whether re-training on the full dataset will improve performance on future unseen data is not strictly something you can test. Empirical evidence of better performance in other related problem sets would be the only source or guidance at the point in time when you must make the decision.

A sanity check would be to test the final re-trained model again on the original test set; expecting that it scores higher than it ever did when the model only saw the train/val set, because it has actually seen the test set during training. This wouldn't make me feel 100% confident that this final model is superior in all future cases, but at least it is as good as it can be with the given data.

Perhaps there are more rigorous arguments against doing what you say (probably academically motovated), however it does seem appealing for practical applications!


Keras - Error when using HDF5Matrix to fit the model

This might be a bug in Keras. However, you might have some luck if the "validation_split" parameter is set to 0.001 as mentioned on the corresponding Github page.


Application of machine learning in your job

Using machine-learning techniques, nowadays computers can automatically see and understand better than years ago. Tasks like recognition, localization, detection, semantic segmentation and related things have better performance today. This progress has promoted other fields too. Automatic cars, using autonomous driving benefit these advantages for pedestrian detection or car detection.

Recommender systems are so popular today. They suggest you exactly what you need based on your actions. On Facebook, you can find your old friends without even trying to. You can see a lot of movies which you may be sure that will like them.

During searching in search engines, you can see exactly what you are looking for without even knowing how the learning algorithm is ranking the results to show you the appropriate result.

Your email service finds spam and junk mails automatically and delete them for you and you can be confident that you won't loose any important thing.

OCR systems transform images of books and make them as text files. Using this, you can search in texts and find what you want without searching a lot of images.

Tasks which are difficult for finding algorithms, like walking of robots, are so better than before, using data and machine-learning.

There have been designed agents which can defeat human brain in famous games, like chess. Even in AAA computer games, the agents use reinforcement learning to learn how to defeat the human rivals using experience of each specific player based on their behavior.

Machine learning is EVERYWHERE.


Similarity between 2 statistical distributions

One method is Kolmogorov-Smirnov test. Kolmogorov-Smirnov test checks whether two samples are drawn from the same continuous distribution where sample sizes can be different. It's p-value is close to 0 when two samples follow the same distribution and close to 1 when they do not follow the same distribution. So you can use 1 - (p-value) as a similarity metric.
import numpy as np
from scipy.stats import ks_2samp

np.random.seed(52)

n1 = 200
n2 = 300

mu_1 = 5
mu_2 = 5.1

sigma_1 = 0.3
sigma_2 = 0.2

sample_1 = np.random.normal(mu_1, sigma_1, n1)
sample_2 = np.random.normal(mu_2, sigma_2, n2)

result = ks_2samp(sample_1, sample_2)

print(result.pvalue)

1.4998994601889137e-08

Note that there are also other methods such as Bhattacharyya distance, Kullback–Leibler divergence. Some implementations for Kullback-Leibner can be found also here.


How to handle preprocessing (StandardScaler, LabelEncoder) when using data generator to train?

It is a correct approach to standardize on your training features. In that way, you ensure not to give any information from the testing set to the training set.
About features scaling, if you have too many samples to fit your scaler at once, you could use the partial_fit (see here) method of StandardScaler in sklearn. Load sequentially your training features and do the partial_fit. Once finished, your scaler is ready to be used on your training/testing batches.
About label encoding, either you already have an array containing all the labels, so you can fit your LabelEncoder, or you would have to load sequentially all your data to get all the different labels before fitting (LabelEncoder does not have partial_fit method).


Visualize Softmax values in CNN prediction

Fastest and easiest would be seaborn. It's built on top of matplotlib and you can plot your stuff with little code. Once you become to hate matplotlib's paradigm or need something fancier, there's plotly.


Highly imbalanced class Machine Learning

What I'd do is to keep the Unknown class and treat the Never-worked and Without-pay as Unknown.

Imputation methods would just reinforce the bias towards the most dominant class, i.e. private in your case. Say you used a simple classifier to impute these $21$ values, it would assign them to another class based on the similarity of the other features (e.g. age, education, ...). Think on what this procedure has done though, it has replaced someone with without-pay to private or something else. This is conceptually wrong, though, especially for a model that tries to predict the income of someone.


Why multiply by 2 when calculating partial derivatives during backpropagation?

You're using the Mean Square Error $\Sigma\frac{1}{N}(y-(Wx+b))^2$as the loss function, if you take the derivative, you will have the $2$. In some materials, we will use $\frac{1}{2}\Sigma\frac{1}{N}(y-(Wx+b))^2$ as the loss function to cancel out the $2$. In fact, this doesn't matter at all and it has no impacts on params optimization.


According to the instructions at both Github and Anaconda cloud, you should try

 conda install -c conda-forge boruta_py


If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

No, adding noise will not help to regularise the model.
This won't help your model generalise better to unseen inputs. It will probably just make your model perform generally worse. Any modifications you make to the training instances should be learnable.
Take an image classification problem as an example (images of a cat or not cat): randomly converting pixels to white in every image will not help your model learn generalisable features of cats; random white noise is not a 'learnable' feature. However by rotating, flipping, cropping, or adjusting contrast of the images, and adding these as new training instances, this would force the model to be more tolerant of differences in each image of cats, and therefore generalise better.
There are many methods for regularising models, such as l1 or l2 regularisation in linear regression or neural nets, and also plenty of other methods specific to other model types.
Making sure your training data is as varied as possible, and using model specific regularisation techniques, are probably closer to 'best practise' (though I don't really like the term since 'best practises' often change, such as new regularisation techniques, optimisations etc etc).
Update:
I've since experimented with adding Gaussian noise to input features, and especially in cases with overfit neural nets this can help the model generalise. It's also possible to add GaussianNoise layers directly in a neural net for this purpose, as in the TensorFlow documentation here.


Is it possible to make use of the CPU RAM, if i'm running of of VRAM, in tensorflow?

If I understand correctly you are essentially looking for a way to use the CPU's RAM as a swap for the GPU's RAM. Unfortunately this isn't as easy to accomplish and might require some low level work. So if you're looking for a simple argument to add to your keras model, as far as I know there is none. Some options:

Some older tensorflow APIs supported this functionality (e.g. dynamic_rnn - see swap_memory parameter). However these are now deprecated in favor of the equivalent keras layer, that to my knowledge does not support this functionality. Theoretically you could use compatibility mode to utilize this.
You can specify what device each tensor will be loaded on beforehand. This isn't equivalent to swapping, but you could offload some computation to be regularly performed on the CPU. You can do this with tf.device.
The same idea as above is often applied to loading and preprocessing the data, which is done on the CPU, while everything concerning the training ops is handled by the GPU. This is convenient because you can have a batch being loaded by the CPU at the same time that the model is being trained by the GPU. I know this is far off from what you're asking, because it requires your model to fit in the GPU entirely, however it fits in the general theme of offloading work to the CPU. This is fully supported by tf.data.Dataset.
Use some trickery like gradient checkpoints to reduce the memory footprint of your models (with the expense of computational speed). This does not have anything to do with offloading work to the CPU, but it could help you fit your model into memory.


binary decision tree hyperplane parallel to axis?

Your understanding is fine.

The hyperplane is $x=5$.

In $3$ dimension, it is a plane that is parallel to the the $y$-$z$ plane, which can be written as $x=0$. It is parallel to the $y$-axis and $z$-axis.

You are right that it is perpendicular to the $x$-axis. The $x$-axis is the normal direction for the plane $x=5$.


What version of spark in latest Cloudera QuickStart VirtualBox?

QuickStart VM latest 5.7 has spark version 1.6.0 by default.


High multi class classification with small data set

Yeah. There have been scenarios like that. Imagenet has more than 1000 classes. Same could be said for certain cifar datasets.

If deep learning, go with one-hot encoding for the labels.

Stick with something like multi-class AUC as metric instead of believing on Validation accuracy.

Go with a keras DNN, then you could try advanced architectures, with the last Dense net to have 1880 neurons each representing one label. Also stick with categorical_crossentropy for loss.

Hope this helps.


Machine learning model using keywords for binary decision

The most obvious binary classification task coming from text data is sentiment analysis. Kaggle is plenty of datasets such as this one on movie reviews.

Sentiment classifiers can be either very simple, like logistic regressions on naive Bayes classifiers, or very complex such as the ones based on RNNs and/or part-of-speech tagging models.

If you are interested in news specifically, similar models can be used to estimate news polarization.


Is data partitioning necessary for an explanatory model and why?

An explanatory model is used to identify and explain what causes some particular outcome (i.e. identify/quantify the drivers of effect).

Even though the model will not be used for prediction, it needs to be accurate enough to adequately predict the response so that you can be sure the conclusions you draw from the model are valid. As an extreme example, if your explanatory model does not predict the outcome at all, then all the inferences you make from this model are useless.

So then, how does one validate model accuracy? In most cases, data partitioning and validation on the hold-out sample is used; in your case the validation is more of a sense-check for accuracy.


Using the Stanford Named Entity Tagger in R

https://github.com/statsmaths/coreNLP  can be used as a wrapper for this library in R. Documentation has good examples for most use cases lie NER and POS.


But, in some cases it doesn't work properly.

As you did not provide your data, I can only make a bold assumption.
The perceptron can only classify linear separably data and will not converge for not-linearly separable data. Thus, this could be a reason for the unintended behaviour.

Cheers!


Yes, you could start with k-means on the 100-dim data. Make sure you have normalized your data to perhaps zero mean and unit variance before running k-means.
Use the elbow method (google it) to determine the optimal number of clusters for your data.
By the way you also want to check if the two dimensions are correlated in any way. If it is then you can use PCA to convert the 2 dimensions into a single dimension that explains most of the variance in your data and then cluster using 50-dim per observation.


How to calculate accuracy, precision and recall, and F1 score for a keras sequential model?

if you want to compute this for every class then:

def recall(y_true, y_pred,class_to_analyse):
     pred = K.argmax(y_pred)
     true = K.argmax(y_true)
     p = K.cast(K.equal(pred,class_to_analyse),'int32')
     t = K.cast(K.equal(true,class_to_analyse),'int32')
     # Compute the true positive
     common = K.sum(K.dot(K.reshape(t,(1,-1)),K.reshape(p,(-1,1))))
     # divide by all positives in t
     recall = common/ (K.sum(t) + K.epsilon)
     return recall

def precision(y_true, y_pred,class_to_analyse):
     pred = K.argmax(y_pred)
     true = K.argmax(y_true)
     p = K.cast(K.equal(pred,class_to_analyse),'int32')
     t = K.cast(K.equal(true,class_to_analyse),'int32')
     # Compute the true positive
     common = K.sum(K.dot(K.reshape(t,(1,-1)),K.reshape(p,(-1,1))))
     # divide by all positives in t
     precision = common/ (K.sum(p) + K.epsilon)
     return precision

def fbeta(y_true, y_pred,class_to_analyse):
    beta = 1 # for f1 score
    precision = precision(y_true, y_pred,class_to_analyse)
    recall = recall(y_true, y_pred,class_to_analyse)

    beta_squared = beta ** 2
    return (beta_squared + 1) * (precision * recall) / (beta_squared * precision + recall)

to make it work you need to create a list of function with a fixed class (ex: recall_1(y_true, y_pred) = recall(y_true, y_pred,class_to_analyse = 1)


why use a perceptron when it seems multiple IF staments do the same thing

Machine learning (perceptrons or not) is all about automatically finding generic but correct rules, be it in the form of If-Else-Rules, encoded formulas, closest occurrences, or others.

The ML algorithm is just a way to (automatically) find this knowledge, whatever the representation may be. In another words, you use it to find the someValue of your example, based on your data. You don't need ML if you can represent such knowledge yourself.


How can I use time stamps for classification?

Not knowing your data in detail, my intuition is that you could go for dummy (one hot) encoding. You could split each day in (say) 10 min. intervals $x$ (144 columns) and attach labels (up/down) $y$. Each time interval $x$ would be one dummy encoded column (true/false).

The model would be a binary classification (logistic) like:

$$ y = \beta_0 + \beta_1 x_1 + ... + \beta_n x_n + u .$$

In case you have enough days, this could be a reasonable predictive model. It is likely, that some of the dummy encoded $x$ are irrelevant. So it could be worth to check a Lasso model, where some (not so helpful $x$) are shrunken or automatically excluded from the model.

https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

Based on the same logic, you could also check boosting, e.g. lightGBM to make the classification.


Low training and validation loss but bad predictions

Perhabs your network is overfitting. Overfitting is where networks tuned its parameters perfectly to your training data and therefore it has very low loss on training set. Unfortunately, it will perform badly when new samples are provided within test set. Overfitting is broadly descipted almost everywhere: https://en.wikipedia.org/wiki/Overfitting

What you can do with overfitting:

try neural network with simplier structure, it should help your network to preserve ability to generalize knowledge,
use early stopping; try to measure validation loss at every epoch. And when it gets higher for like 3 epochs in a row - stop network training,
dropout: dropout is simple technique that prevents big networks from overfitting by dropping certains connection in each epochs training then averaging results. It is easy to use because it is implemented in many libraries like Keras or PyTorch.
$l_1$ and $l_2$ regularization: they are modification to your loss function making it involve sum of weights. Minimizing sum of net's weights prevents situation when network is oversensitive to particular inputs.

The other cause for this situation could be bas data division into training, validation and test set. Training and validation set's loss is low - perhabs they are pretty similiar or correlated, so loss function decreases for both of them. Then relation you try to find could by badly represented by samples in training set and it is fit badly. I would check that division too.


Is there any reason you have to use Naive Bayes? While Naive Bayes does handle multi-class modeling quite nicely, sometimes the "naive" assumption that each word in the text is independent of the others is too naive, especially given the size of your training set. While you may be able to increase the accuracy a little bit by doing more text processing, I wouldn't expect to see significant improvement, especially given the training size.

Since Naive Bayes itself doesn't have much parameter tweaking, if you want to try more processing of the data itself to improve performance, you can try things like text count vectorization or TF-IDF vectorization to represent the text data more holistically than just keyword flagging.

If you are able to implement other models, I would take a look at the scikit learn multi-class models for ideas. Tree based methods are also inherently multiclass, and have more parameters to tweak, so implementing something like Random Forests may suit you better.

As for the training size itself, that does seem a little small. Especially if you are using text data, the more features you include, the more data you need to be able to stand a chance of detecting a signal in the data.


Linear Regression on data with bimodal outcome

One option would be use sklearn.compose.TransformedTargetRegressor  to make the dependent variable more normal distributed.


Accuracy is commonly defined as total nbr of correct predictions / total nbr of predictions.
Imagine a binary classification of color in: White or not White.

Since it's binary, all the white Classifications - a True Positive classification are as important for accuracy as a - True Negative classification, after all, the classification is also correct, hence,
TP+TN/ total nbr of predictions.


How to improve performance of Tensorflow's CNN example?

Coming to Neural Networks We can tweak a lot of parameters that can improve performance of the network. Sometimes changing a small parameter value can result in a big performance increase and sometimes we have to tweak a lot to get good performance. This depends on the problem.
In general the following parameters are often important in terms of getting good performance:
Number of layers: change number of layers by adding more layers or removing some layers. Number of layers.
Number of Kernels
Kernel Size
activation function(most people use relu but we can try more like prelu, elu..etc)
weight initialization techniques: try different weight initializers with respect to activation function used.
Try dropout layers which we reduce overfitting
Try Batch Normalization
use pretrained models(Transfer Learning) which can increase performance very much.
These are some of popular parameters on which performance of model depends. Try tweaking them and get a good knowledge of these.


The penalty of both Lasso and Ridge is proportional to the magnitude of the weight. That is, the penalization added to the cost function is $\lambda ||\omega||_2$ or $\lambda ||\omega||_1$.

Wether it is more convenient to apply regularization or feature selection, Lasso already does some feature selection for you, as the estimated weights for Lasso are sparse (there will be many coefficients equal to 0).

About multi-colinearity, Ridge tends to eliminate variables that are colinear, while Lasso doesn't.

All that I have said is taken from An Introduction to Statistical Learning book, from James, Witten, Hastie and Tibshirani.


Modes are just the analog of centroids in k-means. A distributed implementation assigns points to clusters in parallel on subsets of the data, then computes new centroids in parallel, and finally collects those results to perform a (weighted) average of those centroids to get the real centroids. Similarly here, after cluster assignment, modes are computed on subsets of the data in parallel, but then they need to be collected and combined into the real new modes. For whatever reason they're calling the 'real' modes 'metamodes'.


Why use two LSTM layers one after another?

About your first question:
It is because word-by-word NLP model is more complicated than letter-by-letter one, so it needs a more complex network (more hidden units) to be modeled suitably.

About your second question:
When you want to use two-staged LSTMs, the hidden sequence of first LSTM must be used as input of the second LSTM and the return_sequences option is used to do this.


What would be the mse (mean squared error) of my scaled dataset on the original scale?

TLDR: You need to multiply your scaled mse by $(max - min)^2$.

Let's give our values some names: Let $y_n$ denote the dependant variable, $m$ be the number of observations
\begin{align*}
& min := \min\limits_{n} y_n, \\
& max := \max\limits_{n} y_n
\end{align*}
and let $p_n'$ be the (scaled) predicted value for $y_n$. The scaled value $y_n'$ for $y_n$ is given by
$$ y_n' = \frac{y - min}{max - min}. $$
So scaled mse that you calculated is given by
$$ 0.02 = mse_{\text{scaled}} = \frac1m \sum\limits_n |y_n' - p_n'|^2 = \frac1m \sum\limits_n |\frac{y - min}{max - min} - p_n'|. $$
In order to get the true mse, you want to rescale $y_n'$ to $y_n$ and $p_n'$ to $p_n$ using
$$ p_n' =  \frac{p_n - min}{max - min}.$$
This gives us
\begin{align*}
mse_{\text{scaled}}
& = \frac1m \sum\limits_{n} |\frac{y_n - min}{max - min} - \frac{p_n - min}{max - min}|^2 \\
& = \frac1m \sum\limits_{n} |\frac{y_n}{max - min} - \frac{min}{max - min} - \frac{p_n}{max - min} + \frac{min}{max - min}|^2 \\
& = \frac1m \sum\limits_{n} |\frac{y_n}{max - min} - \frac{p_n}{max - min}|^2 \\
& = \frac{1}{(max - min)^2} \frac1m \sum\limits_{n} |y_n - p_n|^2.
\end{align*}
Rearraging this gives us
$$ mse_{\text{original}} = \frac1m \sum\limits_{n} |y_n - p_n|^2 = (max - min)^2 mse_{\text{scaled}}. $$

However, in general, it is better to first rescale your values and then take the mse because you lose interpretability and, as you see, it is more difficult to rescale the mse than to rescale the predicted values.


Converting feature coefficients to probabilities scores for linear models

No, coeff1/sum(coeff) cannot be a probability due to the fact that it can be bigger than 1. For instance:
coeff1 = 1., coeff2 = -.5, then coeff1/sum(coeff) = 2. The interpretation of coefficients in linear regression is the derivative of the output variable with respect to the given predictor.


Calcualte distance?

Distance = data2 - data1 = [(1,0), (2,0), (3, 7), (4, 7), (5, 7), (6, 0)]

And what will you treat as the same wave?

Any three (or more) nearby points that have the same distance...


Case when Out of Bag Error and Test error differs a lot in Random Forest

What type of sampling method did you use to split the data? I'm guessing, by looking at the ratio of classes in your OOB confusion matrix and your test set confusion matrix that it wasn't stratified, is this correct? For instance, the class ratio in the OOB data is 4000:800 whereas in the test set it is 7761:43. As a starting point, you may want to split your data using stratified sampling so that the classes are fairly represented (in a statistical sense) in both your training data and test data. The ratio of classes in the test set should be identical or almost the same as the ratio used in the training data.

I think this would be a good starting point to figuring out the problem.


There may be two causes to your problem:

In validation the network needs more memory.
There is another problem not directly related to this.

In this Keras issue you can find a discussion of a very similar problem. Basically, you can try:

Reducing the batch size.
If you are using Tensorboad, try disabling it or setting its batch_size parameter.


Is it possible to customize the activation function in scikit-learn's MLPClassifier?

Although @Emre's comment that sklearn wasn't specifically made for the task is correct, nothing stops you from introducing another activation function by implementing it in sklearn/neural_network/_base.py. You should be able to find the file in your installation.

To this end, you would define your single-parameter function and add it in the ACTVATIONS list. Refer to the existing implementations for some guidance how to do it.


How is the output of a maxpool layer window size=1x2 and stride=2 calculated?

According to the paper, maybe "s" represents stride in row, while the stride in column equals 1.


LSTM is often used for Sequence Prediction problems, for example, when the dataset is a time series.

In that kind of dataset, you don't split the dataset the normal (a.k.a random) way to avoid look-ahead bias. We should train with "old observations" and test with "new observations" as you said.

  Should, in the case of LSTMs, the train test (in the toy example above) be 10 000 observations long (i.e. 9 000 old 'train' observations and 1 000 new 'test' ones)?

This sentence is a little bit unclear to me. If you have a dataset with 10 000 observations, you can train with 9000 old observations and test with 1000 new observations. The training set shouldn't contain the test data in any case.

Here's an example of how to split a time series dataset:
We have data from 01-01-2000 to 31-12-2004, we can choose 01-01-2004 as a split day:

Training set: data from 01-01-2000 to 31-12-2003
Test set: data from 01-01-2004 to 31-12-2004

In this case, no knowledge from the future is used in the training phase and we can use the model to predict the data in the test set.


Training an ensemble of small neural networks efficiently in TensorFlow 2

I have a problem with this approach, though I'm up for discussion about it and might be wrong.

Idea: You should train every sub-model on the main task, not the ensemble. For that, you can utilise your GPU to actually calculate everything in parallel (Even with a 5-dimensional input, effectively encoding 5 batches at once, further enhancing the independent training of the sub-models). To do this, my quick bet would be to just have five independent outputs, all optimised with a weight of 1/5th on the same loss you use for the averaging function. Then, do the averaging afterwards, without further optimisation.

Motivation: The fact that you have the averaging layer inside of the model as the output layer means, that the loss differentiation takes place there as well. This means, that during training, you allow interdependancies inbetween the single sub-networks. After short consideration it should be clear that this is not really what your motivation suggests:

  [The motivation for using this kind of ensemble is the following: Especially with small datasets, each of the small networks will yield different results because of different random initializations.

The small networks don't yield anything, because you don't measure and optimise for their yield. In short, you just built a bigger model with a slightly more complicated connectivity.


Needed to normalise the data which encodes the column and then denormalise on the output which gives the correct classification.


1) Whether is there any other consideration for determining the number of predictors in logistic regression model?

The right number of predictors depends on your data and on your theories about data, and on that only. All these rules of thumb seem completely arbitrary to me, and they lack any scientific ground.

  2) How do people calculate min sample size required based on this?

What do you mean with "min sample size"?

EDIT: Of course, the number of explanatory variables cannot be larger that the number of observations. Based on this answer, a rulo of thumb is to use at least 10-20 observations for each variable. I wish also to stress one point: they must be useful variables, i.e. variables with actual explanatory power. If a variable is a linear combination of others, it won't improve the model by any means, and statistical softwares such as R would delete one automatically.

However, my suggestion is to try to always employ all the data available, don't just stick to a minimum threshold.


In handwritten digit recognition problem using logistic regression, what changes needed to add another class "Not a Digit"

Logistic regression is normally used to perform binary classification, which answers a yes or no question, e.g.:

Is this an 8 or not?
Will it rain today or not?

Perhaps I have misunderstood your explanation, but it sounds like you are trying to perform multi-class classification, i.e. to classify an image as one of a certain number of options. So a single image must contain (be classified as) a number from the list: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].

Usually, to add an extra class to a model that already does this, you would need to adjust the final output of the model to predict a one-hot vector that is simply one element longer!

In the specific case of the Scikit-Learn LogosticRegression class, it seems as though you don't need to specify anything - the class will automatically use a multinomial model and relevant optimiser as soon as it sees that data is not binary (i.e. a yes-no model as explained above).

Have a look at this official tutorial, which should a multinomial model trained on a dataset with a target variable that has a total of 20 classes. The number of features (n_classes is equal to 20) is not passed to the model at all, it is inferred.


I used Stargazer in Python. How to convert output to LaTex?

In order to convert the statsmodels (single model) summary to latex, you will use summary.as_latex(), as in model_results.summary().as_latex().

If instead you want stargazed-type tables (regrouping multiple models), you will want to use the corresponding python package: https://pypi.org/project/stargazer


How do I interpret the given classification report?

Precision is the proportion of predictions of that class that are true. So 98% of the predictions for each of your classes are actually of the predicted class, and 2% are actually of the opposite class. Recall is the proportion of the true positives that are identified as such. This means that your model is correctly identifying 100% of the class 0s, but only 72% of the class 1s.

F1-Score is a kind of average of the two; it's an attempt to provide a unified figure of the model's performance, but personally I consider it less useful than the separate figures. It's calculated via the formula 2 x ((precision x recall) / (precision + recall)).

Wikipedia's pages on these metrics are comprehensive:

https://en.wikipedia.org/wiki/Precision_and_recall
https://en.wikipedia.org/wiki/F1_score


You might try Azure Table Storage.  Since you can't lock yourself down to a specific schema (since one data product might be aggregates whereas another might be time series or something else), Azure Table storage would give you the flexibility of storing data from multiple sources, each having their own format.

This would also lend itself to making a system highly scalable, as you could use Azure Service Bus in conjunction with Azure Table Storage.

You might check out this tutorial at Pluralsight, Applied Windows Azure, as it shows a number of examples, one using Table Storage and Service Bus, another using Hadoop, and I suspect that some of these might match the extensibility you are looking for.


After having a look at the article you quote of KDNuggets, in this article they analyze the data from this last year (poll), but they have been doing that for several past years.
Perhaps by looking at past years you can extract some better data.
Also in this other forum https://opendata.stackexchange.com/
they talk about open data, maybe here you can get a more precise answer


How do I create a data set that has a set of features for multiple options, with one option being the expected outcome?

The final feature vector would be a concatenation like (for multi-class prediction):

  Question google count | option A google count | option B google count | option C google count | option C no. words | option A no. words | other features | label
  (1, 2, 3)

There is no need to put features related to option A close to each other (or in any particular order), they just need to be on the same column for all rows regardless of the label.

XGBoost parameters for multi-class classification are:

'objective': 'multi:softprob',
'num_class': 3


how can I recognize multiple faces from one image in python

In general, you perform object detection to find potential faces, and then you perform face recognition on the face(s) detected.

This article walks through the various steps quite well:
https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78

The author also has a Python library that does recognition of multiple faces in a scene with video examples:
https://github.com/ageitgey/face_recognition


How do attention mechanism in CNN for images?

I would suggest you refer to the paper by Hu et al, Show, Attend and Tell: Neural Image Caption Generation with Visual Attention.

The brief idea is that the network learns the 'areas' to focus on that are on the feature maps (the last layer of the feature detectors) which can be in return mapped back to a certain location on the image.
It is not a complex topic but requires some knowledge in RNNs and LSTMs as well that help with sequential data. Best of luck.


The problem you are facing is a time series problem.
Your events are categorial which is a specific case (so most common techniques like arima and Fourier transform are irrelevant).

Before getting into the analysis, try to find out whether the events among nodes are independent. If they are independent, you can break them into sequences per node and analyze them. If they are not (e.g., "Main power supply has a fault alarm" on node x indicates the same event on node y) you should the combined sequence. Sometime even when the the sequence are dependent you can gain from using the per node sequence as extra data.

You data set is quite large, which means that computation will take time. You data is probably noisy, so you will probably have some mistakes. Therefore, I recommend advancing in small steps from simple models to more complex ones.

Start with descriptive statistics, just to explore the data. How many events do you have? How common are they? What is the probability of the events that you try to predict? Can you remove some of the events as meaningless (e.g., by using domain knowledge)?

In case you have domain knowledge that indicates that recent events are the important ones, I would have try predicting based on the n last events. Start with 1 and grow slowly since the number of combinations will grow very fast and the number of samples you will have for each combination will become small and might introduce errors.

Incase that the important event are not recent, try to condition on these events in the past.

In most cases such simple model will help you get a bit above the baseline but not too much. Then you will need more complex models. I recommend using association rules that fit your case and have plenty of implementations.

You can further advance more but try these technique first.

The techniques mentioned before will give you a model the will predict the probability that a node will be down, answering your question (ii). Running it on the sequence of the nodes will enable you to predict the number of nodes that will fail answering question (i).


The goal of activation functions is to introduce non-linearity in the neural network, so it can learn beyond linear relations between input and outputs.
As stated in your image, each neuron output is a linear combination of inputs + a bias, and then all this is sent into a non-linear function (activation function). This allow the network to learn complex relations between input ans outputs.
Most used activation function is ReLU, which is not a probability as it can return a value greater than 1.


TensorFlow is a tool to write computation using data flow graphs; this being said, if you want your app to use a pre-trained model only, there is no requirements to use TensorFlow specifically. You could even use one of the ML library written in Javascript to import and run the pre-trained model.

Javascript DL libs: http://cs.stanford.edu/people/karpathy/convnetjs/, https://github.com/dmlc/mxnet.js/
TensorFlow for mobile: https://www.tensorflow.org/mobile/
Caffe Android: https://github.com/sh1r0/caffe-android-lib
DL4J Android: https://deeplearning4j.org/android


Predictive models with class value belonging to a set of observations

Looks like you have two problems: time-dependant predictive modeling and feature engineering.

1) Time-dependant data

Key words:

  create a model which should be able to predict the future level of a rider in professional cycling

The future Level of the Rider.

That means there is current Level. And Level in the past. And the history of Level change in time for each Rider.

The problem you are trying to solve is time dependant. Your source data can look like:

2) Build a variable to predict

Second. You should build the target feature. This Rider Level (Class).

As far as I understood:

the overall rider 'level' can be a function of his 'classes' in each particular race types
and... the race 'observation' class is a kind of function of {Category, Race Type, Parcours, Distance, Position}

So the only thing you realy need to predict is race result.


Training is not stable with extreme class imbalance

There are probably many different strategies but it's a difficult problem when the imbalance is as severe as it is here.
Without any correction the model is likely to ignore the smallest classes, as you noticed. However forcing the class weight as if the data is balanced is certainly too strong a correction. A middle ground would be to resample the training set instances yourself before fitting the model: by trying different ways to undersample the large classes and/or oversample the small classes you should be able to find an optimal tradeoff between the two extremes (use a separate validation set to determine the optimal combination).

Is there a way to ensure the model predicts approximately the same proportion of samples for each category?

Maybe I misunderstood but this looks like a bad idea: if the true proportions are not equal then the model shouldn't predict equal proportions either. The ideal scenario is for the model to predict the correct label every time, which implies predicting the true proportion for every class.
It might also be useful to analyze the performance in simpler configurations, e.g. by picking a few "average size" classes and observing how well the classifier discriminates between them only. The harder it is for a classifier to predict correctly, the more it relies on basic class proportion since it doesn't know any better.


The answer to your question can be found at: https://stackoverflow.com/a/45997893/5312422

  To vectorize text with skip-grams in scikit-learn simply passing the
  skip gram tokens as the vocabulary to CountVectorizer will not work.
  You need to modify the way tokens are processed which can be done with
  a custom analyzer. Below is an example vectorizer that produces
  1-skip-2-grams,

  from toolz import itertoolz, compose from toolz.curried import map as
  cmap, sliding_window, pluck from sklearn.feature_extraction.text
  import CountVectorizer

class SkipGramVectorizer(CountVectorizer):
    def build_analyzer(self):
        preprocess = self.build_preprocessor()
        stop_words = self.get_stop_words()
        tokenize = self.build_tokenizer()
        return lambda doc: self._word_skip_grams(
                compose(tokenize, preprocess, self.decode)(doc),
                stop_words)

    def _word_skip_grams(self, tokens, stop_words=None):
        # handle stop words
        if stop_words is not None:
            tokens = [w for w in tokens if w not in stop_words]

        return compose(cmap(' '.join), pluck([0, 2]), sliding_window(3))(tokens)

  For instance, on this Wikipedia example,

text = ['the rain in Spain falls mainly on the plain']

vect = SkipGramVectorizer()
vect.fit(text)
vect.get_feature_names()

  this would vectorizer would yield the following tokens,

['falls on',  'in falls',  'mainly the',  'on plain',  'rain spain',
'spain mainly',  'the in']


Detecting synchronization cascades in time series

One option is:

Threshold at ~10 to remove the baseline values.
Fit a clustering algorithm to identify groups.
For each cluster, fit a linear regression. If the linear regression has certain a slope label cluster as a synchronization event.


What layer to consider for embeddings when building a encoder-encoder model?

You either of them:

If you use the output of the LSTM, you will have contextual embeddings, like BERT or ELMo. This means that each time you want to use your embeddings to encode some input text, you need to pass the input text to your encoder and take the output of the LSTM.
If you use the embedding table, you will have non-contextual embeddings, like word2vec. This means that each time you want to use your embeddings to encode some input text, you just need to look up the appropriate vector in the embedding table.


What you are looking for is called "stochastic optimization". You don't need to fit separate models and then combine them.


Conditional attributes in a prediction problem

Decision trees can handle conditions based on several features by design: if the model determines that the feature "at home" is important for the prediction, it will create a node based on this condition close to the root of the tree. By construction other features will appear as conjuncts, e.g. "if at-home==true and featureX==valueX and ... then predict ..." (that's just how a decision tree works).

In case you consider that this particular feature is so important that everything else should depend on it, another option is to train two distinct models: one for "at home" and one for "away". The disadvantage is that a model can only use the instances relevant for its case, so there is less training data for each of them.


I would view my points as elements of $\mathbb{R}^{n+1}$ and use PCA to project my points to $\mathbb{R}^3$ and then scale them to $S^2$.


What are some general tips to improve my MNIST classifier?

You may find this OpenML interesting
In there several benchmarks on different datasets are shown so that you can see how different models score across the same dataset
For your task MNIST the plot shows:

that might give a good idea of how well your model is performing as compared to other implementations/models
Metric button in the upper left corner will allow you to compare in terms of classification metrics and also in terms of execution time


Is R2 score a reasonable regression measure on huge datasets?

The coefficient of determination $r^2$ is defined in terms of variance: it is the proportion of variance in the dependent variable that is explained by the independent variable. Variance is a property of normal distributed data. Hence, the coefficient of determination can only be used when you assume that  both the dependent and independent variables are normally distributed.

Just like other properties of normal distributed data, the estimation of $r^2$ improves when the amount of data increases. With very little data a coincidental colinearity might be the case, but this is not possible with large amounts of data.

Back to your example. Your data is cleary not normal distributed, it has right-skew and has large outliers. For this reason it is not advisable to use   $r^2$. Imagine for example that in the lower left corner (where the majority of the data is) you would observe a negative trend, but in general there is a positive trend. The regression line would be the same and the $r^2$ would be in the same range. This is known as the Simpsons paradox.

In short, if your data is normal distributed you can use $r^2$ for any size of dataset. If it is not normal distributed you cannot use $r^2$.


How neural style transfer work in pytorch?

Question 1

  My question is how the second method maintained the height and widght of style tensor (300, 374)?

Applying the second piece of code does not call all layers but only the ones listed in the layers dict. In contrast to that the first method runs through all layers. Now, if you look at the network architecture:

IN: model
OUT:
Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU(inplace=True)
  (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (3): ReLU(inplace=True)
  (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (6): ReLU(inplace=True)
  (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (8): ReLU(inplace=True)
  (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (11): ReLU(inplace=True)
  (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (13): ReLU(inplace=True)
  (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (15): ReLU(inplace=True)
  (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (17): ReLU(inplace=True)
  (18): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (19): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (20): ReLU(inplace=True)
  (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (22): ReLU(inplace=True)
  (23): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (24): ReLU(inplace=True)
  (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (26): ReLU(inplace=True)
  (27): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (29): ReLU(inplace=True)
  (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (31): ReLU(inplace=True)
  (32): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (33): ReLU(inplace=True)
  (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (35): ReLU(inplace=True)
  (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)

You can see that in fact all max pooling layers are being skipped with the second method. Accordingly, your input does not get pooled, i.e. height and width remain unchanged. (the CNN layers in this network do not change height and width but only the channels)

If you slightly change the second piece of code to:

x = style
x = x.unsqueeze(0)
for name,layer in model._modules.items():
    x = layer(x)
    if name in layers:
        print(x.shape)

It runs through all layers again and gives:

torch.Size([1, 64, 300, 374])
torch.Size([1, 128, 150, 187])
torch.Size([1, 256, 75, 93])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 37, 46])
torch.Size([1, 512, 18, 23])

Question 2

  Why is the optimizer initialization in neural style transfer different from other neural network tutorials?

In image style transfer learning you do not optimize the weights (model parameters) but the target image. Since you would like the target image to resemble your content and style pictures as close as possible. All weights (model parameters) are being kept constant. Therefore, the optimizer needs to receive the target image as an input and provide updates for it (and not for your model parameters like usually).

Question 3

  What is reason behind this optimizer = torch.optim.Adam([target],lr=0.007) >

See Question 2. The target is initialized in line 69. Typically, you use the content image as a start:

target = content.clone().requires_grad_(True).to(device)

As you can see it also activates the gradient for the target.
And this target image is then being optimized in order to minimize the loss
total_loss = content_wt*content_loss + style_wt*style_loss.

A good read on the general approach is Image Style Transfer Using Convolutional Neural Networks.


Using Image Data along with CSV file data for CNN model

Kera's functional API allows arbitrary models to be combined.

One option is to define a "main input" as the image that feeds into a CNN and then define an "auxiliary input" that feeds into a MLP. Both of those sub-models can feed to the same higher layers. The weights of the entire model will be updated during training. The model will learn how to weight the data from csv compared to the image data (in contrast to you picking the relative weights).


Error not decreasing in a 3 layer deep CNN using TensorFlow

Two things:

You're squashing the outputs via the sigmoid function before calculating the loss via tf.nn.sigmoid_cross_entropy_with_logits. This loss function takes the class logits as inputs, meaning you should be passing in the output from the linear layer without any non-linearity activation functions applied after. NOTE that applying a sigmoid function to the logits is a regularization technique used to diminish the effect of the model outputting very large logits, but in a simple three layer CNN like you've got going on here, using this technique is going to hurt more than help.
You should be annealing the learning rate (rather than using a constant learning rate every epoch). As optimization starts to converge on a set of parameters, the parameter update magnitudes should get smaller and smaller otherwise you're very likely to jump past and around where you want to be.

Out of curiosity, what accuracy/loss are you getting?


First of all you have to realize these kind of problems have large amounts of noise compared to signal, because predicting what someone will buy based on a very small window of information is difficult. That said, you are throwing away a lot of information with your current approach. Temporal aspects include a ton of information, for example the sequence in which items were bought etcetera. While this is a lot more complicated than what you are describing now, you could look into recurrent neural networks where you feed history up to the point of prediction as a sequence and predict the item they will buy next as softmax classification. This will depend on the amount of products that you offer whether this is feasible or not. Another advantage is that so-called 'out-of-core' training is relatively easy with neural networks due to the iterative training of batches. Multi-label is also clean, you can just add a number of labels at the end of your graph if necessary.


Python: Handling imbalance Classes in python Machine Learning

This paper suggests using ranking (I wrote it). Instead of using, for instance, SVM directly, you would use RankSVM. Since rankers compare observation against observation, training is necessarily balanced. There are two "buts" however: training is much slower, and, in the end, what these models do is rank your observations from how likely they are to belong to one class to how likely they are to belong to another so you need to apply a threshold afterwards.

If you are going to use pre-processing to fix your imbalance I would suggest you look into MetaCost. This algorithm involves building a bagging of models and then changing the class priors to make them balanced based on the hard to predict cases. It is very elegant. The cool thing about methods like SMOTE is that by fabricating new observations, you might making small datasets more robust.

Anyhow, even though I wrote some things on class imbalance, I am still skeptic that it is an important problem in the real world. I would think it is very uncommon that you have imbalance priors in your training set, but balanced priors in your real world data. Do you? What usually happens is that type I errors are different than type II errors and I would bet most people would be better off using a cost matrix, which most training methods accept or you can apply it by pre-processing using MetaCost or SMOTE. I think many times "fixing imbalance" is short to "I do not want to bother thinking about the relative trade-off between type I and II errors."

Addendum:

  I tried for in-built python algorithms like Adaboost, GradientBoost
  techniques using sklearn. I read these algorithms are for handling
  imbalance class.

AdaBoost gives better results for class imbalance when you initialize the weight distribution with imbalance in mind. I can dig the thesis where I read this if you want.

Anyhow, of course, those methods won't give good accuracies. Do you have class imbalance in both your training and your validation dataset? You should use metrics such as F1 score, or pass a cost matrix to the accuracy function. "Fixing" class imbalance is when your priors are different in your training and your validation cases.


How to include lifecycle configuration in Sagemaker studio's user notebook

Original source
You can follow this tutorial:
https://modelpredict.com/sagemaker-stop-your-instances-when-idle/
Summary
Basically, you have to go to the notebook instance where you want to apply the Lifecycle Configuration, stop it, and go to: Additional configuration > Lifecycle Configuration > [Select your script].
After this, you have to configure the proper permissions for the Lifecycle Configuration script to stop your notebook instance. To do so, create a new policy, and paste the following code:
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
              "sagemaker:StopNotebookInstance",
              "sagemaker:DescribeNotebookInstance"
          ],
          "Resource": "*"
      }
  ]
}

After this, attach it to your Notebook instance and it's done!


Intractability in Variational Autoencoders

I believe that you got bogged down by this thought:

  I understand that it is not possible to analytically compute this integral to optimize for $\theta$.

From the paper I understood that:

$$
p_\theta (x) = \int p_{\theta} (z) p_{\theta} (x|z) dz
$$

Can be analytically solved if we simplify $p_{\theta} (x|z)$.  i.e. in the case where the Bayes' Rule for its posterior is easy.  This integral is instead intractable if the Bayes Rule for $p_{\theta} (x|z)$ turn out to be a very complex function, e.g. $(z|x)$ is very complex.  In general integrating a simple distribution is easy, conditional distributions can be (very) complex to integrate.

The reparametization trick on the other hand is what makes the paper's point.  They do not try to compute the the integrals as you derived them but they attempt to substitute all $(z|x)$ with a $g_{\phi}$ function (parametrized by $\phi$) which will build $z$ from $x$ and $\epsilon$.

The idea is that we choose $p(\epsilon)$ and instead of estimating $q_{\theta}(z|x)$ we estimate $\phi$ (in $g_{\phi}(z_{parametrized}, x$).  Estimating (SGD) a value for a parametrization of a vector function is not difficult.

I understood (analogically and very lousily) the reparametrization they do in not a very different way than what we do with, say, integrals of sphere or cylinder-like surfaces: we reparametrize the variables and recalculate the interval. Here we want the interval to not have the two sums of the KL (including both distributions) but one sum.


which deep learning text classifier is good for health data

Yes, you should split the paragraph to sentences and give those sentences to the model. Your deep structure should be like this:

In the first layer, you must put a word embedding layer to represent a sentence as a sequence of vectors. In the second layer, you must put LSTM to be able to model your sequence vector as a single vector. Now, you can add successive layers with linear, relu or sigmoid activation functions to make your model deeper. In the last layer, you must use sigmoid activation function to do binary classification.


Micro-F1 and Macro-F1 are equal in binary classification and I don't know why

The difference between macro and micro averaging for performance metrics (such as the F1-score) is that macro weighs each class equally whereas micro weights each sample equally. If the distribution of classes is symmetrical (i.e. you have an equal number of samples for each class), then macro and micro will result in the same score.

As an example for your binary classification problem, say we get a F1-score of 0.7 for class 1 and 0.5 for class 2. Using macro averaging, we'd simply average those two scores to get an overall score for your classifier of 0.6, this would be the same no matter how the samples are distributed between the two classes.

If you were using micro averaging, then it would matter what the distribution was. Say that class 1 made up 80% of your data, the formula would then be 0.7*80% + 0.5*(100%-80%) which would equal 0.66, since each sample is weighed equally and as a result the score is representative of the data imbalance. If class 1 made up 50% of your data, the formula would shift to 0.7*50% + 0.5*(100%-50%) which would be 0.6, the same as the result from macro averaging.

If your data was perfectly balanced, then macro and micro averaging will both result in the same score. If not, there's still a chance that they result in the same score depending on the exact distribution of scores (or if your estimator has the same performance for all classes involved).


Implication of Central Limit theorem and its application in machine Learning

Are you familiar with hypothesis testing? Confidence Intervals ? Statistical Significance ? All of these concepts typically rely on the central limit theorem.
One concrete example I can think of relating to ML: Regression
Most approaches to regression typically assume that the error term that appears follows a Gaussian distribution. How do we justify this assumption ? Well the idea is that we can think of the error term as a sum of multiple different error terms.
Now even if these individual error terms do not follow a Gaussian distribution, we can model their sum as a Gaussian because of the Central Limit Theorem.


How to use class_weights with one-hot encoding in keras?

I'm not allowed to comment, but have you tried using the numpy array that you get from class_weight.compute_class_weight(), rather than converting it to a dict? I've always skipped that part, and in your case I would say class_weight=class_weights. Sorry if I'm suggesting something you already ruled out. Good luck.


Good formula for turning star reviews into upvotes

My recommendation would be to explore some statistical approach to represent the reviews/rating pair as significance. For example, to translate the [# reviews, rating] into some test statistic type model, such as Student's t for starters.

Considering your example numbers, some approaches can be:

1)

>>> 5/(1/sqrt(10))
15.811388300841896
>>> 3/(1/sqrt(10))
9.4868329805051381
>>>
>>>
>>> 4/(1/sqrt(60))
30.98386676965934
>>> 5/(1/sqrt(10))
15.811388300841896

2) Or, diving deeper into stats,:

>>> (5-3)/(sqrt(1/10+1/10))
4.4721359549995796
>>> (4-5)/(sqrt(1/60+1/10))
-2.9277002188455996

(you will need to do some work on the alpha level to get at the significance in ex 2 above)

You can see how these work out for you; if too basic/too many inaccurate assumptions/some other filtering that you need more resolution in your model/etc., you can explore ways to better represent:

the sample standard deviation (assumed to be 1 in my examples above), or
the distribution (assumed Gaussian in examples above), or
the appropriate statistical test (plow through Wikipedia), or
etc

Point is you can continuously refine your model, depending how much resolution you need. Only you can make that call. Hope this helps!


How to use ndcg metric for binary relevance

The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same score of 1, and then it wouldn't make much sense to apply the nDCG penalty discounts.
A similar measure often used with binary relevance scores is the mean average precision defined as:
$\text{MAP}=\frac{\sum_{q=1}^Q\text{AveP}(q)}{Q}$,
where $Q$ is the number of queries.
A comprehensive explanation of both nDCG and MAP is available here


Hardware requirements for Linux server to run R & RStudio

There is no ideal configuration, for R or in general - product selection is always a difficult task and many factors are at play. I think that the solution is rather simple - get the best computer that your budget allows.

Having said that, since you want to focus on R development and one of R's pressing issues is its critical dependence on the amount of available physical memory (RAM), I would suggest favoring more RAM to other parameters. The second most important parameter, in my opinion, would be number of cores (or processors - see details below), due to your potential multiprocessing focus. Finally, the two next most important criteria I'd pay attention to would be compatibility with Linux and system/manufacturer's quality.

As far as the storage goes, I suggest considering solid state drives (SSD), if you'd rather prefer to have a bit more more speed than more space (however, if your work will involve intensive disk operations, you might want to investigate the issue of SSD reliability or consult with people, knowledgeable in this matter). However, I think that for R-focused work, disk operations are much less critical than memory ones, as I've mentioned above.

When choosing a specific Linux distribution, I suggest using a well-supported one, such as Debian or, even better, Ubuntu (if you care more about support, choose their LTS version). I'd rather not buy parts and assemble custom box, but some people would definitely prefer that route - for that you really need to know hardware well, but potential compatibility could still be an issue. The next paragraph provides some examples for both commercial-off-the-shelf (COTS) and custom solutions.

Should you be interested in the custom system route, this discussion might be worth reading, as it contains some interesting pricing numbers (just to get an idea of potential savings) and also sheds some light on multiprocessor vs. multi-core alternatives (obviously, the context is different, but nevertheless could be useful). As I said, I would go the COTS route, mainly due to reliability and compatibility issues. In terms of single-processor multi-core systems, your budget is more than enough. However, when we go to multiprocessor workstations (I'm not even talking about servers), even two-processor configurations can go over your budget easily. Some, not far away, such as HP Z820 Workstation. It starts from 2439 USD, but in minimal configuration. When you upgrade it to match your desired specs (if it's even possible), I'm sure that we'll be talking about 5K USD price range (extrapolating from the series' higher-level models). What I like about HP Z820, though, is the fact that this system is Ubuntu certified. Considering system compatibility and assuming your desire to run Ubuntu, the best way to approach your problem is to go through Ubuntu-certified hardware lists and shortlist systems that you like. Just for the sake of completeness, take a look at this interesting multiprocessor system, which in compatible configuration might cost less than from HP or other major vendors. However, it's multimedia-oriented as well as it's reliability and compatibility are unknown, not to mention that it's way over your specified budget.

In terms of R and R-focused software, I highly recommend you to use RStudio Server instead of RStudio, as that will provide you with an opportunity to be able to work from any Internet-enabled location (provided you computer will be running, obviously). Another advice that I have is to keep an eye on alternative R distributions. I'm not talking about commercial expensive ones, but about emerging open source projects, such as pqR: http://www.pqr-project.org. Will update as needed. I hope this is helpful.


Couple of Things straight up:

Inject some noise in the process. When the gan or autoencoder learns that there is some noise it will start to generalise better

Use weaker architectures. (Analogy to weak learners, you cant build random forest wit hstrong trees). Basically allowing for your Architecture to be weak enough to be able to generalise and not learn everything by heart, i.e. map Input to Output.


How to ensemble classifier incorporating all features in python?

Boosting - is the ensemble which tries to add new models that do well where previous models lack. Bagging in scikit lets you send the base classifier as the parameter. Go through 1.11 of the link to understand more!

But since, you already have in mind that SVM performs better Voting Classifier which is present in sklearn.ensemble lets you give weights to the classifiers which you seem to be performing well. For instance in your question, SVM can be given more weight. It also has another parameter 'voting'. If hard, uses predicted class labels for majority rule voting else if soft, predicts the class label based on the argmax of the sums of the predicted probabilities.


Neural network back propagation gradient descent calculus

We have $$\hat{y}_{\color{blue}k}=h_1W_{k1}^o + h_2W_{k2}^o$$

If we let $\hat{y} = (\hat{y}_1, \ldots, \hat{y}_K)^T$, $W_1^o=(W_{11}, \ldots, W_{K1})^T$, and $W_2^o=(W_{12}, \ldots, W_{K2})^T$

Then we have $$\hat{y}=h_1W_1^o+h_2W_2^o$$

I believe you are performing a regression, $$J(w) = \frac12 \|\hat{y}-y\|^2=\frac12\sum_{k=1}^K(\hat{y}_k-y_k)^2$$

It is possible to weight individual term as well depending on applications.


How to get sentence embedding using BERT?

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT.
They also have a github repo which is easy to work with.


Why does "classification+regression" work better than "classification" in my NN?

You are using accuracy to estimate the performance of your model, which is a bad idea. You can't understand the limitations and advantages of your model using accuracy as a evaluation metric. Consider the following situations

Your model predicts 36 instead of 37
Your model predicts 106 instead of 37

Accuracy as a evaluation metric, treats both of them in the same way. I'd suggest you to go with MAE or RMSE for evaluation. You can train the network either using classification or regression, but make sure that you evaluate it using the right metric.

Btw, the multihead idea is really good.


Is LASSO regression implemented in Statsmodels?

The question-asker resorted to scikit-learn until now, but statsmodels has just come out with its implementation of Lasso regression. The docs here are pretty self-explanatory and concise.


Are there any projects to work on the Internet to fight against COVID?

You might want to have a look at this Kaggle initiative and this github repository with South Korea data


Classification accuracy based on top 3 most likely classifications

Yes, it's common to consider that the prediction is made of multiple answers (typically top N most relevant answers) and use a performance measure based on that.

Currently you're treating the problem as a classification problem but logically this is more like a recommendation problem or an information retrieval problem (like results from a search engine). Usually for this kind of problem the gold answer would also consists of a list of several items, but apparently your dataset contains a single answer for every instance.

Answer to comment: a couple of papers using some top N performance measures (note: it's just a quick selection based on the keyword "information retrieval")

https://www.microsoft.com/en-us/research/publication/letor-benchmark-collection-research-learning-rank-information-retrieval/
http://informationr.net/ir/18-2/paper582.html

The CLEF series of Shared Tasks have proposed many datasets and evaluation measures across the years, it's probably a good source for resources and papers... if you have a bit of time to explore it ;)


Is it possible to train this image classifier?

For sure, is a really feasible application! Indeed, there does exist Deep Learning models for that.

I suggest you to start by doing transfer learning on a pre-trained model like this: https://github.com/satojkovic/DeepLogo Typically you remove the last layers of the network and add layers that adapt to your data. Then, train these last new layers with your data (freezing the layers you have kept from the original model).

Another good advice is to perform data augmentantion on your dataset to include ligth, scale and rotation variants of your images.

Summary:

It's a feasible task
Check out pre-build models
Learn about transfer learning on pre-trained models
Learn about data augmentation to include robustness against different scenarios and env. conditions


Unsupervised Learning - Using the Outcome of Learning

Unsupervised Learning is a collection of tools and you can use those tools for many purposes.
Sometime, it's just for data visualisation / data exploration (see UMAP for example), to better understand the problem and get the idea how to tackle it. Sometimes it will lead to business decision like building processes for a specific cluster or building submodels for specific clusters.
Sometime it helps deal with sub-problems of data-science, like outliers identification (see Hdbscan or isolation forests for example). What to do with those outliers depends on your business case.
Sometimes it helps define intermediate 'targets' to perform semi-supervised or supervised learning, like using some self-organisaed map, identify a cluster and use that cluster as a target. It help when you are not sure you have identified all your positive targets or when you don't wnat to label everything.


Joining tables from different locations in Bigquery

You can't join two tables in different data set which are in different locations. As an alternative solution, you can copy datasets between regions using BigQuery Data Transfer Service. Here is the documentation link Copying Datasets:
bq mk --transfer_config \
      --project_id=myproject \
      --data_source=cross_region_copy \
      --target_dataset=dataset_us \
      --display_name='Copy Dataset' \
      --params='{"source_dataset_id":"dataset_eu","source_project_id":"myproject"}'


Manual feature engineering based on the output

No you should not do this, it is causing a data leak. Data leaks happen when the data you are using to train a machine learning algorithm happens to have the information you are trying to predict.

It will give your model information about your test data during training. It will cause your test scores to be overly optimistic and will make the model worse at generalizing to totally unseen data.

Good resource on data leakage


How does PCA and clustering treat different type of predictors where order of their values signify the result differently?

Similar combinations of values will be clustered together regardless of their meaning in your model.  The meaning of the clusters is for the model builder to determine.

You could engineer another feature (call it ‘goodness’) that takes the GDP and divides by child mortality (for instance) in order to add more variables to determine clusters, by dividing GDP by mortality counties with lower GDP and higher mortality rates would have a smaller “goodness” value while higher GDP and lower mortality would have higher “goodness” values and then would give the model a better chance to cluster on this factor.


A3C - Turning action probabilities into intensities

In general, with Policy Gradients (PG) you can have two 'flavors' of them: Stochastic and Deterministic.

In your case, you can build a NN that outputs continuous actions or one that approximates the sufficient statistics of a probability distribution that you will sample the actions from. The main references are: Theoretical Proofs and NNs with DPG.

There are no rules on which one to use. There are many examples in the papers I cite that DPG outperforms SPG but also the opposite. Sometimes you really want the output to be deterministic. For example, in pricing: If a client comes in a store and you use SGD for assigning prices, the client will encounter every single time different prices (small differences but still differences).

In my personal experience I find it a bit hard to stabilize the DPG with NNs and you need also to come up with a good exploration strategy. However, once the network is stabilized you get the continuous action values that could control your vehicle. This is a detailed example of controlling a car with DPG using tensorflow and Keras.

For general theoretical exploration about the two frameworks I suggest you   this Master Thesis. It provides a concise overview of the two methods (no NN implementations). You can look up also the answer I gave here if you decide to get on board with PGs.


is there any whatever2vec to generate one vector for one document?

Yes, there are such things, but they aren't really related to word2vec.

First of all, Bag of Words is the old "something2vec" approach of representing documents in a vector space. Of course, the resulting vectors are very sparse, and distance does not always behave well in such cases, and it is better to use dot-product based similarities like cosine.

But you can reduce the dimensionality of this vector space with SVD or NMF (Non-Negative Matrix Factorization), and this will give you what you want: dense vectors of small dimensionality for which you can compute the distances.

Then you can go further and use Topic Modelling, which also will give you a vectorial representation of documents, and you can use it for comparing them.


Probability of event given two depandant events

Let $A$, $B$, $C$ be three (potentially dependent) events. By the probability chain rule:
$$ P(A,B,C)=P(A|B,C)P(B,C)=P(A|B,C)P(B|C)P(C) $$
So we can do
\begin{align}
P(A|B,C)
&= \frac{ P(A,B,C)}{P(B|C)P(C)} \\
&= \frac{ P(B,C|A) P(A) }{ P(B|C)P(C) } \\
&= \frac{ P(B|A,C) P(A) P(C) }{ P(B|C)P(C) } \\
&= \frac{ P(B|A,C) P(A) }{ P(B|C) }
\end{align}

See this related post and this one as well.


increase performances in neural networks

basically you should try to use regularization/dropout/batch-normalization and find out the optimal hyper-parameters to improve performance of your model.

After applying these optimizations you can use Randomized Search to find optimal hyper-parameters


When one model is superior in real world use?

Maybe I did not get the question but all looks fine. This is how you do model selection. You have several models (either same algorithm with different parameters or different algorithms. does not matter) and then you perform cross validation to get the best model according to empirical errors coming from validation set. The best model wins the game and is chosen. Everything seems to be right.


Can you clarify pls, where is the suggestion, that we have to select max depth for pruning? As you said it is supposed to be done automatically due to some criterion.
Here is some example of post-pruning:

https://stackoverflow.com/questions/49428469/pruning-decision-trees

Yes, we should select 'K' fold to leave some data for the test of pruning efficiency. How many? It depends on your data. You can check out this Stanford lecture slides, for example, where K = 10 is suggested:

https://web.stanford.edu/class/stats202/content/lec19.pdf

as an example. Usually, the K will depend on the size of your data, based on the size of the test fold you want to leave out.


What is the difference between "Adding more LSTM layers" or "Adding more units on existence layers"?

With the layers - you are trying to generate higher and higher level features as you proceed from layer to layer. Based on the problem at hand, anything from 2 and above layers could serve the purpose. Given that LSTMs operate on sequence data, it means that the addition of layers adds levels of abstraction of input observations over time. In effect, chunking observations over time or representing the problem at different time scales. This works!
With the units - you are trying to capture the function in a better way. Too less and you could underfit. Too many and there could be overfitting.
By adding more layers, you can reduce the number of units in each layer and bring about a greater 'depth' instead of 'width'. What this simply means is that you can approximate the function better using a 'right' combination of width and depth. To arrive at 'right' there is no option other than to play around for hours with different combinations.
One option is the 'stretch pant' approach. Take much higher sizes than when you actually need. Then use regularisation techniques to fit better.


Is confusion matrix possible in one column

Confusion matrix needs groun truth values and predicted values.

You have _____, so you need other part for this multiclass confusion matrix.


Why do activation functions have to be monotonic?

The monotonicity criterion helps the neural network to converge easier into an more accurate classifier.
See this stackexchange answer and wikipedia article for further details and reasons.

However, the monotonicity criterion is not mandatory for an activation function - It is also possible to train neural nets with non-monotonic activation functions. It just gets harder to optimize the neural network.
See Yoshua Bengio's answer.


Keras error "Failed to find data adapter that can handle input" while trying to train a model

There is something wrong with your data. {""} means you have a Python dict that only contains an empty string.


How to get tf tensor value computed in loss function in keras train_on_batch without computing it twice or writing custom loop?

I've solved it(discovered control dependencies and remembered variables)
Basically, I create an assign diff to a variable operation, and with the help of control_dependencies force tf to do this operation every time op is calculated, this way when I get this variable, it doesn't cause graph recalculation

diff_var = tf.Variable()

def custom_loss(labels, predictions):
    diff = labels - predictions
    diff_var_op = diff_var.assign(diff)
    with tf.control_dependencies([diff_var_op]):
        return tf.square(diff)

test code

import tensorflow as tf
sess = tf.Session()
var1 = tf.Variable(1, dtype=tf.float32)
var2 = tf.Variable(2, dtype=tf.float32)
counter = tf.Variable(1, dtype=tf.float32)
var2_op = tf.square(var2)*counter

diff_var = tf.Variable(10, dtype=tf.float32, trainable=False)

diff = var2_op - var1
diff_var_op = diff_var.assign(diff)

with tf.control_dependencies([diff_var_op]):
    op = tf.square(diff)

sess.run(tf.global_variables_initializer())

print('diff var:', sess.run(diff_var))     #10
print('counter:', sess.run(counter))       #1
print('op:', sess.run(op))                 #9
print('diff var:', sess.run(diff_var))     #3
print('-')
counter_op = tf.assign_add(counter, 1)
print('counter:', sess.run(counter))       #1
print('diff var:', sess.run(diff_var))     #3  #still the same
print('var2:', sess.run(var2_op))          #4
print('-')
sess.run(counter_op)
print('after counter_op')
print('counter:', sess.run(counter))       #2
print('var2:', sess.run(var2_op))          #8
#still the same even though var2 has changed because of counter_op
print('diff var:', sess.run(diff_var))     #3
print('op:', sess.run(op))                 #49 #running full op
print('-')
print('after op')
print('diff var:', sess.run(diff_var))     #7
#variable changed, no operations involved


Have a look at matplotlib's annotate.

Here are some examples. I believe it works well with seaborn too.


Even a simple Internet search reveals numerous papers on graph clustering approaches and algorithms. This paper is most likely the best starting point, as it presents a rather comprehensive overview of the topic in terms of the problem as well as approaches, methods and algorithms for solutions. The rest you can find easily via online search. In regard to graph clustering visualization, I recommend you to check my relevant answer - I'm pretty sure that the tools I reference there are able to visualize graph clusters as well.


There are tons of materials on financial (big) data analysis that you can read and peruse. I'm not an expert in finance, but am curious about the field, especially in the context of data science and R. Therefore, the following are selected relevant resource suggestions that I have for you. I hope that they will be useful.

Books: Financial analysis (general / non-R)

Statistics and Finance: An Introduction;
Statistical Models and Methods for Financial Markets.

Books: Machine Learning in Finance

Machine Learning for Financial Engineering (!) - seems to be an edited collection of papers;
Neural Networks in Finance: Gaining Predictive Edge in the Market.

Books: Financial analysis with R

Statistical Analysis of Financial Data in R;
Statistics and Data Analysis for Financial Engineering;
Financial Risk Modelling and Portfolio Optimization with R
Statistics of Financial Markets: An Introduction (code in R and MATLAB).

Academic Journals

Algorithmic Finance (open access)

Web sites

RMetrics
Quantitative Finance on StackExchange

R Packages

the above-mentioned RMetrics site (see this page for general description);
CRAN Task Views, including Finance, Econometrics and several other Task Views.

Competitions

MODELOFF (The Financial Modeling World Championships)

Educational Programs

MS in Financial Engineering - Columbia University;
Computational Finance - Hong Kong University.

Blogs (Finance/R)

Timely Portfolio;
Systematic Investor;
Money-making Mankind.


What happens when scikit-learn does a Lasso Model?

Lasso() created a Lasso regressor object. The .fit() argument makes the regressor "learn" from your data. The parameters are tuned in order to "fit" the training data.

Once the regressor object is fit, you can use it to .predict() on test data.


I want to be able to collapse and sum values dependent on the gene name

You can use dplyr, where df is your data.frame:

df %>% group_by(gene) %>% summarise_if(is.numeric, sum)

This will sum each column separately (assuming it is numeric), grouped by each gene. Any column that is not included in the group_by or is not numeric will be dropped at this stage.


Differential Evolution optimal tolerance parameter

After 4 days of hard continuous computing the results suggest that indeed the tol=0.01 parameter seems to be the best one. This of course depends on the topology of the data and other factors but if the literature suggests that plus my own findings as well this might hint that it might be a universal value applicable for perhaps all shapes of data.

There is a very low correlation between the tolerance parameter and the output result with a sample size of >1000. The average value barely changes as we increase the tol parameter from 1e-10 to 1e-0 (0.9 being the max value I tried). However the computing time drastically lowers itself from around 7 minutes to 5 minutes per run. Since there is no change to the output value but only in the duration of the calculation it is rational to use 0.01 as a tolerance parameter. However above 0.01 the values actually start to increase, which means that 0.01 is as good as it gets.

As for premature terminations of the optimizer, this should not be a problem since this can still be calibrated with the recombination parameter (CR= crossover probability), by setting it to a low value for a delayed convergence and a high one for an earlier one. This in it's turn also has to be optimized.

One thing is worth noting is that using a Latin Hypercube as init values is not good. I have found much higher results with random sampling, much better coverage, and if there is a problem with clustering of the values, then just increase the number of runs. There is no way the differential evolution optimizer will find the best value in 1 run, so it has to be run multiple times either way.

Unfortunately I can't provide the data, but if anyone else has the time to verify these findings on their own data that would be good too.


Why this sequential model is not starting?

from keras.models import Sequential
from keras.layers import InputLayer

model = Sequential()
model.add(InputLayer(input_shape))
model.add(BatchNormalization(axis = 3))

This should work. The first error was because the model was getting reassigned. The second error was because 'Input' is a function of 'layers' class but not a class, 'InputLayer' is a class.


You can write code to analyze this. But this is very much data specific.

In some cases like k-mrans, you already get cluster averages. So you can easily compare the averages.

If you don't have major correlations in your data, then histograms can be insightful. Make stacked histograms, colored per cluster, to see if there is a different distribution in each attribute. Etc.
There are so many ways to do this, and they are not specific to clustering.


Speed of training decrease by adding more GPUs

Using GPUs can accelerate training but as you increase the number of GPUs your training as to be distributed which means your data has to be moved to several GPUs which can cost in term of bandwidth. I would profile the training and see the time spent to move the data to GPUs and et results back. Plus it's harder to syncronize training like this.

If you are using tensorflow 1.14+ try to change the distribution method to "MirroredStrategy" . This tend to work better with multiple GPUs I found.


Which courses should I take in order to learn ML and AI?

In my opinion, start with:

Coursera's Andrew Ng's classes

Machine Learning (very good for the maths)
then go to his deep learning class (I haven't done it but I only hear good things from it)

Udacity had a very good Data Science Specialisation divided in 4 classes. I don't know if it's still available (I did it two years ago) but it is probably either still there or an improved version has replaced it. The 4 courses where:

Intro to Data science
Visualisation
Machine Learning
Text Mining

For the rest, I suggest you to do a quick google search...


How can I use two different datasets as a training model for svm

From a proper methodological standpoint, you should do the scaling you're proposing after the two sets are merged.

Either way, the model is not going to be able to differentiate between, say, an anomalous reading from one generating process that falls into the range generated by the other generating process unless you also make a variable indicating the source system that the observation came from (assuming all else is equal).

You need to make sure that both sets of observations actually represent the same population of possible observations in order to make this modeling decision.


You can add class_weights with dictionary containing class weights, e.g.:
class_weight = {0: 1., 1: 20.}

While SMOTE can be used to synthesize new examples for the minority class (the process is called oversampling) in order to get equal weights.
For Orange please check this link.
Please, provide more information so that we can help you.


convert keywords in one column into several dummy columns

Let's say that your dataframe is called df and the column with your preprocessed text as text.

What you want to do, as you already thought, is to apply a CountVectorizer.

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

count = CountVectorizer(binary = True)
bow = count.fit_transform(df['text'].values)

pd.DataFrame(bow.toarray(), columns = count.get_feature_names())

If you remove the binary parameter from the CountVectorizer you will get the actual frequency of the keyword and not just the appearance with 0-1.


I think this covers your issue in the Keras documentation
https://keras.io/callbacks/#create-a-callback

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []

    def on_batch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))

model = Sequential()
model.add(Dense(10, input_dim=784, kernel_initializer='uniform'))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

history = LossHistory()
model.fit(x_train, y_train, batch_size=128, epochs=20, verbose=0, callbacks=[history])

print(history.losses)
# outputs
'''
[0.66047596406559383, 0.3547245744908703, ..., 0.25953155204159617, 0.25901699725311789]
```


Implement the

Hungarian matching algorithm

It will find the best mapping.


Datasets with 50+ attributes and 2+ tables

You can find the dataset from the following links.

IEEE Dataport (https://ieee-dataport.org/datasets)
https://datasetsearch.research.google.com/


Finding value of theta in linear classification

Since it is an exam question, I will give you hints and leading questions instead of answers.
Note that $x^T w$ must be well-defined, and $x$ is 2-dimensional. What is the dimension of $w$?
How is bias incorporated into the model?
The margin satisfies $\widehat{y} = 0.$ What is the equation of the margin (as a function of x and bias)?
P.S. I expect a typo/vagueness in this question.


Is it possible for a model with a large amount of data to perform very well and reach an extremely low cost within a single epoch?

A few things come to mind here:

If you are using a pre-trained MobileNetV2 on a task which is similar to the pre-training, then you may not need much fine-tuning to get good results. This may explain why you are seeing good training results.
For the poor testing results, are you transforming your frames in the same way you did for training? Any differences between the training pipeline or phone testing pipeline you can think of? Is it possible to test some of your good training results with the phone testing pipeline as a sanity check?


How to train neural network a math multiplication table?

As your function is linear and you have a hidden size of 128, Your model might be suffering from overfitting. Try to lower the hidden layer to 2 or 4 and try to regularize the model using dropout. And also neural net requires a lot of data to train. With small data neural net does not perform well. So if you are unwilling to provide extra data ,you should go for linear model like linearRegressor from sklearn.It will do your task just fine with proper hyperparameter. But if you want to train a neural net then you have to provide a huge amount of data and also choose proper hidden size and regularizer.


Self-attention mechanism did not improve the LSTM classification model

appears that the implementation of the self-attention mechanism has no effect on the model so I think my implementations have some problem.

The first thing I noticed is your base model is doing badly, with low F1 scores, especially for the classes 1..7. Could it be there is just not that much signal in the training data, and the 2-layer LSTM has already sucked it all out?
My second thought is that self-attention is good at finding connections between items however far apart they are in the sequence. But an LSTM is also fairly good at that. So maybe more of the same is not what you need. Self-attention is an essential part of a transformer, because it is the only component that works across the sequence; the only other component is the FFN, which operates on each item in isolation.
Having got those two things off my chest, one thing I've noticed in experiments (and, again, these are in the context of transformers) is the importance of the residual connection. If I take out the residual connection that sits between self-attention and the FFN I get a significantly worse model. (Even though I still have the residual connection across the whole of each layer.)
So you could try adding a residual connection across the self-attention.
The other idea is to move self-attention earlier. Put it before the LSTMs, rather than after them. (Again with a residual connection across it.)
Here is the idea, in (untested) code. I've moved the self attention module to before the first LSTM layer, and then take the sum of the input and the output - that is what makes the residual connection across it.
...

x = Masking(mask_value=0.0, input_shape=(X_train.shape[1], X_train.shape[2])))

x2 = SeqSelfAttention(attention_type=SeqSelfAttention.ATTENTION_TYPE_MUL,
attention_activation='sigmoid')(x)

x = x + x2

x = Bidirectional(LSTM(lstm_unit, dropout=dropout,return_sequences=True))(x)

...

Note that you can't use the Keras sequential class (because the data flow is no longer sequential).
Ref: https://keras.io/examples/nlp/text_classification_with_transformer/ The residual part is hard to spot at first, as it is done as the input to the two layer norm calls.


How to tackle too many outliers in dataset

It looks like most of your data is at 0, right? Either:

Those values are not really 0 but 'missing' and should be treated as missing, rather than as 0. If a lot is missing, this will indeed be problematic for models that do not handle these, because you have to infer some value for them. You can't really drop all these if there are a lot. You could impute a mean or pick a model that handles missing values. Or:

The distribution of these values isn't normal/Gaussian, in which case there's no strong reason to believe that values outside the IQR are 'outliers'. In particular looks like it could be exponential, or a mixture of two distributions in the second case. Just use the values as is. For linear models, some transformation of the input might be better, like its log, but depends a lot on what you're doing.


The training dataset is only used to fit/train the model. The training will extract information from your training data and store it in your model so after training you no longer need it. You should never include training data when you later check the performance/accuracy of your model, that is only the test set.

For predictions and computation of anomaly scores you only need to use the dataset you actually want to check for anomalies/outliers.

For streaming you should train first on offline data which only contains inliers or has a know ratio of outliers. Once it is trained you can use it to make predictions on new streamed data without comparing to your offline data.


How to Connect Convolutional layer to Fully Connected layer in Pytorch while Implementing SRGAN

You simply reshape the tensor to (batch_size, n_nodes) using tensor.view(). In your specific case this would be x.view(x.size()[0], -1).


ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

(Assuming the error comes from the 2nd line)
This question on StackOverflow has a great answer to your problem
Essentially, change
 if ((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up'):

to this
if (((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up')).any():

or to this, if all of the values should be True
if (((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up')).all():

Hope this helps :)


Loop through df.columns checking whether there is a column name or if it is empty:

import pandas as pd

# create df with two empty columns
df = pd.DataFrame({'a': [1,2], 'b': [1,2], 'c': [1,2], 'd':[1,2]})
df.columns = ['', 'b', '', 'd']

df with empty columns:

       b     d
 0  1  1  1  1
 1  2  2  2  2

# new of list columns names must be the same length as number of empty columns
empty_col_names = ['x', 'y']
new_col_names = []
count = 0

for item in df.columns:
    if item == '':
        new_col_names.append(empty_col_names[count])
        count += 1
    else:
        new_col_names.append(item)

df.columns = new_col_names

df with filled columns:

    x  b  y  d
 0  1  1  1  1
 1  2  2  2  2


Giving a score (from 1 - 5) using weights

If I understand the question correctly, this 0-5 weighting is quite arbitrary and is based on your criteria, right? e.g. how loyal some customer is. In this case, a practical way of doing it could be the following:

Transform values of all features to be on the range (0,5). In order to achieve this normalize them to take values into (0, 1) and then multiply with 5 (see here for normalization with pandas)
Apply weights on the features: each weight should be in the range (0,1).
For each customer, multiply weights with transformed feature values (in a similar way as you showed) and calculate the average. This should be in (0,5).

In case that weights represent something that you can get past information on E.g. weights might mean how likely it is that the certain customer would be interested in some product and you can have access on historical information about whether customers with those characteristics (features) have bought it in the past, you can then train a probabilistic classification model and then rescale this probability into (0, 5) in order to turn it into weights. It this context, you might find useful this and this discussions.